debugging - freeCodeCamp.org

From Symptoms to Root Cause: How to Use the 5 Whys Technique

Ashutosh Krishna — Thu, 23 Apr 2026 16:48:41 +0000

Most teams don't struggle because they can't fix problems. They struggle because they fix the wrong thing.

An API fails in production. You restart the service, errors go away, and it feels resolved. Until it happens again. And again. What's happening here is simple: you're treating symptoms, not the underlying cause.

The 5 Whys technique is a straightforward way to deal with this. It comes from the Toyota Production System and was designed to help teams dig deeper into problems instead of settling for quick fixes.

The idea is simple. Ask "why" repeatedly until you reach the real cause.

But in practice, this is where things go wrong.

Teams often:

Stop too early
Assume answers without checking data
Focus on people instead of systems
Treat "five" as a rule instead of a guideline

So even though the process looks structured, the outcome is still shallow.

In this article, we'll focus on how to actually use the 5 Whys in real situations. Not just the theory, but what it looks like when you apply it to an engineering problem.

Here's What We'll Cover:

What is the 5 Whys Technique?
Origins of the 5 Whys Method
How to Conduct an Effective 5 Whys Analysis
Real-World Example: Applying 5 Whys in an Engineering Scenario
When to Use (and When Not to Use) 5 Whys
Benefits of the 5 Whys Technique
Common Pitfalls and Limitations
Tips for Using 5 Whys Effectively
Summary

What is the 5 Whys Technique?

The 5 Whys technique is a way to break down a problem by repeatedly asking why it happened, with the goal of reaching a cause that actually explains the issue and can be addressed.

At its core, it's not about the number five. The name can be misleading. What matters is the process of following a chain of cause and effect until the explanation stops being superficial and starts becoming useful.

Each answer you uncover should move you one level deeper. You start with what went wrong, then explore what led to it, and continue until you reach something that is both believable and actionable. In most real situations, that final answer is not a single event but a gap in a system, a missing check, or an assumption that was never validated.

The technique became widely known through the Toyota Production System, where it was used to improve processes by focusing on causes rather than quick fixes.

That context is important because it highlights the original intent. The goal was not just to explain problems, but to prevent them from happening again.

A simple example makes this clearer. Imagine a mobile app suddenly starts crashing after a release. Asking "Why?" might look like this:

Why is the app crashing? → Because a null value is being accessed in the code.
Why is there a null value? → Because the API response is missing a required field
Why is the field missing? → Because a recent backend change made the field optional.
Why was this change not handled in the app? → Because the app assumes the field is always present.
Why was this assumption not caught earlier? → Because there are no contract tests validating API responses.

At this point, the issue is no longer just "fix the null check". The deeper problem is the lack of validation between systems, which allows breaking changes to slip through.

A useful way to think about the 5 Whys is that it forces you to stay with the problem a little longer than you normally would. Most of the time, the first explanation feels sufficient, so it's easy to stop there. This method pushes you to go one step further, and then another, until the explanation holds up under scrutiny.

At the same time, it's not a rigid formula. You might reach a solid root cause in three steps, or it might take more than five. The quality of the reasoning matters more than the count.

Origins of the 5 Whys Method

The 5 Whys method comes from the Toyota Production System, a manufacturing approach focused on continuous improvement and problem solving at the source.

It's often associated with Sakichi Toyoda, whose philosophy was simple: don’t just fix a problem. Understand why it happened so it doesn't happen again.

Inside Toyota, this wasn't treated as a formal tool or checklist. It was part of the day-to-day way of working. When something went wrong on the production line, the goal wasn't to get things running quickly and move on. The goal was to stop, investigate, and make sure the same issue wouldn't repeat.

That mindset is important to understand. The 5 Whys was never meant to be a rigid exercise where you ask five questions and stop. It was a way to encourage deeper thinking and accountability in processes.

Another key idea in the Toyota system is that problems are usually caused by processes, not people. Instead of asking "who made the mistake", the focus is on "what allowed this mistake to happen". The 5 Whys fits naturally into this approach because it pushes you toward system level causes rather than individual blame.

Over time, the method spread beyond manufacturing and is now used in software engineering, product teams, operations, and many other fields. The context has changed, but the core idea remains the same: if you don't understand the cause, you're likely to see the same problem again.

This origin story is useful not just as background, but as a reminder of intent. The value of the 5 Whys doesn't come from the questions themselves. It comes from the discipline of not settling for the first answer.

How to Conduct an Effective 5 Whys Analysis

A 5 Whys analysis works best when it is treated as a structured way of thinking, not a checklist to rush through. The quality of the outcome depends less on how many times you ask "why" and more on how carefully you reason through each step.

It helps to approach it in stages, each with a clear purpose.

Step 1: Define the Problem Clearly

Start with a problem statement that is specific and observable. Avoid vague descriptions like "the system is slow" or "things are failing". Instead, describe what actually happened in a way that can be verified.

For example, "API response time exceeded 5 seconds for 30 percent of requests between 2 PM and 3 PM" is much more useful than "API is slow".

A clear problem statement keeps the analysis grounded. If the starting point is fuzzy, the entire chain of reasoning will drift.

Step 2: Ask "Why" Iteratively

Once the problem is defined, begin asking why it happened. Each answer should directly address the question before it and naturally lead to the next one.

The key here is continuity. Every step should feel like a logical extension of the previous one. If you find yourself jumping topics or introducing unrelated explanations, it's a sign that the chain is breaking.

Keep going until the answers stop being immediate symptoms and start pointing toward underlying conditions or decisions.

Also, don't force the process to stop at five. Some problems may need fewer steps, while others may need more. What matters is reaching a point where the explanation is meaningful and actionable.

Step 3: Validate Each Answer with Evidence

This is where many analyses go wrong. It's easy to come up with plausible answers, but plausibility is not enough.

Each "why" should be backed by some form of evidence. This could be logs, metrics, recent changes, or direct observation. If an answer can't be verified, treat it as a hypothesis and confirm it before moving forward.

Without validation, the entire analysis becomes a chain of assumptions. Even if the final answer sounds reasonable, it may not reflect reality.

Step 4: Identify the Root Cause

A good root cause is one that explains the sequence of events and can be acted upon to prevent the issue in the future.

In many cases, this turns out to be a gap in a process rather than a single technical failure. It could be a missing validation step, an incomplete test, or an assumption that was never challenged.

If the final answer still feels like a symptom, you probably need to go one level deeper. On the other hand, if the answer points to something you can change in your system or workflow, you are likely in the right place.

Step 5: Define Corrective Actions

The analysis is only useful if it leads to meaningful action.

Once you've identified the root cause, the next step is to define changes that prevent the problem from happening again. These should go beyond quick fixes and address the underlying issue.

For example, instead of just fixing a bug, you might introduce better testing, add monitoring, or improve review processes.

Good corrective actions share a few traits: they're specific, practical to implement, and they directly address the root cause identified in the analysis.

Real-World Example: Applying 5 Whys in an Engineering Scenario

To see how this works in practice, let’s walk through a realistic backend issue. The goal here is not just to reach an answer, but to show how each step builds on evidence and leads to something actionable.

The Problem:

Users report intermittent failures while fetching order details:

GET /api/orders/{id}
→ HTTP 500 Internal Server Error

Application logs show:

// Java 21 example (Spring Boot style logging)
logger.error("Database connection timeout while fetching order", ex);

At this point, it's tempting to conclude that the database is the problem. But that's only what we can see on the surface.

Applying the 5 Whys

1. Why did the API return a 500 error?

Because the database query timed out.

This is directly supported by the error logs, so we can treat it as a confirmed fact.

2. Why did the query time out?

Because the database connection pool was exhausted.

Metrics show that all available connections were in use during peak traffic.

3. Why was the connection pool exhausted?

Because some requests were holding database connections for too long.

Slow query logs confirm that a subset of queries had unusually high execution times.

4. Why were some queries slow?

Because a recently introduced feature added a query on a non-indexed column.

Looking at recent deployments reveals a change that introduced filtering without proper indexing.

5. Why was an unoptimized query deployed to production?

Because there is no performance validation step in the development or release process.

There are no checks in code review or CI/CD to catch inefficient database queries before deployment.

Root Cause

The issue is not the timeout itself.

It's this:

The system allows inefficient database queries to reach production without any safeguards.

What a Shallow Fix Would Look Like

If we stopped early, we might:

Increase the database timeout
Increase the connection pool size

These might reduce the frequency of failures, but they don't solve the underlying problem.

What a Strong Fix Looks Like

A proper 5 Whys analysis leads to changes that improve the system:

Add appropriate indexing for frequently queried fields
Introduce query performance checks in CI/CD pipelines
Add monitoring and alerts for slow queries
Include database considerations in code reviews

Why This Example Matters

The difference between a shallow fix and a real solution is depth.

The first explanation often feels sufficient, especially under pressure. But stopping there means the issue is likely to return in a different form.

The value of the 5 Whys comes from following the chain all the way to something you can change in your system.

When to Use (and When Not to Use) 5 Whys

Like any problem-solving method, the 5 Whys is useful in the right context and less effective in others. Knowing when to apply it is just as important as knowing how to use it.

If used appropriately, it can uncover meaningful insights. If used in the wrong situation, it can lead to oversimplified or misleading conclusions

When to Use 5 Whys

The 5 Whys is most useful when your goal is to understand why something happened, not just to fix it and move on.

It works well in situations where problems are recurring or not fully explained by the first answer. For example, production incidents, repeated bugs, or issues that reappear after a quick fix are strong signals that you need deeper analysis. In these cases, the technique helps uncover what is happening beneath the surface.

It's also effective during retrospectives and postmortems. When a release doesn't go as expected or a sprint runs into issues, the 5 Whys helps teams move beyond observations like "this failed" and get to "why did this fail in the first place".

In general, use it when:

The problem is not obvious
The issue has occurred more than once
You want to prevent recurrence, not just resolve the current instance

When Not to Use 5 Whys

The 5 Whys has its limits, and using it in the wrong context can lead to oversimplified conclusions.

If a problem involves multiple interacting factors, a single chain of "why" questions may not capture the full picture. Complex systems often have several contributing causes, and forcing them into one linear explanation can hide important details. In such cases, the 5 Whys should be combined with other approaches.

It's also less effective when there's not enough data. If each answer is based on assumptions rather than evidence, the analysis quickly becomes unreliable. The method depends on validation at every step.

Another limitation is in time-critical situations. During an active incident, the priority is to restore the system. The deeper analysis should happen later, once things are stable.

Finally, if your goal is quantitative analysis or optimization, the 5 Whys alone isn't enough. You'll need more data-driven methods to support decision making.

A simple rule of thumb is this. If you are trying to learn from a problem, use the 5 Whys. If you are trying to fix something immediately or analyze complex data, use it carefully or alongside other techniques.

Benefits of the 5 Whys Technique

The 5 Whys technique is simple, but it offers several powerful benefits that can help you solve problems more effectively and make lasting improvements. Here are the key advantages:

Simple and Easy to Apply

One of the biggest strengths of the 5 Whys is how easy it is to start using. You don't need special tools, training, or complex frameworks. It can be applied in a quick discussion, during debugging, or as part of a formal postmortem.

This low barrier makes it accessible across teams, regardless of experience level.

Encourages Deeper Thinking

The method naturally pushes you to go beyond the first explanation. Instead of reacting to what's visible, it encourages you to question why the problem occurred in the first place.

This shift from surface-level fixes to deeper understanding often leads to better decisions.

Promotes System-Level Improvements

When used correctly, the focus moves away from individual people and toward systems. Instead of asking who made a mistake, the analysis asks what allowed the mistake to happen.

This leads to improvements in processes, safeguards, and overall system design rather than one-off fixes.

Works Well in Team Settings

Because the approach is simple, it's easy for multiple people to contribute. Different perspectives help uncover gaps that might otherwise be missed.

It also creates a shared understanding of the problem, which is valuable during retrospectives and incident reviews.

Helps Prevent Recurring Issues

Quick fixes often solve the immediate problem but don't stop it from happening again. The 5 Whys helps identify underlying causes, which makes it easier to prevent similar issues in the future.

Over time, this leads to more stable systems and fewer repeated incidents.

Common Pitfalls and Limitations

While the 5 Whys technique is useful, it’s not always perfect. There are some limitations to keep in mind, so you can use it effectively and know when it might not be enough.

Stopping Too Early

One of the most common mistakes is ending the analysis after the first or second answer. These early answers usually describe symptoms, not causes.

Stopping too soon leads to fixes that address the surface but leave the underlying issue unresolved.

Treating Assumptions as Facts

It's easy to come up with explanations that sound reasonable. But without evidence, they're just assumptions.

If each step isn't validated with logs, metrics, or observations, the entire analysis can drift away from reality.

Focusing on Individuals Instead of Systems

Answers like "someone made a mistake" don't add much value. While they may be true, they don't explain why the system allowed that mistake to have an impact.

Focusing on processes and safeguards leads to more meaningful improvements.

Oversimplifying Complex Problems

The 5 Whys follows a linear chain of reasoning, but real-world systems often have multiple contributing factors.

Relying on a single chain can hide important interactions. In such cases, the method should be combined with other approaches.

Treating It as a Rigid Formula

The name suggests asking "why" five times, but this shouldn't be taken literally. Some problems require fewer steps, while others need more.

Forcing the structure can lead to artificial or weak conclusions.

Not a Replacement for Deeper Analysis

The 5 Whys isn't designed for every type of problem. For complex system failures, performance optimization, or data-heavy investigations, additional tools and methods are often required.

It works best as a starting point or a complement to other techniques, not a complete solution on its own.

Tips for Using 5 Whys Effectively

To get the most out of the 5 Whys technique, there are a few tips that can help you use it effectively. These will guide you to ask the right questions and reach useful, actionable insights.

Start with a Clear, Specific Problem

A vague problem leads to vague answers. Spend a little extra time making sure the problem statement is precise and based on observable facts. This keeps the analysis grounded and avoids unnecessary detours.

Base Every Step on Evidence

Treat each answer as something that needs to be verified. Use logs, metrics, recent changes, or direct observations to support your reasoning. If something can't be validated, call it out as a hypothesis and confirm it before moving forward.

Keep the Chain Logical and Connected

Each "why" should naturally follow from the previous answer. If the reasoning starts to jump between unrelated ideas, pause and re-evaluate. A clean, logical chain is a strong indicator that you're on the right track.

Focus on Systems, Not Individuals

Avoid stopping at explanations that point to human error. Instead, ask what allowed that error to have an impact. This shift in thinking leads to improvements that actually reduce the chances of similar issues in the future.

Do Not Force Exactly Five Steps

The number five is a guideline, not a rule. Some problems become clear in three steps, while others need more exploration. Stop when you reach a cause that's both convincing and actionable.

Involve the Right People

If possible, do the analysis as a group. People from different parts of the system bring different perspectives, which helps uncover details that might otherwise be missed. It also creates shared ownership of both the problem and the solution.

Turn Insights into Actions

The analysis only matters if it leads to change. Make sure the final outcome includes clear, practical steps that address the root cause. Without this, even a well-done analysis has limited impact.

Summary

The 5 Whys is a simple technique, but using it well takes some discipline.

At its core, it's about resisting the urge to stop at the first explanation. By following the chain of cause and effect, you move from symptoms to something you can actually fix. In many cases, that turns out to be a gap in a process rather than a one-off failure.

When applied thoughtfully, it helps teams learn from problems instead of just reacting to them. Over time, this leads to better systems, fewer recurring issues, and more confidence in how problems are handled.

The key is to treat it as a way of thinking, not just a set of steps.

How to Fix a Failing GitHub PR: Debugging CI, Lint Errors, and Build Errors Step by Step

qacheampong — Wed, 22 Apr 2026 17:19:57 +0000

While many guides explain how to set up Continuous Integration pipelines, not very many show you how to debug them when things go wrong across multiple layers.

This is a common experience when contributing to open source: you make a small change, open a pull request, and suddenly everything fails.

Not just one check, but multiple:

Lint errors
YAML validation issues
Build failures
Deployment failures

Even more confusing, you may see errors in parts of the codebase you didn’t modify.

In this article, you'll learn how to debug these issues step by step. The goal is not just to fix one pull request, but to understand how CI systems validate your changes.

This guide is based on a real debugging experience from contributing to an open source documentation project.

While this example comes from a documentation project, the debugging workflow applies to many repositories that use CI pipelines, linting tools, and automated builds.

Prerequisites

To follow this guide, you should have:

Basic familiarity with Git and pull requests
A GitHub account
Some exposure to CI/CD concepts (helpful but not required)

Understanding the CI Pipeline (What’s Actually Happening)

In many projects, you will see the term CI/CD, which stands for Continuous Integration and Continuous Deployment (or Delivery).

In this guide, we'll focus specifically on the CI part – that is, Continuous Integration. This refers to the automated checks that run when you push code or open a pull request. These checks validate your changes before they're merged into the main codebase.

CD (Continuous Deployment/Delivery), on the other hand, typically handles what happens after those checks pass, such as deploying the application.

Understanding this distinction is important because most of the issues we debug in this guide happen during the CI stage.

Most repositories run multiple automated checks when you open a pull request:

Linting tools (for example, markdownlint, yamllint) enforce formatting rules
Build systems (for example, mdBook) validate structure and generate output
Deployment checks (for example, Netlify) ensure that the site can be built and served
Merge controllers (for example, Tide) enforce approval policies

A key point to remember: CI systems validate the entire set of files in your commit, not just the lines you changed.

How a CI Pipeline Processes Your Pull Request

When you push code or open a pull request, the CI pipeline runs several checks in sequence.

Let’s visualize how these checks are connected in a typical CI pipeline.

Figure: A simplified CI pipeline showing how linting, build, and deployment checks are executed sequentially.

The above diagram shows a sequential CI pipeline with feedback loops, where failures at any stage return you to fix the issue before continuing.

Let’s break down what this diagram shows:

You start by pushing code or opening a pull request.
The CI pipeline begins running automated checks.
The first set of checks typically includes linting tools like markdownlint or yamllint.
- If linting fails, the pipeline stops, and you must fix formatting issues before continuing.
- If linting passes, the pipeline moves to the build step (for example, mdBook in documentation projects).
- If the build fails, it usually means there is a structural issue, such as duplicate entries or invalid references.
After a successful build, deployment checks (such as Netlify previews) run.
- If deployment fails, the issue is often related to configuration or build output.
If all steps pass, the pull request becomes ready for review.

A Practical Debugging Workflow

Step 1: Fix Authentication and Permission Issues

Before CI runs, your push can fail due to authentication errors.

Example error:

refusing to allow a Personal Access Token to create or update workflow

This happens because GitHub requires special permissions when your commit includes files under:

.github/workflows/

The solution is to regenerate your Personal Access Token (PAT) with:

repo access
workflow permission

Step 2: Run Lint Checks Locally

Relying only on CI feedback slows you down because you have to push changes and wait for the pipeline to run before seeing errors.

Running checks locally allows you to catch issues immediately before pushing your code.

In practice, you should do both:

Run checks locally to catch errors early and reduce iteration time
Use CI as the final validation to ensure your changes meet the repository’s standards

Think of local checks as your first line of defense, and CI as the final gate before your code is accepted.

Here's an example (Markdown linting):

npm install -g markdownlint-cli2
markdownlint-cli2 docs/**/*.md

Step 3: Fix Common Markdown Lint Errors

Here are some common issues you may encounter:

1. Non-descriptive links

Non-descriptive links like "here" don't give readers any context about where the link leads. This makes documentation harder to understand and less accessible, especially for users relying on screen readers.

Instead of writing:

[here](https://example.com)

Use descriptive text like:

[command help documentation](https://example.com)

2. Line length violations

Many projects enforce a maximum line length (often around 80 characters) to improve readability across different devices and editors.

If a line is too long, you can split it into multiple lines without changing the meaning.

To do this, break the line at natural points such as spaces between words or after punctuation. Avoid breaking words or disrupting the sentence structure.
For example:

This is a long sentence that should be split across multiple
lines to satisfy lint rules.

3. List indentation issues

List indentation errors occur when nested list items aren't aligned consistently. This can break formatting and cause linting errors.

To avoid this, just make sure you use consistent spacing (usually 2 spaces per level).

Example (incorrect):

- Item 1
 - Subitem

Correct version:

- Item 1
  - Subitem

Step 4: Fix YAML Inside Markdown Code Blocks

YAML has strict formatting rules, including proper indentation, key-value structure, and consistent spacing.

Even when YAML appears inside a markdown code block, tools like yamllint still validate its structure.

Example (incorrect):

metadata:
annotations:

Correct version:

metadata:
  annotations:
    capi.metal3.io/unhealthy: "true"

In the incorrect example, annotations is not properly nested under metadata, and no key-value pair is defined.

In the corrected version:

annotations is properly indented under metadata
a valid key-value pair is added (capi.metal3.io/unhealthy: "true")

This structure satisfies YAML’s requirement for proper hierarchy and formatting.

Step 5: Fix Build Errors After Lint Passes

Passing lint checks doesn't guarantee that your build will succeed.

This is because linting focuses on syntax and formatting, while the build process validates the structure and integrity of the entire project.

Build failures often occur due to issues such as:

Duplicate entries in navigation files
Missing or incorrectly referenced files
Invalid configuration settings

Even if your syntax is correct, the build system ensures everything connects properly.

For example, in documentation projects using tools like mdBook, a duplicate entry in SUMMARY.md can cause the build to fail even when all files pass lint checks.

Step 6: Debug Cascading CI Failures

CI pipelines are layered. One failure can trigger multiple downstream failures.

For example, imagine a YAML indentation error:

YAML error → build fails → deploy fails → multiple checks fail

To fix this:

Identify the first failing step in the CI logs
Fix that issue
Re-run the pipeline

In this example, the YAML indentation error is the root cause. Once you fix the YAML formatting, the lint check passes, which allows the build to proceed and the deployment step to succeed.

This is why it is important to always fix the first failure in the pipeline rather than trying to address all errors at once.

Step 7: Handle Git Issues During CI Debugging

When working with updated branches, you may encounter:

Diverged branches
Rebase conflicts
Push rejections

To resolve these issues, you typically need to update your branch using one of two approaches:

Option 1: Rebase (clean history)

git pull --rebase

Rebasing rewrites your commit history so your changes appear on top of the latest version of the branch.

Use carefully:

Only rebase your own branches
Avoid rebasing shared branches

Option 2: Merge (safer)

git pull --no-rebase

Merging preserves the full commit history and is safer when working with others, but it may introduce additional merge commits.

Pushing your changes safely

After updating your branch, you may need to push changes:

git push --force-with-lease

Avoid using:

git push --force

The --force option can overwrite the other contributors’ work. The --force-with-lease option is safer because it only pushes if the remote branch has not changed unexpectedly.

Key Takeaways

CI validates your entire commit, not just the specific lines you changed
Linting and build systems enforce different rules
YAML inside markdown must be structurally correct
Documentation builds can fail due to structural issues
Running checks locally significantly reduces debugging time

Conclusion

Debugging a failing pull request isn't just about fixing syntax errors.

You also need to understand how different systems interact:

Version control
CI pipelines
Linting tools
Build processes

Once you understand how these systems work together, you can debug issues systematically instead of guessing.

The next time your pull request fails, you will know exactly where to start and how to fix it.

Debugging CI issues may feel overwhelming at first, but with a structured approach, you can turn failures into a clear path for improvement.

How to Troubleshoot Ghost CMS: Fixing WSL, Docker, and ActivityPub Errors

Abdul Talha — Thu, 19 Mar 2026 17:28:52 +0000

Setting up Ghost CMS (Content Management System) on your local machine is a great way to develop themes and test new features. But if you're using Windows or Docker, you might run into errors that stop your progress. And debugging takes time away from your actual development work.

In this guide, you'll learn the root causes and exact fixes for three common Ghost CMS deployment errors:

Error 1: SQLite installation failures on Windows.
Error 2: Docker containers crashing with Code 137 (memory limits).
Error 3: "Loading Interrupted" errors in the ActivityPub Network tab.

By the end of this article, you'll have a stable, working local Ghost setup. You'll know how to properly use WSL for Node.js apps, manage Docker resources, and successfully configure Ghost's new social web features.

Error 1: SQLite Installation Failures on Windows

The Symptom

When you run the command ghost install local on a Windows machine, the setup fails. You will see a long list of red text in your terminal that looks like this:

Error: Cannot find module 'sqlite3'
...
node-pre-gyp ERR! stack Error: Failed to execute...
...
MSB4019: The imported project "C:\Microsoft.Cpp.Default.props" was not found.

The error usually mentions "sqlite3" and says it "failed to execute" or is "missing."

The Cause

Ghost uses SQLite to store your blog's data. SQLite is a "native module." This means it needs a small piece of code that must be built to fit your computer's system perfectly.

Because Ghost was created to run on Linux servers, it expects to find Linux build tools to make these files. Windows uses different tools and a different way of organising files. When the Ghost CLI tries to build the SQLite files on Windows, it can't find the tools it needs, so the installation stops. Using WSL gives Ghost the Linux environment it expects.

How to Fix it:

You can use Windows Subsystem for Linux (WSL) to create a working setup.

Open your WSL terminal (like Ubuntu).
Check your tools by running node --version, npm --version, and python3 --version.
Install the Ghost CLI globally inside WSL:
```
npm install -g ghost-cli@latest
```
Run the local setup command:
```
ghost install local
```
Start the server:
```
ghost start
```

How to Verify:

Open your web browser and go to http://localhost:2368. You should see the default Ghost welcome page load without errors.

Error 2: Docker Container Exiting with Code 137

The Symptom:

When you're running Ghost using Docker Compose, the containers crash. The terminal logs show Ghost admin container exiting with code 137 or Admin service killed due to memory constraints.

The Cause:

So why does this happen? Well, error code 137 means your computer ran out of memory (RAM) and stopped the container. This usually happens if you try to run the full Ghost developer setup (which includes 15+ extra tools) on a standard computer.

How to Fix it:

To fix this error, you can switch from the complex setup to a simple setup using the official Ghost Docker image.

To do this, first stop and remove the broken containers:

docker-compose down -v
docker system prune -a

Then create a new docker-compose.yml file with only the basic tools (Ghost and a database):

services:
  ghost:
    image: ghost:latest
    restart: always
    ports:
      - "2368:2368"
    environment:
      database__client: mysql
      database__connection__host: mysql
      database__connection__user: root
      database__connection__password: yourpassword
      database__connection__database: ghost
      url: http://localhost:2368
    volumes:
      - ghost_content:/var/lib/ghost/content

  mysql:
    image: mysql:8.0
    restart: always
    environment:
      MYSQL_ROOT_PASSWORD: yourpassword
      MYSQL_DATABASE: ghost
    volumes:
      - mysql_data:/var/lib/mysql

volumes:
  ghost_content:
  mysql_data:

Then start the simple setup:

docker-compose up -d

How to Verify:

Type docker-compose ps in your terminal. You should see both the ghost and mysql containers listed with a status of "Up".

Error 3: "Loading Interrupted" in Network Analytics

The Symptom:

When you click the Analytics → Network tab in your local Ghost admin panel, the page shows a "Loading Interrupted" error. Your terminal logs show 404 errors and webhook failures:

INFO "GET /.ghost/activitypub/v1/feed/reader/" 404 52ms
ERROR No webhook secret found - cannot initialise

The Cause:

The Network tab acts as an ActivityPub reader, not a normal analytics dashboard. This error happens because ActivityPub is not set up for local use. It needs extra tools (Caddy, Redis) and a clean web address without port numbers to work.

How to Fix it:

To fix this error, just run Ghost with its required Docker tools and update your local config file to turn on the social web features.

First, start the required tools (Caddy, MySQL, Redis) from your Ghost folder:

SSH_AUTH_SOCK=/dev/null docker compose up -d caddy mysql redis

Then open your config.local.json file. Set the URL to a clean localhost address (remove the :2368 port) and turn on the developer features:

{
    "url": "http://localhost",
    "social_web_enabled": true,
    "enableDeveloperExperiments": true
}

Stop your current Ghost process:

pkill -f "yarn dev:ghost"

And restart Ghost with the new settings:

yarn dev:ghost

How to Verify:

Log back into your Ghost admin panel and click Analytics → Network. The error message will be gone, and you will see the ActivityPub feed instead.

Conclusion

Local setups can be hard, especially when mixing Windows, Docker, and new features like ActivityPub.

By fixing these three errors, you did more than just get Ghost running. You learned how to bypass Windows limits using WSL, how to manage Docker memory, and how Ghost routes social web traffic.

You now have a stable, fast, and fully working Ghost CMS workspace ready for your content.

Let’s connect! You can find my latest work on my Technical Writing Portfolio or reach out to me on LinkedIn.

How to Debug React State Updates Like a Pro (Without Polluting Production)

Kelechi Apugo — Mon, 02 Feb 2026 21:22:46 +0000

When you’re debugging a large React codebase, you might start to feel like a detective. Especially when you are looking for unexpected state changes, components that re-render when they like, or Context values that disappear into thin air without any prior warning sign.

And the main difficulty isn’t necessarily what went wrong – it’s pinpointing where it went wrong.

React offers powerful ways to change state, but it doesn’t specify who or what caused those changes. In large apps with many layers of components, hooks, and contexts, this lack of insight can turn simple bugs into frustrating, time-consuming puzzles.

This is where more innovative debugging methods become crucial. Before now, the go-to solution was to sprinkle console.log calls at key points or to fall back to DevTools.

But these days, you can write a small but powerful utility function that can catch the criminal involved in the crimes against your codebase. This utility function can log changes, display meaningful stack traces, and work smoothly with useState, useReducer, Context providers, and custom hooks. And all of the above can occur while remaining invisible in production.

This article guides you through how to use this helper function to improve clarity, minimise guesswork, and debug efficiently without affecting performance or code cleanliness in your live environment.

The Problem
Why This Problem Exists
My Solution: createDebugSetter
- Practical Examples of createDebugSetter
Best Practices for using createDebugSetter
Things to Avoid
Bonus: How to Convert createDebugSetter to a Hook
Conclusion

The Problem

React’s state system is powerful, but it hides too much information when something goes wrong – for example, when an unexpected update happens or a component re-renders endlessly. React doesn’t tell you what triggered the update, what changed, or why it happened. This lack of visibility creates several challenges.

The first is that you can’t easily see which component, function, or effect initiated a state update. In large applications, where the same state may be modified from multiple places, this quickly turns debugging into guesswork. Without clear traces, developers often sprinkle console.log throughout their code to find the source of a single update.

Secondly, React lacks a built-in method for directly comparing previous and current values. This complicates diagnosing whether a bug stems from an incorrect calculation, a faulty API response, or erroneous business logic. The challenge increases with nested objects, arrays, or shared context.

Thirdly, Context updates can trigger re-renders across the entire tree, even for components wrapped in memoisation. But React doesn’t explain why a particular provider changed, leaving teams to wonder what triggered the cascade.

Finally, infinite loops caused by effects, unstable dependencies, or repeated setState calls provide no clues in the console. You only see symptoms like “loading…” repeating endlessly, with no indication of the source.

All of this makes debugging complex React apps frustrating, slow, and often misleading without additional tools or structured techniques.

Why This Problem Exists

React intentionally conceals its internal update process to keep the framework fast and predictable.

Because of this:

setState() doesn’t report where it was called from
Context re-renders can originate from anywhere.
State overrides can happen silently.
Debugging often relies on manually adding console logs.

In large applications, this lack of visibility makes it nearly impossible to trace unexpected state changes.

My Solution: `createDebugSetter`

A small helper function, createDebugSetter, wraps your state setter and logs:

The label of the state
The new value
A complete stack trace showing exactly where the update originated

And best of all, it automatically disables itself in production using NODE_ENV so there’s no impact on your live app.

createDebugSetter

export function createDebugSetter(
  label: string,
  setter:
    | React.Dispatch>
    | React.Dispatch
): React.Dispatch<React.SetStateAction<unknown>> | React.Dispatch<unknown> {
  // In production, return the original setter unchanged
  if (import.meta.env.PROD /* vite-react */) {
    return setter;
  }

  // Create a wrapper that logs before calling the original setter
  return (value: React.SetStateAction | unknown) => {
    // Log the state change
    console.groupCollapsed(
      `%c🔄 [${label}] State Update`,
      "color: #adad01; font-weight: bold;"
    );
    console.log("🆕 New value:", value);
    console.trace("📍 Update triggered from:");
    console.groupEnd();

    // Call the original setter
    setter(value);
  };
}

The function above is a debugging wrapper for React state setters that logs during development.

It takes two parameters: a label for identification and the setState function you want to debug. In development mode, every state update triggers a collapsible console log that shows the new value and the stack trace of the update's origin. In production, the hook skips entirely and returns the original setter unchanged, ensuring zero runtime overhead in deployed applications.

How it does it:

 // In production, return the original setter unchanged
  if (import.meta.env.PROD /* vite-react */) {
    return setter;
  }

In production, the createDebugSetter function returns the React setState as-is. This is because we don’t want to log anything when our code is running in a production environment. Here, we’re using the import.meta.env.PROD from React-vite. It returns a Boolean value that tells us if it’s in the production environment or not.

If it’s not in production, we return the modified setter below.

// Create a wrapper that logs before calling the original setter
  return (value: React.SetStateAction | unknown) => {
    // Log the state change
    console.groupCollapsed(
      `%c🔄 [${label}] State Update`,
      "color: #adad01; font-weight: bold;"
    );
    console.log("🆕 New value:", value);
    console.trace("📍 Update triggered from:");
    console.groupEnd();

    // Call the original setter
    setter(value);
  };

This new setter function first logs a collapsible console group with the label and emoji. Then it shows the new value being set. After that, it displays a stack trace showing where the update was triggered. Lastly, it calls the original setter to actually update the state.

Practical Examples of createDebugSetter

Let’s now see how createDebugSetter can be used in several places within a codebase.

Context Providers

You can use createDebugSetter within a Context provider to log state changes when setState is called. This can help log and trace state changes whenever setState is called in a Context Provider anywhere in the application.

import { createContext, useContext, useState, type ReactNode } from "react";
import { createDebugSetter } from "../utils/createDebugSetter";

interface User {
  name: string;
  email: string;
  role: string;
}

interface UserContextType {
  user: User | null;
  setUser: React.Dispatchnull>>;
  login: (name: string, email: string) => void;
  logout: () => void;
}

const UserContext = createContextundefined>(undefined);

export function UserProvider({ children }: { children: ReactNode }) {
  const [user, setUserOriginal] = useStatenull>(null);

  // Wrap setter with debug functionality
  const setUser = createDebugSetter(
    "UserContext",
    setUserOriginal
  ) as React.Dispatchnull>>;

  const login = (name: string, email: string) => {
    setUser({
      name,
      email,
      role: "user",
    });
  };

  const logout = () => {
    setUser(null);
  };

  return (
    
      {children}
    
  );
}

In the above code sample, we create a modified setUserOriginal called setUser that uses createDebugSetter under the hood. We then expose it to the context value instead of setUserOriginal .

Whenever setUser is called, it triggers createDebugSetter which does its job of checking the environment the code is running in, and returns a modified setter that will call setUserOriginal after the logging process, or will return setUserOriginal as-is.

This is useful because Context updates can trigger many re-renders. This reveals exactly who changed the shared state.

useState

As you saw in the Context provider example above, we can use the same technique in regular components that use React state setters (just as in Context providers). We log and trace the value. It also shows where it was triggered from within the component or application.

import { useState } from "react";
import { useDebugSetter } from "../hooks/useDebugSetter";

export function UseStateExample() {
  const [count, setCountOriginal] = useState(0);
  const [name, setNameOriginal] = useState("React");

  // Wrap setters with debug functionality
  const setCount = useDebugSetter("Counter", setCountOriginal);
  const setName = useDebugSetter("Name", setNameOriginal);

  const handleIncrement = () => {
    setCount(count + 1);
  };

  const handleDecrement = () => {
    setCount(count - 1);
  };

  const handleNameChange = () => {
    setName(name === "React" ? "Vite" : "React");
  };

  return (
    "20px",
        border: "1px solid #ccc",
        borderRadius: "8px",
        margin: "10px",
      }}
    >
      useState Example
      Open the console to see debug logs when state changes.

      "15px" }}>
        
          Count: {count}
        
        
        
      

      "15px" }}>
        
          Name: {name}
        
        
      
    
  );
}

This works exactly like the Context providers example. The only differences are that the component uses the setCount and setName functions out of the box in buttons and related components. Also, unlike the Context provider, this component has local state that can be passed to its child components if needed.

This is ideal for monitoring unforeseen local state changes or loops triggered by effects.

useReducer

React reducers are used to calculate complex logic before updating the state. This can introduce unwanted side effects during the complex phase. createDebugSetter can help in debugging, as shown below:

import { useReducer } from "react";
import { createDebugSetter } from "../utils/createDebugSetter";

interface CounterState {
  count: number;
  step: number;
}

type CounterAction =
  | { type: "increment" }
  | { type: "decrement" }
  | { type: "reset" }
  | { type: "setStep"; step: number };

function counterReducer(
  state: CounterState,
  action: CounterAction
): CounterState {
  switch (action.type) {
    case "increment":
      return { ...state, count: state.count + state.step };
    case "decrement":
      return { ...state, count: state.count - state.step };
    case "reset":
      return { ...state, count: 0 };
    case "setStep":
      return { ...state, step: action.step };
    default:
      return state;
  }
}

export function UseReducerExample() {
  const [state, dispatchOriginal] = useReducer(counterReducer, {
    count: 0,
    step: 1,
  });

  // Wrap dispatch with debug functionality
  const dispatch = createDebugSetter("CounterReducer", dispatchOriginal);

  return (
    "20px",
        border: "1px solid #ccc",
        borderRadius: "8px",
        margin: "10px",
      }}
    >
      useReducer Example
      Open the console to see debug logs for reducer actions.

      "15px" }}>
        
          Count: {state.count}
        
        
          Step: {state.step}
        

        "10px" }}>
          
          
          
          
        
      
    
  );
}

dispatchOriginal, which is the main dispatch function, is replaced with a custom function called dispatch that uses createDebugSetter. When the custom dispatch function is called, it does the job of createDebugSetter and by extension, the job of dispatchOriginal .

This is perfect for logging reducer actions and understanding complex state transitions.

Custom Hooks

Custom hooks are not left out of the equation, as they can use setState in some cases. They’re also capable of running complex logic that could backfire when updating state.

import { useState, useEffect } from "react";
import { useDebugSetter } from "../hooks/useDebugSetter";

// Custom hook that manages a timer
function useTimer(initialSeconds: number = 0) {
  const [seconds, setSecondsOriginal] = useState(initialSeconds);
  const [isRunning, setIsRunningOriginal] = useState(false);

  // Wrap setters with debug functionality
  const setSeconds = useDebugSetter("Timer.seconds", setSecondsOriginal);
  const setIsRunning = useDebugSetter("Timer.isRunning", setIsRunningOriginal);

  useEffect(() => {
    if (!isRunning) return;

    const interval = setInterval(() => {
      setSeconds((prev) => prev + 1);
    }, 1000);

    return () => clearInterval(interval);
  }, [isRunning, setSeconds]);

  const start = () => setIsRunning(true);
  const stop = () => setIsRunning(false);
  const reset = () => {
    setSeconds(0);
    setIsRunning(false);
  };

  return {
    seconds,
    isRunning,
    start,
    stop,
    reset,
  };
}

export function CustomHookExample() {
  const timer = useTimer(0);

  const formatTime = (seconds: number) => {
    const mins = Math.floor(seconds / 60);
    const secs = seconds % 60;
    return `${mins.toString().padStart(2, "0")}:${secs
      .toString()
      .padStart(2, "0")}`;
  };

  return (
    "20px",
        border: "1px solid #ccc",
        borderRadius: "8px",
        margin: "10px",
      }}
    >
      Custom Hook Example
      Open the console to see debug logs for internal hook state changes.

      "15px" }}>
        "24px", fontWeight: "bold" }}>
          {formatTime(timer.seconds)}
        
        
          Status: {timer.isRunning ? "Running" : "Stopped"}
        

        "15px" }}>
          
          
          
        
      
    
  );
}

As shown in previous examples, setSecondsOriginal and setIsRunningOriginal are replaced with setSeconds and setIsRunning. The latter uses the createDebugSetter helper function. This enables console log statements to be printed every second for better visualisation.

Custom hooks often hide multiple internal updates, making it hard to see exactly where each begins.

Best Practices for Using `createDebugSetter`

When using helper functions like createDebugSetter, it’s best to keep in mind why you’re actually using them. For our purpose here, we’re using it to debug a React application. So I’ll share some tips that will help with this debugging process.

Use Clear Labels

Using labels that can say where createDebugSetter was triggered from is a step in the right direction. Detailed labels will help you better understand where and why the issue may be occurring. Also, keep in mind that the createDebugSetter utility function could be used in several places in your application, and improper labelling could make debugging difficult.

Using the name of the component or area that calls it as the label for createDebugSetter can also be a good pointer for clear labelling, as shown below.

// Bad
createDebugSetter("aaa", setUser)
createDebugSetter("1", setUser)

// Good 
createDebugSetter("UserContextProvider", setUser)
createDebugSetter("From UserContextProvider", setUser)
// Too long but can still work
createDebugSetter("From UserContextProvider in user-context.tsx file", setUser)

Use `createDebugSetter` Only in Dev Mode

Using createDebugSetter only in a development environment can prevent many headaches. It’s not a good practice to mistakenly expose or log sensitive data in production. Also, logging in production can cause cluttering.

Use `createDebugSetter` with React DevTools

The createDebugSetter may not be enough for some complex bugs. You can use createDebugSetter and React DevTools for a more powerful/thorough debugging session. Although createDebugSetter cannot be directly integrated with React DevTools, it shows who triggered the update, whereas React DevTools displays what was re-rendered.

Place `createDebugSetter` in `utils`

createDebugSetter is a utility function, as I have mentioned above. This means you should place it in a utils folder so any team member can access and use it when needed across your React application.

Things to Avoid

Avoid using debug setters in production builds. While they are safe, unnecessary logs can slow down debugging tools. Also, sensitive credentials could be logged mistakenly. There are professional tools you can use, such as Sentry, that let you trace errors and debug your app effortlessly.
Don’t conditionally wrap setter functions within components. Perform wrapping outside renders to prevent the creation of new setter identities.
Don’t rely on it to replace proper state architecture. This tool helps identify issues, but doesn’t fix poor state design.
Don’t depend solely on console logs. Use it as part of a broader debugging workflow, not as the only strategy.

Bonus: How to Convert `createDebugSetter` to a Hook

Converting to a Hook

The plain createDebugSetter function works, but it creates a new wrapper function on every render when used inside React components. By converting it into a custom hook with useCallback, we can ensure that the wrapper function maintains a stable reference across re-renders, preventing unnecessary performance overhead and making it safe to use in dependency arrays.

Here’s the hook version:

import { useCallback } from "react";

export function useDebugSetter<T>(
  label: string,
  setState: React.Dispatch>
): React.Dispatch<React.SetStateAction<T>> {
  const debugSetter = useCallback(
    (newValue: React.SetStateAction) => {
      // Only log in development
      if (!import.meta.env.PROD) {
        console.groupCollapsed(
          `%c🔄 State Update: ${label}`,
          "color: #2fa; font-weight: bold;"
        );
        console.log("🆕 New value:", newValue);
        console.trace("📍 Update triggered from:");
        console.groupEnd();
      }

      setState(newValue);
    },
    [label, setState]
  );

  // In production, return the original setter (no wrapping overhead)
  // In development, return the debug wrapper
  return import.meta.env.PROD ? setState : debugSetter;
}

How the Hook Version Works

The core difference between the useDebugSetter hook and createDebugSetter is that the function is wrapped in a useCallback that logs debug information before calling the original setter. Apart from this, all other components of the functions remain the same.

Why the Hook Version is Better

The hook version is superior for component usage because it leverages useCallback memoisation of the debug wrapper. This means the function reference stays the same across renders, avoiding potential re-render cascades when the setter is passed to child components or used in useEffect dependencies.

The plain function, by contrast, generates a brand new wrapper on every render, which can break React's optimisation strategies and cause subtle bugs. In production, both versions simply return the original setter, so there's no performance difference there – but in development, the hook prevents unnecessary work.

When to Use the Hook

Use useDebugSetter whenever you're inside a React component and need to debug state updates. This covers the vast majority of cases: wrapping useState setters, passing debug setters to child components, or including them in effect dependencies.

Only reach for the plain createDebugSetter function when you're working outside React components entirely, such as in utility modules, global stores, or configuration files where hooks can't be used. For day-to-day component debugging, the hook is the right choice.

Conclusion

Debugging React state doesn’t have to be guesswork. With a simple helper, you can instantly see what changed, who changed it, where the change originated, and how your app reached that state – all without touching your production environment.

This small utility function can save hours spent searching through your codebase, making you faster, more precise, and more confident in your React application’s behaviour.

Once you adopt this approach, you’ll never debug state the old way again. 🚀

Why is Debugging Hard? How to Develop an Effective Debugging Mindset

Tapas Adhikary — Wed, 14 Jan 2026 01:23:54 +0000

For years, developers have been told that coding was their primary job. They were encouraged to write clean code, learn tools, understand frameworks, and ship features faster.

But in the actual world of Software Engineering, especially in product-focused companies and customer-facing systems, coding is only half the work. The other half is just as important, and it’s the process called Debugging.

What is Debugging?
Why this Guide?
Why is Debugging Hard?
What is a Mental Model?
The Debugging Mental Model Framework
How to Apply the Debugging Mindset Framework to Code
The Debugging Mindset Framework is Tool Agnostic
What’s Next?
Before We End...

What is Debugging?

Debugging is the practice and methodology developers use to identify issues or problems within a system. Usually, an issue or an unexpected behaviour/problem is known as a bug. The process of debugging, then, is to identify the bug – followed by an attempt to eliminate it or fix it.

Debugging becomes necessary when assumptions break, customers report issues, products behave unexpectedly, or metrics go red. It’s the practice that keeps a software product reliable, teams calm, and users trusting what you build.

Yet, strangely, debugging rarely gets the same respect and attention as coding. It’s often treated as a necessary evil, something that you “figure out along the way” rather than a skill to be learned deliberately.

Why this Guide?

The general neglect of basic debugging skills is catching up with us.

Today, with AI tools, generating code is easier than it has ever been. You can create boilerplate, scaffold components, write functions, establish relations, and even build entire applications in minutes.

But when things go wrong (as they always do), AI doesn’t sit with your product logs, customer complaints, partial failures, and confusing edge cases. Debugging still falls to the human to tackle, and that’s where many devs struggle.

Over the last two decades, I’ve build many products and worked with many developers across experience levels. I’ve noticed a consistent pattern: most debugging failures are not tool failures. They are thinking failures. People jump to fixes too quickly. They start guessing. They panic. They change code without understanding why it broke in the first place.

That’s why I am writing this debugging mindset tutorial. This guide will NOT:

Teach you tools
Share tricks

But it will enable you to think things through when things break.

Alongside this article, I’m also creating a free YouTube course called “Thinking in Debugging”. It’s a practical series on how professional developers approach debugging in JavaScript, React, CSS, and real-world frontend systems. Here is the first session from the course:

In modern software development, writing code gets you started. But debugging is what makes you reliable. Reliability is the most important trait both an engineer and a product must have.

Why is Debugging Hard?

Here’s how most developers debug code:

Something is broken
Let me change the line
Let’s refresh (wishing the error would go away)
Hmm… still broken!
Now, let me add a console.log()
Let me refresh again (Ah, this time it may…)
Ok, looks like this time it worked!

This is reaction-based debugging. It’s like throwing a stone in the dark or finding a needle in a haystack. It feels busy, it sounds productive, but it’s mostly guessing. And guessing doesn’t scale in programming.

This approach and the guessing mindset make debugging hard for developers. The lack of a methodology and solid approach makes many devs feel helpless and frustrated, which makes the process feel much more difficult than coding.

This is why we need a different mental model, a defined skillset to master the art of debugging. Let’s understand what a mental model is and what the debugger’s mindset should be.

What is a Mental Model?

A mental model drives us to think and make decisions. Our brain is at the centre of it. It collects information, processes it, and helps us make those decisions.

When we encounter an issue in programming and we need to find the root cause to fix it, we need to rely on various information and inputs to make logical decisions. We need to create a mental model.

Good debuggers don’t fight bugs. They investigate them. They don’t start with the mindset of “How do I fix this?”. They start with, “Why must this bug exist?” This one question changes everything.

When you ask about the existence of a bug, you go back to the history to collect information about the code, its changes, and its flow. Then, you feed this information through a “mental model” to make decisions that lead you to the fix.

Now, let’s learn about this debugging mental model. This isn’t merely a tool – this is a way of thinking.

The Debugging Mental Model Framework

Before we take a deep dive into the debugging mental model, the key idea is that you never touch the fix until the hypothesis survives reality.

So in this context, what does hypothesis mean?

A Hypothesis is an idea that is suggested as the possible explanation for something but has not yet been found to be true or correct.

With this, let’s get started understanding the debugging mental model framework. It consists of multiple steps or phases that you must go through to find the root cause of a bug and fix it. Once you understand the framework, we’ll apply it to an actual bug in some JavaScript code to make our learning practical.

Let’s Go.

Step 1: Bug Found

The first step is identifying the bug. You or someone else (QA, Customer, and so on) has found that something is wrong. It could be a UI glitch, the wrong output, slow performance, or anything else that is not working as promised and expected.

At this stage, the unexpected behaviour should be documented with enough proof, like logs, screenshots, and steps, for anyone else to reproduce the bug easily. As a developer, don’t panic that something isn’t working as expected. Also, don’t code yet.

Step 2: Define the Facts

Once the bug is found and reported, the next stage is defining or establishing the facts. Facts are things that you can prove, not guesses. For example:

This component renders twice.
This API returns correct data.
This function receives a string, not a number.

Here are a few examples of guesses, but not facts:

React is acting weird.
The API must be slow.
This worked yesterday.
It works on my machine 😁.

Defining facts means writing down only what you can prove. What actually happened? What did the user see? What error was thrown? What data was received? Facts are observable, repeatable, and not an outcome of your emotions.

Defining the facts also empowers you to be aware of the code flow and business cases. So this phase is your opportunity to carefully review the code, requirements, and learn about it, irrespective of who wrote it. Once you know the facts, note them down.

Step 3: Identify Your Assumptions

Every bug is based on a broken assumption. Assumptions often feel harmless because they usually work, until they don’t. Examples:

I assumed this was a number.
I assumed useEffect would run only once.
I assumed the state updates immediately.
I assumed the API always returns data.

Here, the goal is to surface those hidden beliefs. Ask yourself, what must be the actual reason for this code to work as expected? The moment your answer is an assumption, you’re off track. You then recollect, think carefully, stop blaming the system, and start questioning the mental model.

Most bugs are not caused by bad code, but by unverified assumptions.

Step 4: Form a Hypothesis

This is where the actual debugging of the code begins. Once the facts are clear and assumptions are visible, the debugging makes its way forward.

Now you’ll need to form a hypothesis. A hypothesis is a simple cause-and-effect statement: If this assumption is wrong, then the behaviour makes sense. If not, provide a fix.

You may have logs from customers and the best debugging tools from management. But without a good hypothesis, logs become noise and tools become unnecessary. With a good hypothesis, debugging stops being reactive and becomes investigative.

Step 5: Verify the Hypothesis

A hypothesis has no value without meeting reality. You’ll need to verify if your hypothesis is realistic. How do you do that? This is where you use the tools with a purpose. A console.log() statement, a breakpoint, and a network inspection are some of the actions you can perform to answer the question: Is my hypothesis true or false?

If the hypothesis fails, you discard it and move to the next. That’s progress, not failure. On the other hand, if the hypothesis holds, the fix should become clear. You’re no longer making code changes to make the bug disappear suddenly – rather, you’re correcting the root cause.

Putting Everything Together

As we now understand each of the phases, let’s visualise them together and see the bigger picture. I would encourage you to take a pause here and look carefully at each of the boxes below. Now, try processing your understanding from whatever you learned so far about them. Promise yourself that you will apply these to your day-to-day development journey.

Sounds good? Theoretically, it does. But you may have doubts about how all these strategies can work practically. Now, we will apply these to a problem statement and see the practicality of it.

How to Apply the Debugging Mindset Framework to Code

Let’s take an example of a bug that has confused millions of developers across the globe 😀.

Here’s the code:

function fetchUser() {
  let user;

  setTimeout(() => {
    user = { name: "Alex" };
  }, 1000);

  return user;
}

console.log(fetchUser());

The Output: It logs undefined to the browser’s log.

The Bug: I Set the User… Why is it undefined?

Now, let’s apply the debugging mental model framework.

Step 1: Bug Found

Here, the observation is that the function returns undefined. There are no errors in the console. The code looks correct. The scariest bugs are the ones that don’t throw errors.

Step 2: Define the Facts

So, what are the provable facts you see here?

fetchUser() runs.
setTimeout is scheduled.
return user runs.
The user is undefined at return time.

Remember that facts are the things you can prove, not what you believe.

Step 3: Identify Your Assumptions

Now, ask yourself, “What am I assuming here?”. Here are a few common beginner assumptions for this case:

JavaScript runs line-by-line synchronously.
The setTimeout blocks execution.
Code waits for 1 second.
The user variable is assigned before the return from the function.

Most async bugs come from the assumptions about execution time.

Step 4: Form a Hypothesis

Next, we need to form a hypothesis to introduce structured thinking. The function returns undefined. If our assumptions were right, the user variable should have the assigned value. It seems that there’s something wrong with the assumptions.

Does the setTimeout really block execution?
Does the code really wait for 1 second?
If JavaScript doesn’t wait for setTimeout, then return user will execute before the assignment. This is how the user variable could be undefined. It seems like we’re dealing with the Async operation here. This is the aha moment – that’s our hypothesis.

We aren’t fixing anything yet. We’re predicting behaviour.

Step 5: Verify or Kill the Hypothesis

Now, we need to verify our hypothesis. Let’s use console.log() for that. We’ll add two logs, one inside the setTimeout before assigning the user variable value, and the other just before returning the user from the function.

function fetchUser() {
  let user;

  setTimeout(() => {
    console.log("Inside timeout");
    user = { name: "Alex" };
  }, 1000);

  console.log("Before return:", user);
  return user;
}

console.log(fetchUser());

Execute the code, and here are the observations:

“Before return:” logs first.
“Inside timeout” logs later.

This means that our hypothesis survives the reality. We proved that debugging is not guessing – it’s about ordering the execution time correctly in our heads.

Step 6: Fix With the Proof

Now our fix becomes obvious, not a guess or magic. If we want the user’s value to be logged instead of undefined, we can fix it in multiple ways, like using a callback function or a promise object.

With a callback: Define a callback function that gets called after the time expires. The callback function takes the value as a parameter and assign to the user before logging it to the console.

function fetchUser(callback) {
  setTimeout(() => {
    callback({ name: "Alex" });
  }, 1000);
}

fetchUser(user => console.log(user));

With Promise Object: Alternatively, we can use the Promise object. The promise resolves after 1 second, and we log the user details with the help of the .then() handler method.

function fetchUser() {
  return new Promise(resolve => {
    setTimeout(() => {
      resolve({ name: "Alex" });
    }, 1000);
  });
}

fetchUser().then(user => console.log(user));

Let’s now visualise all the stages together with respect to the problem we discussed:

The Debugging Mindset Framework is Tool Agnostic

Note that the debugging mental model teaches you how to observe, think through, and justify your beliefs to find the root cause of the issue. Once confirmed, you need to use your programming language knowledge and coding skills to implement the fix. The debugging mindset or mental model framework itself is technology and tool agnostic.

It doesn’t belong to JavaScript, React, Python, or any specific tool. The need for facts, assumptions, hypotheses, and verification exists in every technology stack. Today, you might be debugging a React component. Tomorrow it could be CSS layout, backend logic, or a memory leak. The same thinking applies. This is why experienced developers adapt more quickly to new programming languages, frameworks, or tools. They carry this mindset with them.

What’s Next?

Technologies evolve, frameworks come and go, but the debugging mental model framework remains constant. So focus on that. Have a mindset to own up to the issues you’ve found in a software product. No development is bug-free. You create bugs sometimes, so you should just proudly own them. And now, you should have the mindset to confidently fix them.

I would like to invite you to join my free course Thinking in Debugging. In it, we won’t only set up this mental model, but also realise it by debugging JavaScript, React, and CSS with DevTools, Debugger, and Profiler.

Before We End...

That’s all! I hope you found this article insightful.

Let’s connect:

Subscribe to my YouTube Channel.
Check out my courses, 40 Days of JavaScript and 15 Days of React Design Patterns.
Follow on LinkedIn if you don't want to miss the daily dose of up-skilling tips.
Join my Discord Server, and let’s learn together.
Follow my work on GitHub.

See you soon with my next article. Until then, please take care of yourself and keep learning.

How to Make Bluetooth on Android More Reliable

Nikheel Vishwas Savant — Wed, 03 Sep 2025 07:00:00 +0000

You may have had this happen before: your wireless earbuds connect perfectly one day, and the next they act like they’ve never met your phone. Or your smartwatch drops off in the middle of a run. Bluetooth is amazing when it works, but maddening when it doesn’t.

I work as a Bluetooth software engineer on wearable devices like smart-glasses, and I’ve spent more time than I’d like to admit chasing down why these things break.

In this article, I’ll give you a peek behind the curtain: how Android’s Bluetooth stack actually works, why it sometimes feels unpredictable, and what you can do as a developer to make your apps or system more reliable.

Bluetooth in Plain English

At its core, Bluetooth is just a conversation between two devices. But it isn’t one simple line of communication – it’s multiple layers stacked on top of each other.

The radio (Controller): Sends and receives the actual signals over the air medium.
The software brain (Host stack): Decides whom to talk to and how, as well as if it wants to.
Profiles: Define the purpose of the conversation – like streaming music or syncing health data.
Protocols: Define how to talk to the other device.

There are two big “flavors” of bluetooth:

Classic (BR/EDR): Used for things like headphones and car kits. Can lift more weight.
Low Energy (LE): Used for fitness bands, beacons, and most wearables. Can sustain longer.

Most modern gadgets use both at once. That’s powerful, but it also opens the door for more things to go wrong.

Why Android Adds Its Own Quirks

On Android, Bluetooth isn’t just one neat package. It’s a chain of moving parts:

Your app calls BluetoothAdapter.
Those go into system services like AdapterService.
Then into native code through JNI (Java Native Interface).
Then into the chip vendor’s Bluetooth stack.
Finally, it hits the radio hardware.

Every phone maker ships a slightly different Bluetooth chip and firmware. That means the exact same Bluetooth app might behave differently on a Samsung, a Pixel, or any other budget phone running Android.

The Real Problems Behind “It Just Disconnected”

Here are a few of the common headaches I see, explained simply:

Bonding issues (the “lost keys” problem)

When two Bluetooth devices pair, they exchange encryption keys (link keys for Classic, Long Term Keys for LE) and store them in non-volatile memory. These keys are what let the devices recognize each other later and reconnect securely without asking the user again.

A “mismatched memory” problem happens when one device’s stored keys don’t match the other’s anymore. This can be caused by:

A firmware update or OS upgrade that wipes or regenerates keys.
A factory reset or “forget device” on one side but not the other.
Keys being corrupted or evicted by the system to free up storage.

From the user’s perspective, the device may still look paired (shows up in the Bluetooth menu), but connections mysteriously fail with errors like “Authentication Failed” or “Insufficient Encryption.” The only cure is usually to delete the device on both ends and re-pair, which feels ridiculous to non-technical users.

Timing mismatches

Bluetooth devices don’t just chat whenever they want, they agree on a connection interval – essentially a schedule for when each side will “wake up” and exchange packets. Think of it as two people agreeing to meet every 30 minutes at a café.

A mismatch happens when:

The two sides negotiate different intervals but don’t fully agree (for example, one thinks it’s 30ms, the other 50ms).
One side’s firmware update or configuration change alters its timing policy.
Radio conditions cause one side to miss multiple scheduled check-ins, drifting the clocks apart.
Power-saving logic (like a phone going into Doze mode) silently stretches out the interval.

This explains why a connection might work fine at first but start failing later: the devices initially synced on an interval, but then one side’s policy or behavior shifted. From the user’s perspective, it looks like audio stuttering, laggy input (on game controllers), or random disconnects after “it was working fine before.”

Unexpected disconnections

When a Bluetooth link ends, the radio layer (the controller) and the higher-level OS stack (the host) are supposed to exchange clear signals. The controller sends an HCI Disconnection Complete event (basically: “Goodbye, we’re done”). And the host should then update its internal state, clean up the GATT/ACL session, and be ready for reconnection.

But in practice, this doesn’t always line up:

Sometimes the controller says goodbye cleanly, but the host stack doesn’t update its state properly. The app still “thinks” the connection is active, so reconnect attempts silently fail.
Some platforms aggressively cache connection state (especially iOS). If the OS believes the connection is still valid, it won’t trigger a new connection attempt until you toggle Bluetooth or reboot.
A race condition can occur if the disconnection event happens while another operation (for example, service discovery, bonding, or encryption setup) is in flight. The OS may get confused about what state the device is really in.
On some devices, a fast reconnect attempt after a clean disconnection collides with internal cooldown timers. The controller ignores it, leaving the app waiting.

From the user’s perspective, the device looks “stuck.” The only way to recover is to toggle Bluetooth, restart the app, or power cycle the accessory, even though technically nothing “failed.”

How Developers Can Do Better

If you’re building a Bluetooth app, here are a few habits that save a lot of pain:

Check for bonded devices first

One of the most common causes of failed connections is mismatched bonding information: the phone and the accessory no longer share the same encryption keys. Even if the device appears in the UI, the OS may have lost its keys.

Before attempting a connection, always query the system’s bonded device list with BluetoothAdapter.getBondedDevices(). For example:

if (adapter.getBondedDevices().contains(targetDevice)) {
    targetDevice.connectGatt(context, false, gattCallback);
} else {
    showToast("Please re-pair this device to restore the connection.");
}

This ensures you only attempt secure connects to devices the OS still trusts. If the target device isn’t in the bonded list, you can give the user a clear instruction (“Please re-pair this device”) instead of leaving them with confusing connection errors.

Handle callbacks carefully

Another subtle pitfall is assuming that a STATE_CONNECTED event means a connection was successful. In reality, onConnectionStateChange() can report a connected state even when the underlying operation failed, the real result is in the status argument. To avoid chasing phantom connections, always check both status and newState:

if (status == BluetoothGatt.GATT_SUCCESS &&
    newState == BluetoothProfile.STATE_CONNECTED) {
    gatt.discoverServices();
} else {
    gatt.close();
}

This pattern prevents you from attempting service discovery on a dead connection and ensures stale sessions are closed promptly, leaving the stack ready for a clean retry.

Expect failures

Bluetooth connections fail all the time in the real world – devices drift out of range, interference spikes in the 2.4 GHz band, or the radio is simply busy. The worst thing an app can do is retry instantly in a tight loop, which drains the battery and makes the stack unstable.

A better approach is to implement exponential backoff like this:

long delay = (long) Math.min(250 * Math.pow(2, attempt), 30000);
new Handler(Looper.getMainLooper()).postDelayed(connectAction, delay);

This means your first retry happens quickly (~250 ms), but subsequent retries slow down (500 ms, 1 s, 2 s…), capped at a reasonable maximum. Backoff makes your app resilient without overwhelming the radio or the OS.

Use the right tools

Without visibility into what’s happening under the hood, connection problems look random. Tools like nRF Connect let you interactively scan, connect, and run GATT operations against your device, while Android’s Bluetooth HCI snoop log reveals the actual packets being exchanged. For example:

Settings.Secure.putInt(context.getContentResolver(), "bluetooth_hci_log", 1);

Once enabled, you can capture a logcat trace and confirm whether a failure is due to missing keys (Insufficient Authentication), a timing mismatch, or interference. Using these tools not only helps you debug your app, it also proves whether the issue lies in your code, the OS, or the accessory firmware.

Bigger Lessons

Working with Bluetooth taught me lessons that apply to engineering in general:

Wireless is never perfect, so always build with recovery in mind.
Logs and metrics aren’t optional. They’re your map through the chaos.
The simplest solution usually survives best in the messy real world.

Conclusion

Bluetooth is messy because it’s a chain of hardware, firmware, and software all trying to cooperate. On Android, the variety of chips and vendors makes it even trickier.

But that doesn’t mean you’re helpless. By understanding how the layers work and designing your apps with retries, checks, and proper logging, you can make Bluetooth feel a lot less “weird” for your users.

The next time your earbuds misbehave, you’ll know – it’s not you. It’s just Bluetooth being Bluetooth.

⚡ This is the first of a number of articles I’m going to write on Bluetooth development. In the next one, we’ll dive deeper into how to build a secure Bluetooth Low Energy (BLE) GATT client and server on Android. Stay tuned!

How to Debug Kubernetes Pods with Traceloop: A Complete Beginner's Guide

Opaluwa Emidowojo — Fri, 29 Aug 2025 16:09:24 +0000

Debugging Kubernetes pods can feel like detective work. Your app crashes, and you're left wondering what happened in those critical moments leading up to failure. Traditional kubectl commands show you logs and statuses, but they can't tell you exactly what your application was doing at the system level when things went wrong.

What if you had a flight recorder for your applications, something that captures every system call in real-time, so you can "rewind" and see the exact sequence of events that led to a crash? That's what Traceloop does. It continuously traces system calls in your pods, giving you a detailed replay of what happened before, during, and after issues occur.

In this guide, you’ll learn how to use Traceloop's system call tracing to debug pod issues that would otherwise be nearly impossible to diagnose.

Prerequisites

Before we begin, here are some prerequisites – things you’ll need to know and have:

Basic Kubernetes concepts: Understanding of pods, deployments, services, and namespaces
kubectl fundamentals: Comfortable with commands like kubectl get, kubectl describe, kubectl logs, and kubectl exec
Container basics: Understanding how containerized applications work
Basic Linux concepts: Understanding of processes and system calls (helpful, but we'll explain as we go)

Technical Requirements

Kubernetes cluster access: Local (minikube, kind, Docker Desktop) or cloud-based cluster
kubectl installed and configured to connect to your cluster
Sufficient permissions (cluster admin or equivalent RBAC) to:
- Install and run eBPF-based tools (Traceloop uses eBPF)
- Create/modify pods and deployments
- Access pod logs and system-level data
Linux-based Kubernetes nodes: Most clusters already run on Linux.

System Requirements

Extended Berkeley Packet Filter (eBPF) support: Used for tracing and monitoring at the kernel level. Kernel version 5.10+ recommended.
Sufficient cluster resources: Traceloop runs alongside your applications

What is Traceloop?
How Traceloop Works
How to Set Up Traceloop
Your First Trace: Hands-On Tutorial
Step-by-Step Debugging Walkthrough
Real-World Debugging Scenarios
Best Practices
Conclusion

What is Traceloop?

Traceloop is a system call tracing and observability tool that works across containerized environments, from Docker containers running locally to pods in production Kubernetes clusters. But before we discuss what that means, let's talk about why system calls matter for debugging.

Every time your application does anything (like opening a file, making a network request, allocating memory, or crashing), it has to interact with the operating system through system calls. These are the fundamental building blocks of how any program interacts with the world around it.

Here's where traditional debugging falls short: when your container crashes, the logs might tell you "segmentation fault" or "out of memory," but they don't tell you the sequence of events that led there. Did the application try to access a file that didn't exist? Was it making network calls that failed? Did it run out of file descriptors?

Traceloop captures this missing piece. It sits at the kernel level using eBPF technology, recording every system call your application makes in real-time. Think of it as installing a dashcam in your application. It's always recording with minimal resources, and when something goes wrong, you have the footage.

Strace is another popular debugging tool – but it requires you to know that there's a problem first. With Traceloop, we can conveniently run it continuously in the background with minimal overhead. If your container crashes at 3am, you can immediately "rewind the tape" and see exactly what system calls happened leading up to the crash.

This helps debug intermittent issues that happen randomly in production but never when you are watching. Because Traceloop is always recording, you finally have visibility into what your application was doing when these mysterious failures occur.

How Traceloop Works

Now that you understand what Traceloop does, let's look under the hood at how it captures and processes system calls in your containerized environments.

The Technical Foundation

Traceloop is built on eBPF, a technology that allows programs to run safely in the Linux kernel without changing kernel code. Think of eBPF as a way to install "hooks" directly into the kernel that can observe everything happening on your system with minimal performance impact.

Unlike traditional monitoring tools that work from userspace, eBPF programs run in kernel space, giving them access to system calls as they happen, without relying on the application logging appropriate error messages. This is why Traceloop can capture events that never make it to application logs, like failed system calls or crashes that happen before the application can write anything.

The Flight Recorder Architecture

Traceloop uses eBPF maps as an overwriteable ring buffer. Imagine a tape recorder that continuously records over itself. It's always capturing system calls, but it only keeps the most recent data in memory. When something goes wrong, the recording automatically preserves what happened leading up to the incident, just like an airplane's flight recorder after a crash.

This approach solves the production debugging problem: you don't need to predict when issues will happen or attach debuggers after the fact. The recording is always running, waiting for you to need it.

System Call Capture Flow

Here's how Traceloop captures and processes system calls across your Kubernetes environment:

Application pods generate system calls through normal operation – opening files, making network connections, allocating memory.
eBPF probes (also called hooks) intercept these system calls at the kernel level before they're processed.
Traceloop recorder captures the events, buffers them, and adds container context using Inspektor Gadget enrichment (pod name, namespace, container ID).
Output stream formats the data and makes it available for analysis in real-time or after an incident.
Traceloop user views and analyzes the captured trace to diagnose the root cause of issues.

Below is a visual representation of the flow. The key advantage is that Traceloop sees everything your application does, even actions that fail silently or happen too quickly for traditional logging to catch. This gives you complete visibility into your application's interaction with the operating system.

Container Isolation and Context

One of Traceloop's strengths is understanding containerized environments. It doesn't just capture raw system calls – it adds context about which pod, container, and namespace generated each call. This means you can trace specific applications without getting overwhelmed by system calls from other containers running on the same node.

This container awareness makes Traceloop particularly powerful in Kubernetes environments where you might have dozens of pods running on a single node, but you only care about debugging one specific application.

How to Set Up Traceloop

Before we can start tracing system calls, we need to set up Traceloop in your Kubernetes environment. Traceloop is part of the Inspektor Gadget ecosystem, which provides flexibility in how you use it.

Installation Overview

This setup:

Deploys Inspektor Gadget components to all worker nodes
Eliminates the download and initialization overhead on each use, as components are pre-loaded and ready
Eliminates the need to reinstall or reconfigure for each debugging session – just run your traces immediately
Requires cluster admin permissions
Works best for teams doing regular debugging

Installation Requirements

First, ensure your cluster meets the requirements:

Kubernetes cluster with Linux nodes
eBPF support
kubectl installed and configured
Cluster admin permissions

Install kubectl gadget

The recommended way is using krew (kubectl plugin manager):

# Install krew if you don't have it
curl -fsSLO "https://github.com/kubernetes-sigs/krew/releases/latest/download/krew-linux_amd64.tar.gz"
tar zxvf krew-linux_amd64.tar.gz
./krew-linux_amd64 install krew
export PATH="${KREW_ROOT:-$HOME/.krew}/bin:$PATH"

# Install kubectl gadget
kubectl krew install gadget

Alternatively, you can install directly:

# For Linux/macOS
curl -sL https://github.com/inspektor-gadget/inspektor-gadget/releases/latest/download/kubectl-gadget-linux-amd64.tar.gz | sudo tar -C /usr/local/bin -xzf - kubectl-gadget

# Verify installation
kubectl gadget version

Deploy Inspektor Gadget to Your Cluster

Deploy the Inspektor Gadget components to your cluster:

kubectl gadget deploy

This installs the necessary DaemonSets and RBAC configurations that allow gadgets like Traceloop to run on your cluster nodes.

Alternatively, you can also deploy using Helm.

Verify Installation

Check that the gadget pods are running:

kubectl get pods -n gadget

You should see gadget pods running on each node in your cluster.

Your First Trace: Hands-On Tutorial

Now let's capture our first system call trace. We'll create a simple scenario and watch what happens at the system level.

Setting Up the Test Environment

First, create a dedicated namespace for our tracing experiments:

kubectl create ns test-traceloop-ns

Expected output:

namespace/test-traceloop-ns created

Next, create a simple pod that we can interact with:

kubectl run -n test-traceloop-ns --image busybox test-traceloop-pod --command -- sleep inf

Expected output:

pod/test-traceloop-pod created

This creates a BusyBox container that sleeps indefinitely, giving us a stable target for tracing.

Starting Your First Trace

Next, start tracing system calls for our test pod:

kubectl gadget run traceloop:latest --namespace test-traceloop-ns

This command starts the flight recorder. You'll see column headers showing what information Traceloop captures:

K8S.NODE    K8S.NAMESPACE    K8S.PODNAME    K8S.CONTAINERNAME    CPU    PID    COMM    SYSCALL    PARAMETERS    RET

The trace is now running in the background, continuously recording system calls from our pod.

Generating System Calls

With the trace running, let's generate some activity. In a new terminal window, run a command inside your test pod:

kubectl exec -ti -n test-traceloop-ns test-traceloop-pod -- /bin/sh

Once inside the container, run some basic commands:

ls /
echo "Hello World" > /tmp/test.txt
cat /tmp/test.txt

Collecting the Trace

Back in your original terminal where Traceloop is running, press Ctrl+C to stop the recording and see the captured system calls.

You'll see output similar to this:

K8S.NODE            K8S.NAMESPACE        K8S.PODNAME          K8S.CONTAINERNAME    CPU  PID    COMM  SYSCALL      PARAMETERS                   RET
minikube-docker     test-traceloop-ns    test-traceloop-pod   test-traceloop-pod   2    95419  ls    openat       dfd=-100, filename="/lib"    3
minikube-docker     test-traceloop-ns    test-traceloop-pod   test-traceloop-pod   2    95419  ls    getdents64   fd=3, dirent=0x...          201
minikube-docker     test-traceloop-ns    test-traceloop-pod   test-traceloop-pod   2    95419  ls    write        fd=1, buf="bin dev etc..."   201
minikube-docker     test-traceloop-ns    test-traceloop-pod   test-traceloop-pod   2    95419  ls    exit_group   error_code=0                 0

Understanding Your First Trace

Let's break down what we're seeing:

K8S.PODNAME: Which pod generated these system calls
PID: Process ID of the command that ran
COMM: The command name (ls, echo, cat)
SYSCALL: The actual system call made (openat, write, exit_group)
PARAMETERS: Arguments passed to the system call
RET: Return value (0 usually means success)

This trace shows the ls command opening the /lib directory, reading directory entries, writing the output to stdout, and exiting successfully.

Clean Up

Remove the test resources:

kubectl delete pod test-traceloop-pod -n test-traceloop-ns
kubectl delete ns test-traceloop-ns

You can now see exactly what your applications are doing at the kernel level, something that traditional logs and kubectl commands can't show you.

Let's try this with an application that crashes.

Step-by-Step Debugging Walkthrough

Now that you know how to capture traces, let's take a look at a real debugging scenario. We'll create an application that crashes and use Traceloop to uncover the root cause. Something that would be nearly impossible with traditional kubectl debugging.

The Scenario: A Mysterious Crash

Let's create a Python application that has a subtle bug. It tries to write to a file it doesn't have permission to access, then crashes. This mimics real-world scenarios where applications fail due to permission issues, missing files, or resource constraints.

Setting Up the Problematic Application

First, we’ll create a new namespace for our debugging exercise:

kubectl create ns debug-traceloop-ns

Now, let's create a pod with an application that will crash:

kubectl run -n debug-traceloop-ns crash-app --image=python:3.9-slim --restart=Never -- python3 -c "
import time
import os
print('App starting...')
time.sleep(5)
print('Trying to write to restricted file...')
try:
    with open('/etc/passwd', 'w') as f:
        f.write('malicious content')
except Exception as e:
    print(f'Error: {e}')
    exit(1)
"

This creates a pod that will:

Start successfully
Try to write to /etc/passwd (a restricted system file)
Fail and crash with exit code 1

Starting the Trace Before the Crash

Here's the key difference from traditional debugging. We start tracing before we know there's a problem. In a real scenario, you'd have Traceloop running continuously.

kubectl gadget run traceloop:latest --namespace debug-traceloop-ns

The trace starts recording immediately. You'll see the column headers, and the flight recorder is now capturing every system call.

Observing the Application Behavior

In another terminal, check the pod status:

kubectl get pods -n debug-traceloop-ns -w

You'll see the pod go through these states:

Pending → Running → Error → CrashLoopBackOff

Traditional debugging would show you:

kubectl logs -n debug-traceloop-ns crash-app

Output:

App starting...
Trying to write to restricted file...
Error: [Errno 13] Permission denied: '/etc/passwd'

But this doesn't tell you exactly what the application tried to do at the system level.

Collecting and Analyzing the Trace

Back in your Traceloop terminal, press Ctrl+C to stop the recording. You'll see system calls like this:

K8S.NODE        K8S.NAMESPACE      K8S.PODNAME  COMM    SYSCALL    PARAMETERS                           RET
minikube-docker debug-traceloop-ns crash-app    python3 openat     dfd=-100, filename="/etc/passwd"    -13
minikube-docker debug-traceloop-ns crash-app    python3 write      fd=3, buf="App starting..."         16
minikube-docker debug-traceloop-ns crash-app    python3 openat     dfd=-100, filename="/etc/passwd"    -13
minikube-docker debug-traceloop-ns crash-app    python3 exit_group error_code=1                        0

Reading the System Call Story

The trace reveals the exact sequence of events:

openat filename="/etc/passwd" RET=-13: The application tried to open /etc/passwd for writing
- Return code -13 = EACCES (Permission denied)
write buf="App starting...": Normal logging output (successful)
openat filename="/etc/passwd" RET=-13: Second attempt to open the restricted file (still denied)
exit_group error_code=1: Application exits with error code 1

What Traceloop Revealed

Traditional debugging told us "Permission denied" but Traceloop shows us:

Exactly which file the application tried to access
When the permission denial happened in the execution flow
How many times it tried (twice in this case)
The exact system call that failed (openat)

Real-World Applications

This same approach works for debugging:

File not found errors: See exactly which files your app is looking for
Network connection failures: Observe failed connect() system calls with specific addresses
Memory issues: Watch mmap() and brk() calls that fail
Container startup problems: See which system calls fail during initialization

Clean Up

Remove the test resources:

kubectl delete pod crash-app -n debug-traceloop-ns
kubectl delete ns debug-traceloop-ns

Key Takeaway

Traditional Kubernetes debugging shows you what went wrong after it happened. Traceloop's continuous recording shows you exactly how it went wrong at the system level. This level of detail is invaluable for debugging complex production issues where the logs don't tell the full story.

Real-World Debugging Scenarios

Now that you understand the fundamentals, let's explore common production issues and how Traceloop helps diagnose them. These scenarios mirror real problems you'll encounter in Kubernetes environments.

Scenario 1: Container Startup Failures

The problem: Your pod gets stuck in CrashLoopBackOff with unhelpful logs.

Traditional kubectl commands show limited information:

kubectl describe pod failing-app
# Events: Back-off restarting failed container

kubectl logs failing-app
# (Empty or minimal output)

System calls show the application tried to:

Access configuration files that don't exist
Connect to services that aren't available
Write to directories without proper permissions

Key system calls to watch:

openat with -2 return (file not found)
connect with -111 return (connection refused)
access with -13 return (permission denied)

Scenario 2: Memory and Resource Issues

The problem: Application performance degrades or gets OOMKilled.

What Traceloop shows:

mmap calls failing (memory allocation issues)
brk system calls indicating heap growth
File descriptor exhaustion through failed openat calls
Excessive write calls indicating memory pressure

Example pattern:

SYSCALL    PARAMETERS           RET
mmap       length=1048576       -12  # ENOMEM - out of memory
brk        brk=0x55555557d000   0    # Heap expansion
openat     filename="/tmp/..."   -24  # EMFILE - too many open files

Scenario 3: Network Connectivity Problems

The problem: Service-to-service communication fails intermittently.

Traditional debugging limitations:

Application logs show "connection timeout"
Network policies seem correct
DNS resolution appears to work

What Traceloop reveals:

Exact IP addresses and ports being attempted
DNS resolution patterns through openat on /etc/resolv.conf
Failed connect calls with specific error codes
Socket creation and binding issues

Key indicators:

SYSCALL    PARAMETERS                    RET
socket     family=AF_INET, type=SOCK     3
connect    fd=3, addr=10.96.0.1:443     -110  # ETIMEDOUT
close      fd=3                         0

Scenario 4: Configuration and Secret Issues

The problem: Application can't access mounted secrets or config maps.

What system calls reveal:

File access patterns for mounted volumes
Permission checks on secret files
Configuration file parsing attempts

Common patterns:

Multiple openat attempts on different config file paths
access calls checking file permissions before opening
Failed reads from mounted secret volumes

Scenario 5: Performance Bottlenecks

The problem: Application response times are slow without obvious cause.

Traceloop analysis:

Excessive fsync calls (disk I/O bottlenecks)
Many futex calls (lock contention)
Frequent recvfrom timeouts (network issues)
Repeated file system operations

Performance indicators:

SYSCALL     FREQUENCY    ISSUE
fsync       High         Disk I/O bottleneck
futex       Excessive    Lock contention
poll        Many         Waiting for I/O
recvfrom    Timeouts     Network delays

Best Practices

When to Use Traceloop

Traceloop is most useful when you’re dealing with the kinds of problems that are notoriously difficult to pin down. If you’ve ever struggled with debugging intermittent crashes that don’t happen on demand, or run into confusing permission and access issues, this is where it works best.

It also helps uncover performance bottlenecks at the system level and provides visibility into application behavior during tricky startup failures. Another common use case is diagnosing network connectivity problems between pods, where other tools usually can't help

Of course, not every problem requires system call tracing. For application-level issues, logs and APM tools are more effective. Cluster-level concerns are often better handled with kubectl describe or by looking at events, and if you’re primarily monitoring resources, standard metrics and dashboards show you what's happening.

Performance Considerations

Like any tracing tool, Traceloop adds some overhead, but it keeps the overhead low. You can keep it efficient by narrowing the scope of your traces. For example, filtering by namespace with --namespace specific-ns, or targeting specific pods using --podname target-pod. In high-traffic environments, it’s best to run traces for shorter periods, and node-specific tracing can further isolate debugging when you don’t want to instrument the entire cluster.

In most cases, Traceloop uses very little CPU and memory, thanks to its eBPF-based approach. This makes it lighter than traditional tools like strace. The actual cost depends on the volume of system calls being recorded, so it’s a good practice to monitor resource usage in your own environment to confirm it’s operating within acceptable limits.

Integration with Your Workflow

Traceloop works well in dev and production workflows. In development, it’s a powerful way to understand how your application interacts with the system. You can use it to confirm that your app handles edge cases correctly, or to validate permission and resource configurations before promoting workloads into production.

In production environments, you can deploy it in different ways. Depending on how much overhead you're okay with, some teams run it continuously on a small subset of nodes, while others use it only when traditional debugging methods don’t provide enough insight. Pairing Traceloop with your existing monitoring and logging stack can give you a much more complete picture of system behavior.

It also helps with teamwork. Sharing trace outputs makes it easier for teams to reason about complex issues together. The data it provides can guide improvements in error handling and logging, and documenting common system call patterns can help onboard new developers more quickly.

Security Considerations

Because Traceloop records low-level system activity, you need to be mindful of what it captures.

What Traceloop Can See:

System call parameters (such as filenames and network addresses)
Process information and command arguments
File access patterns and permissions

Privacy Measures:

Limit trace duration to minimize data collection
Use namespace isolation to avoid capturing unrelated workloads
Apply data retention policies for trace outputs
Watch for sensitive information in file paths or system call parameters

Conclusion

Traceloop doesn’t just tell you something went wrong – it shows you how. By recording every system call in real time, it turns mysterious Kubernetes failures into solvable problems. Whether the issue happened seconds ago or in the middle of the night, the tool gives you the ability to rewind, inspect, and respond with confidence.

When to Use It

Keep in mind that Traceloop complements your existing debugging toolkit rather than replacing it. Reach for it when logs don’t tell the whole story, when intermittent problems are hiding in the shadows, when kubectl commands leave you guessing, or when you need to see how your application is really interacting with the system.

Once you’re comfortable with Traceloop, you can add more tools. Inspektor Gadget offers other tools for network, security, and performance debugging that pair well with Traceloop. Integrating it into your incident response workflow, sharing insights across your team, and even considering continuous tracing for critical workloads are good things to try next.

The next time you run into a stubborn Kubernetes pod failure, you won’t be stuck speculating. With Traceloop, you can “rewind the tape” and see exactly what happened. System call tracing may sound complex at first, but in practice, it’s one of the most powerful ways to truly understand how applications behave in containerized environments.

PS: Have any questions about Traceloop or want to share your debugging challenges? The Inspektor Gadget team and community hang out in the #inspektor-gadget channel on Kubernetes Slack. It's a great place to get help from the engineers who built these tools, share experiences, and maybe even contribute to making the ecosystem even better.

You can also connect with me on LinkedIn if you’d like to stay in touch. If you made it to the end of this tutorial, thanks for reading!

Learn Embedded Systems Firmware Basics – A Handbook for Developers

Soham Banerjee — Mon, 23 Jun 2025 21:21:32 +0000

Have you ever wondered how your fridge knows when to cool, or how a coffee machine knows when to stop pouring? Behind the scenes, these devices are powered by embedded systems – small, dedicated computers designed to perform specific tasks reliably and efficiently.

An embedded system typically goes through a simple but powerful cycle:

Sense – Gather information from the environment using sensors.
Process – Use software logic to decide what to do with the data.
Act – Trigger a response, like turning on a motor or lighting an LED.

Each project begins with a use case – a specific goal like brewing coffee or controlling a car’s fuel injection. From that, engineers define system requirements, which are split into:

Hardware (for example, microcontrollers, sensors, actuators)
Software (what we call embedded software)

This handbook focuses on the software side of embedded systems: how we write code to make embedded systems intelligent. Embedded software runs on resource-constrained devices like microcontrollers, which may have just a few kilobytes of memory. The software might need to be highly efficient, reliable, and often capable of working in real-time.

But embedded software isn't just about writing code – it’s also about understanding:

How hardware works
How to manage memory and power
How to handle timing and communication
How to build robust, fail-safe systems

While embedded systems development isn’t typically research-focused in most industry roles, it demands a broad skill set, from low-level programming to system-level design. What makes this field especially exciting is how it brings together diverse domains like machine learning, digital signal processing (DSP), and control systems, all of which can be applied directly in real-world devices.

In this article, I’ll give you:

A high-level overview of what embedded software involves
Key concepts every developer should know
A tour of commonly used tools and frameworks
Resources to help you learn and understand basics.

Whether you're just curious or planning a career in embedded systems, this guide is your launchpad.

HW Layer: Microcontroller
Firmware Design and Tools
Tools and Concepts for Embedded Development
Bare Metal, RTOS, and Embedded Operating Systems
Designing Drivers for Embedded Systems
Security in Embedded Systems
Debugging and Forensics in Embedded Systems
Automation and Testing in Embedded Systems
Where to Go from Here

This article offers a broad overview of embedded firmware development, but it doesn’t cover every aspect, particularly advanced software architecture frameworks or comprehensive lists of open source software and tools. Where appropriate, I have included external resources that were valuable in expanding my own understanding.

Prerequisites

You don’t need to be an expert to follow this guide, but some prior knowledge will help you get the most out of it:

Basic C or C++ programming**:** Familiarity with functions, pointers, and memory concepts is helpful.
Computer architecture fundamentals**:** Understanding what a CPU does, how memory works, and basic instruction execution will make embedded concepts clearer.
Electronics basics (optional): Knowing how sensors, resistors, or microcontrollers interact at a circuit level is useful but not mandatory.
Comfort with the command line**:** Especially for working with build systems, compilers, and flashing tools.

This guide is ideal for students, engineers, or hobbyists looking to deepen their understanding of how software interacts with hardware in real-world systems.

With that, let’s start from the ground up, hardware. Throughout this guide, most examples will reference ARM Cortex-M microcontrollers, as they are among the most commonly used in the embedded world.

HW Layer: Microcontroller

One of the most important knowledge blocks in embedded firmware development is understanding how a microcontroller (MCU) works and how it connects to sensors, actuators, and other microcontrollers.

If you’re familiar with basic computer architecture (like instruction sets and memory organization), that knowledge translates well to embedded systems. In fact, Computer System Organization, often taught in computer science and electrical engineering programs, is a great foundation for understanding microcontrollers.

What is a Microcontroller?

A microcontroller is a compact computing unit that includes:

A CPU (Central Processing Unit or Microprocessor)
Memory (Flash and RAM)
Peripherals (for I/O, timers, communication, and so on)

In essence, it's a tiny computer-on-a-chip, optimized for specific control tasks like reading sensors or driving motors.

By contrast, a microprocessor is just the CPU. It requires external memory and peripherals to function. Microcontrollers are self-contained and better suited for embedded applications.

For example, this reference manual for the STM32F4 series (from STMicroelectronics) provides detailed documentation on not just the CPU but each peripheral’s functionality and the register map.

Instruction Set Architecture (ISA)

A microprocessor executes a series of instructions defined by its Instruction Set Architecture (ISA). ISA as defined by ARM is a part of the abstract model of a computer that defines how the CPU is controlled by the software. The ISA acts as an interface between the hardware and the software, specifying both what the processor is capable of doing as well as how it gets done.

For example:

ARMv7 – used in ARM Cortex-M3.
ARMv7E – used in Cortex-M4 and M7.

Many vendors (for example, STMicroelectronics, NXP, TI) manufacture MCUs that support ARM ISAs but include their own peripheral sets. Understanding the ISA is essential for low-level coding and interpreting assembly instructions.

This ARMv7-M architecture reference manual provides more details on v7 Architecture.

Memory in Microcontrollers

Most microcontrollers typically feature two types of memory:

Flash – Stores your code and read-only data.
RAM – Used during program execution to hold:
- The heap (for dynamic memory)
- The stack
- The .data and .bss sections (initialized/uninitialized global/static variables)

Later sections have resources that go deeper into memory mapping and how these regions interact during runtime.

Clock and Power Management

Microcontrollers are digital logic devices built from:

Combinatorial logic – Logic gates that evaluate outputs instantly
Sequential logic – Relies on clocks to move through states

The clock tree distributes timing signals across the CPU and peripherals. MCUs often support multiple clock sources (internal RC, external crystal, PLL), and use prescalers to drive components at different frequencies.

For power-sensitive applications, MCUs offer multiple low-power modes:

Sleep – CPU off, timers and peripherals are mostly active, memory is retained
Deep Sleep – CPU off, most clocks off, memory is retained, wake-up is slower than sleep, power consumption is lower than Sleep
Standby – CPU off, few interrupts are active, everything else is powered down, memory is not retained. Lowest power mode.

These modes reduce power consumption by turning off clocks and disabling unused peripherals. Designing the system to switch in and out of low-power states effectively is a core skill in embedded software development.

This article talks about Clock Trees and Oscillators for the ARM Cortex microcontrollers.

Interrupts

Interrupts let MCUs react to asynchronous events, like button presses or sensor signals.

An interrupt temporarily pauses normal code execution to run a dedicated handler. After it’s serviced, the CPU resumes its previous task. They are vital for:

Fast event response
Reduced polling
Efficient power use (for example, waking from sleep)

Timers

Timers are built-in peripherals used to track time or generate events.

Common uses are:

Implementing software delays
Creating precise software timers
Waking up from low-power modes

Mastering timers helps with real-time behavior and precise event scheduling.

Communication Protocols

Microcontrollers often need to talk to other devices via built-in communication peripherals:

UART (Universal Asynchronous Receiver/Transmitter): Serial communication between two devices, great for logs and debugging.
I²C (Inter-Integrated Circuit): Two wire protocol for talking to sensors and EEPROMs.
SPI (Serial Peripheral Interface): High Speed, full-duplex protocol for devices like Flash or displays.
USB (Universal Serial Bus): Complex but widely used for PCs, data acquisition and HID devices.

Here’s a figure showing multiple peripherals connected to a MCU:

DMA or Direct Memory Access is an important peripheral which can be used to transfer data to/from memory without CPU involvement. It improves performance and allows the CPU to perform other tasks or enter low power mode to reduce power consumption.

This article provides a good overview of the communication protocols I2C, UART and SPI.

We’ve now covered the essential building blocks of microcontroller hardware – from memory and clocks to interrupts and communication buses.

Next, we’ll explore the software principles and tools that bring these microcontrollers to life, including compilers, debuggers, and embedded development frameworks.

Firmware Design and Tools

Designing Embedded Software

Even though embedded systems operate under unique hardware constraints, software design principles are still crucial. Applying them thoughtfully becomes even more important when memory, CPU cycles, and responsiveness are limited.

Most Embedded firmware projects begin with a structured design approach:

Understand the problem statement
List assumptions
Define use cases
Define system and software requirements
Create high-level architecture
Drill down to detailed design and implementation

If you’re new to software design, check out my article on design principles.

Here’s a figure showing the five blocks of software design:

Using Design Patterns

Once you're designing individual components, design patterns help you write scalable and maintainable code. Here are some common patterns in embedded systems:

Publisher-Subscriber (Observer) – Useful for decoupling event producers and consumers (for example, sensor data being broadcast to multiple modules).
Singleton – Ensures only one instance of a module or resource manager exists (for example, for drivers or HAL layers).
Adapter – Translates between incompatible interfaces (for example, wrapping platform-specific code into a portable application layer).
State Machine – Represents system behavior as transitions between states (for example, Bluetooth states: IDLE → SCANNING → CONNECTING → CONNECTED → DISCONNECTED).

Design patterns often need to be adapted for memory and timing constraints, but the core concepts remain highly relevant.

There are lot of great resources on design patterns – here are a few that helped me:

Book: Head-first Design patterns - A great book to get understand the concept of design patterns
Book: Design Patterns: Elements of Reusable Object-Oriented Software
Course: Object-Oriented Programming and Design Patterns in C#
Article on HSM: Hierarchical State Machine Overview (Barr Group)

Programming Languages for Embedded Systems

While any language can theoretically be used if it compiles to machine code, in practice, three dominate the embedded world:

C – The industry standard. Provides deterministic behavior and low-level access, making it ideal for memory and timing-sensitive code.
C++ – Adds object-oriented features while maintaining control. Once considered risky in embedded due to synthesized code and overhead, it’s now widely adopted where systems benefit from abstraction and modularity.
Rust – A memory-safe alternative gaining traction in safety-critical and open-source embedded development.

Languages like Python (via MicroPython or CircuitPython) are used in educational or prototyping contexts but are not suitable for production due to performance and memory overhead.

Some resources on programming languages that might be helpful to understand concepts:

The Embedded Rust Book
C Programming Language by K&R
Inside the C++ Object model – There are a lot of books and lectures on C++, but for embedded, understanding the object model benefits a lot.

Data Structures Matter

Embedded systems require careful data handling due to strict memory and timing constraints. Mastering core data structures is essential:

Arrays – fixed-size data.
Linked Lists – Common in software timers, queues.
Stacks and Queues – Task scheduling, event management and data storage.
Bitfields/Flags – Memory efficient state representation.
Binary Trees – Used in routing tables or decision logic.

You'll often build event queues, circular buffers, or timer lists, all of which rely on these foundational structures.

There are a lot of resources for understanding data structures, but I have found this one to be helpful for learning and practicing: GeeksForGeeks DSA Tutorial. And here’s a full course on DSA if you want to dive deeper.

Bit Manipulation: A Core Embedded Skill

Unlike general-purpose software, embedded systems often require low-level access to registers and require precise bit control:

Setting and clearing individual bits
Using bitwise operators like AND (&), OR (|), XOR (^)
Bit masking and shifting (<<, >>)

Mastering bit hacks is essential for writing hardware drivers or manipulating control registers.

This resource provides a good number of examples for bit manipulation: Stanford Bit Hacks.

Tools and Concepts for Embedded Development

Cross Compilation

Embedded code is compiled on a host (like your PC) for a target architecture using cross-compilers.

To do this, you need:

A compiler (for example, arm-none-eabi-gcc for ARM Cortex-M) that compiles high level language code into Assembly language instructions.
A linker to layout and combine object files.
A Makefile or build system to organize and automate compilation, linking and binary creation.

Here’s an example to compile a main.c to create a main.elf that can be flashed on the device:

arm-none-eabi-gcc main.c -o main.elf

A Makefile is a script used by the make build automation tool to compile and link programs to create a binary. It defines how to build your program from source files, manages compilation order based on dependencies and defines commands to complete the build.

For example, lets write a Makefile for building a project for an ARM Cortex-M4 target that has three source files: a main.c, utils.c, and sensor.c

CC = arm-none-eabi-gcc
CFLAGS = -c -mcpu=cortex-m4 -mthumb -Wall -O2
LDFLAGS = -mcpu=cortex-m4 -mthumb
TARGET = main.elf
OBJS = main.o utils.o sensor.o
SRC = main.c utils.c sensor.c

$(TARGET): $(OBJS)
    $(CC) $(OBJS) -o $(TARGET)

main.o: main.c
    $(CC) $(CFLAGS) main.c

utils.o: utils.c
    $(CC) $(CFLAGS) utils.c

sensor.o: sensor.c
    $(CC) $(CFLAGS) sensor.c

clean:
    rm -f *.o *.elf

In the above makefile, here’s a description of the flags:

-mcpu=cortex-m4: Targets the ARM Cortex-M4 processor.
-mthumb: Enables Thumb instruction set, which is used by ARM Cortex-M series.
-Wall: Enables all common warnings.
-O2: Optimization level 2 for balance between performance and code size.

Makefiles can seem intimidating, but they’re just scripts that define how to build your program from source. Once you understand the basics, they’re a huge productivity booster.

A linker script tells the linker (ld) how to organize the program in memory where to place code, data, stack, heap, and so on. It's crucial for embedded systems because you're working with limited memory and specific memory-mapped hardware.

Here’s an example of a simple linker script for a STM32F4 microcontroller:

/* STM32F4 Cortex‑M4 Simple Linker Script */

ENTRY(Reset_Handler)

/* Define memory regions based on STM32F4 datasheet */
MEMORY
{
  FLASH (rx) : ORIGIN = 0x08000000, LENGTH = 1024K
  RAM   (rwx): ORIGIN = 0x20000000, LENGTH = 128K
}

/* Section layout */
SECTIONS
{
  /* Interrupt vectors and code go into Flash */
  .isr_vector :
  {
    KEEP(*(.isr_vector))    /* Keep vector table (reset, etc.) */
  } > FLASH

  .text :
  {
    *(.text*)               /* All code */
    *(.rodata*)             /* Read-only data */
    . = ALIGN(4)
    _etext = .             /* End of code (used for data init) */
  } > FLASH

  /* Initialized data: load from Flash, run in RAM */
  .data : AT(_etext)
  {
    _sdata = .            /* Start of .data in RAM */
    *(.data*)
    . = ALIGN(4)
    _edata = .            /* End of .data */
  } > RAM

  /* Uninitialized data (zero-filled) */
  .bss :
  {
    _sbss = .
    *(.bss*)
    *(COMMON)
    . = ALIGN(4)
    _ebss = .
  } > RAM

  /* Define stack end (top of RAM) */
  _estack = ORIGIN(RAM) + LENGTH(RAM);
}

Descriptions of the above file:

MEMORY: Defines your microcontroller’s memory layout – 1 MB Flash and 128 KB SRAM.
ENTRY(Reset_Handler): Sets the reset handler as the program entry point.
.isr_vector and **.**text: Code sections placed in Flash. .isr_vector must use KEEP() so it's not removed during linking.
.data : AT(_etext): Loads initialized variables from Flash but places them in RAM.
**.**bss: Zero-initialized data, allocated in RAM
_estack: Defines the initial stack pointer using the end of RAM.

Here are some sources to understand Makefiles, cross-compilation, and Linkers. And just note that using Makefile in a project is the best way to learn and master Makefiles:

Makefiles:
Linker Scripts:
- Interrupt Blog on Linker Scripts
- Intro to Linker Files – Medium

Flashing the Binary

Once you’ve compiled your code into a binary file, the next step is to flash it into the target microcontroller’s non-volatile memory via SWD (Serial Wire Debug) or JTAG. Flashing tools like OpenOCD, ST-Link, J-Link, or vendor-specific utilities manage this process.

What Is Flashing?

Flashing is the process of writing a compiled firmware image (typically a .bin or .hex file) into the microcontroller’s Flash memory. This enables the embedded system to retain and run your code even after power is removed.

The flashing tool communicates with the microcontroller over SWD or JTAG to:

Halt the MCU (if needed)
Access the internal flash controller
Erase the relevant flash sectors
Write the binary data to specific memory addresses
Verify that the data was written correctly

OpenOCD (Open On-Chip Debugger) is a powerful, open-source utility that facilitates debugging and flashing of ARM-based microcontrollers. It supports a wide variety of hardware interfaces and microcontroller families, including STM32.

OpenOCD provides:

Flashing capabilities for .elf, .bin, and .hex files
Debugging via GDB (GNU’s open source debugger) integration
Support for multiple debug probes (J-Link, ST-Link, CMSIS-DAP)
Scripting via configuration files for board-specific and target-specific setups

A simple command to flash a binary using OpenOCD might look like this:

bashCopyEditopenocd -f interface/stlink.cfg -f target/stm32f4x.cfg -c "program main.elf verify reset exit"

This tells OpenOCD to:

Use the ST-Link interface
Load the STM32F4 target configuration
Program main.elf into flash
Verify it was written correctly
Reset the MCU
Exit the session

For a detailed walkthrough, check out: OpenOCD Deep Dive – Kickstart Embedded

Bare Metal, RTOS, and Embedded Operating Systems

When writing embedded software, you can approach the problem in three main ways, each with its own trade-offs:

Bare-Metal Programming
Real-Time Operating Systems (RTOS) (like FreeRTOS, Zephyr)
Embedded Operating Systems (like Embedded Linux)

The best choice depends on your use case, application’s complexity, hardware constraints, and real-time needs.

Most Modern 32-bit microcontrollers (for example, STM32, NXP, Renesas) come with vendor-provided development tools that include:

HAL (Hardware Abstraction Layer) libraries
Startup code and linker scripts
Peripheral drivers
Sometimes even middleware like USB, BLE, or file system stacks

These tools (like STM32Cube Config Tools) simplify setup and peripheral configuration, helping you get started quickly, without needing to write low-level code manually.

Benefits of HALs:

Rapid prototyping and development
Clean, reusable APIs for peripherals
Great for onboarding and small teams

Drawbacks:

Code bloat – HALs support many edge cases and configurations, which can inflate your binary size
Extra latency – HAL often inserts unnecessary layers that reduce performance.

For performance-critical systems, developers often replace HAL drivers with custom, low-level implementations.

Bare-Metal Programming

Bare-metal programming is the most direct and lightweight approach. There’s no OS, and your code runs directly on the hardware with full control.

Typical setup includes:

Include the correct header files, especially MCU and peripheral-specific headers provided by the vendor’s HAL (Hardware Abstraction Layer).
Implement a main() function with an infinite loop (while(1))
Perform all hardware initialization before entering the loop
Use Interrupts to handle asynchronous events.
Continuously check and control inputs/outputs inside the loop

This assumes your toolchain provides startup code and memory setup from the vendor.

#include "MCU_Header.h"

int main(void) {
    /* Initialize the MCU and the peripherals */
    init_clock();
    init_peripherals();

    /* runs in a loop forever */
    while (1) {
        // Task 1 : Read sensor data
        read_sensor(); 
        // Task 2 : Update the actuator based on the sensor data
        update_actuator(); 
    }
}

How does it run?

When the device powers on or resets, the startup code provided by the vendor is executed first. This code:

Initializes the reset vector
Copies initialized data from Flash to RAM
Zeros out the .bss section (for uninitialized global/static variables)
Calls your main() function

After calling main(), the system enters an infinite loop where your logic runs. The only other context switch occurs when an interrupt is triggered, briefly diverting control to an Interrupt Service Routine (ISR), after which it returns to the main loop.

When to use it:

Simpler applications (for example, blinking LEDs, reading sensors)
Ultra-low-power or ultra-low-latency needs
When every byte of Flash and RAM matters

Pros:

Minimal memory usage
Maximum control
Great for learning

Cons:

No built-in task management or scheduling
Can become hard to maintain for complex systems

This resource provides good details and example on Bare Metal Programming. For more details, this book is great as well: ARM Baremetal Ebook.

Real-Time Operating Systems (RTOS)

A Real-Time Operating System (like FreeRTOS, Zephyr) adds lightweight multitasking capabilities to your embedded application. It allows you to split your software into independent tasks that run concurrently and communicate through queues, semaphores, or message passing.

RTOS kernels often support different scheduling strategies like:

Rate Monotonic Scheduling (RMS) – Tasks with shorter periods get higher priority
Earliest Deadline First (EDF) – Tasks are prioritized based on impending deadlines

Example use cases:

A drone where sensor data, motor control, and telemetry need to run in parallel
A medical device where timing is critical for safety
Rockets

Typical RTOS features:

Task scheduling
Timers
Inter-task communication
Interrupt handling integration
Power management

Pros:

Modular code structure with tasks
Easier to scale as complexity grows
Deterministic execution (when configured correctly)

Cons:

Slightly higher memory footprint than bare-metal
Learning curve for scheduling and priority tuning

RTOS Scheduling techniques are interesting – this part of the docs talks about Zephyr scheduling.

Embedded Operating Systems

Sometimes an embedded system is powerful enough to run a full-fledged OS like Embedded Linux, Android Things, or Windows IoT Core. This is common on devices with a display, networking stack, or file system.

It’s best used when the system requires multitasking, user interfaces, file systems, or network stacks, and when there’s plenty of processing power (for example, ARM Cortex-A).

Think of:

Smart home hubs
Automotive infotainment
Industrial gateways

This table provides a high level methodology for choosing the right type of OS based on your application:

Criteria	Bare Metal	RTOS	Embedded OS
System Complexity	Low	Medium	High
Memory Footprint	Very Low	Moderate	High
Real-Time Guarantees	Limited	Yes	Depends on Kernel Design
Learning Curve	Steep for scaling	Moderate	Steeper (OS internals, tools)
Use Case Examples	Blinking LED, sensor polling	Drones, medical devices	Gateways, touchscreens

To understand OS fundamentals, this is a great book: Operating System Concepts and this is a great course: UC Berkeley: CS162.

So far, we’ve looked at how embedded applications are structured, whether using bare-metal loops, RTOS multitasking, or full operating systems. But regardless of which execution model you choose, your software ultimately needs to interact with the hardware.

This is where driver development comes in. Drivers form the crucial link between your code and the peripherals it controls, whether it's reading temperature, blinking an LED, or transmitting data over SPI. Let’s take a closer look at how to design robust, portable drivers for embedded systems.

Designing Drivers for Embedded Systems

When working with embedded software, one of the most practical and common tasks you’ll encounter is driver development.

A driver is a piece of software that enables the microcontroller (MCU) to interface with a hardware peripheral. This could be a temperature sensor, a motor controller, a display, or even a wireless module.

Drivers act as a bridge between your hardware and the application logic. They abstract away the raw register-level programming so that higher-level code can use clear function calls like read_temperature() or start_motor().

What Goes Into a Driver?

A typical embedded driver will include:

Configuration – Setting up the peripheral with initial parameters (for example, baud rate for UART)
Initialization – Preparing the peripheral for use, including enabling clocks and interrupts
Calibration (if needed) – Adjusting the peripheral based on specific environment or use case
Register Access – Reading from and writing to hardware registers (if applicable)
Power Management – Enabling/disabling the peripheral to save power or putting the peripheral into a low power mode
Interrupt Management – Handling asynchronous events triggered by the peripheral

Here’s a simplified view of a sensor driver API:

void sensor_init(void);
void sensor_calibrate(void);
float sensor_read_temperature(void);
void sensor_sleep(void);
void sensor_write(uint8_t reg, uint8_t value); // Assumption : 8 bit register address and 8 bit data value

The actual implementation might involve:

Register definitions from the peripheral’s datasheet
Bit manipulations for control and status registers
Interrupt Service Routines (ISRs)
Timing and delay management

Platform Abstraction: Why It Matters

One of the most important principles in driver design is decoupling the application from the platform. This makes your code easier to:

Port to different MCUs
Adapt for similar hardware (for example, different sensor models)
Test across simulated or real environments

Platform-Agnostic Design Example (in C++) :

Let’s say you're writing a driver for a temperature sensor:

// Abstracts the HW platform on which the sensor driver is being written
class TemperatureSensorPlatform {
public:
    void i2cInit(void);
    void i2cWrite(uint8_t reg, uint8_t value);
    uint8_t i2cRead(uint8_t reg);
};

// Creates a generic Temperature sensor driver interface
class TemperatureSensor {
public:
    virtual void init() = 0;
    virtual float read() = 0;
    virtual void sleep() = 0;
};

You can implement this interface differently for a specific type of temperature sensor and also add the platform support for the HW platform you are writing the driver on for example STM32.

class TempSensorTMP117 : public TemperatureSensor {
public:

    TempSensorTMP117(TemperatureSensorPlatform platform) : 
    _platform(platform)
    TemperatureSensor()
    {}

    void init() override {
        // TMP117-specific register configuration
    }

    float read() override {
        // Read ADC value and convert
        return 25.4f;
    }

    void sleep() override {
        // Put sensor in low-power mode
    }
private:
    TemperatureSensorPlatform _platform; // Implements the I2C driver for STM32
};

Your application code now depends on the TemperatureSensor interface and Temperature Sensor Platform passed in the constructor making it portable and testable across temperature sensors and HW platforms.

One of my previous articles provides details on how to interface a sensor and how to design a driver for it.

Designing robust and modular drivers helps your firmware interact seamlessly with hardware, but in today’s connected world, that’s only part of the challenge. As embedded devices increasingly communicate with other systems, security becomes just as critical as functionality.

Now that we’ve covered how to interface with hardware, let’s explore how to protect those systems from unauthorized access, tampering, and data breaches.

Security in Embedded Systems

Security is often overlooked in embedded development but it shouldn’t be. Embedded systems are increasingly connected to networks, cloud services, or other devices, which makes them vulnerable to attacks like unauthorized access, firmware tampering, or data leaks.

Even simple devices like smart plugs or fitness trackers can be exploited if their firmware is insecure.

Key Security Practices

Secure Boot: Ensure the firmware is cryptographically signed and verified before execution. This prevents unauthorized firmware from running.
Firmware Update Integrity: Use encrypted or signed updates, especially for Over-the-Air (OTA) upgrades. Unprotected updates can be a major attack vector.
Lock Debug Interfaces: After flashing the final firmware, disable or lock access to JTAG, SWD, or UART debug ports to prevent reverse engineering.
Minimal Exposure: Disable unused peripherals (for example, Bluetooth, USB, network interfaces) and avoid exposing debug info (like UART prints) in production.
Watchdog Timers: While not security features per se, watchdogs help ensure system recovery in the event of unexpected software behavior – which could result from attacks or bugs.

Security should be layered, as no single mechanism is sufficient on its own. Build security into every stage of the development process, from boot to communication to update handling.

Whether you're designing a consumer product or an industrial controller, proactive security practices are essential for protecting user data, system reliability, and device reputation.

This resource provides a good understanding of Embedded Systems Security: BlackBerry QNX: Embedded System Security Guide

Debugging and Forensics in Embedded Systems

Debugging embedded systems is one of the most challenging and fascinating aspects of development. Unlike in desktop or web applications, bugs in embedded systems often manifest as unexpected hardware behavior rather than error messages.

For example, suppose your code is supposed to blink an LED once per second:

If the LED stays on, your delay code might be broken.
If it blinks erratically, you might have a timing bug.
If it doesn’t blink at all, you might never be reaching that part of your code or the hardware might not be configured correctly.

Why Debugging is Critical

Embedded systems directly control real-world hardware, often in critical or safety-sensitive environments. A small bug can lead to large consequences.

Historical Note: During the Apollo 11 moon landing, the onboard computer started throwing alarms due to a task overflow. The system restarted and was able to recover itself and allowing the mission to continue safely.

Debugging and post-mortem analysis (forensics) are essential skills for embedded developers.

Common Debugging Tools and Techniques

1. Print Statements (UART Logging)

The simplest and most common method. They send debug messages over a serial connection (UART).

You can use printf() or similar to track variable values, function entries/exits, and system state

Pros: Easy to implement
Cons: Can affect timing – not usable if UART is unavailable or disabled

2. Trace Variables

In systems without output peripherals (like UART), you can use trace flags, setting bits in a global variable to indicate code progress.

uint32_t trace_flags = 0;

void init_sensor() 
{
    trace_flags |= (1 << 0); // Bit 0: sensor init started
    // ...
    trace_flags |= (1 << 1); // Bit 1: sensor init complete
}

You can then examine trace_flags in memory to track execution flow, even post-mortem. The trace flags can be printed out or dumped via lldb or gdb.

3. Hardware Debugging: JTAG, SWD, and Debuggers

Modern microcontrollers (like ARM Cortex-Ms) support hardware debugging interfaces such as:

JTAG (Joint Test Action Group)
SWD (Serial Wire Debug)

These allow a debugger to:

Pause execution
Set breakpoints
Inspect and modify memory
Single-step through code

ARM CoreSight is a debug and trace architecture developed by ARM for its processor cores (like Cortex-M, Cortex-A, Cortex-R). It provides a set of hardware modules built into ARM-based chips that allow developers to:

Debug the system while it's running (non-intrusively)
Trace code execution, memory accesses, and peripheral activity
Analyze system performance and find hard-to-catch bugs

In short: CoreSight lets you look inside your embedded system while it's alive and working, without halting it unnecessarily.

Why CoreSight Exists

Traditional debugging tools (like breakpoints or single-stepping with JTAG) are often intrusive (they pause the system), limited (can't capture what happened right before a crash), or not suitable for real-time systems.

CoreSight solves these by enabling real-time tracing and non-intrusive observation of what's happening inside the chip.

Popular Debug Tools:

ST-Link – HW from STMicrocontrollers
J-Link – Universal debugger supporting a wide range of MCUs
OpenOCD – Open-source interface for hardware debugging
GDB / LLDB – Command-line debuggers used alongside the above

Single-stepping is most effective when compiler optimizations are off. With optimization, code might be reordered, inlined, or even eliminated.

4. Using Map and Disassembly Files

When debugging complex issues, especially crashes or memory overflows, you'll need to go deeper.

Map Files show the layout of functions and variables in memory (Flash and RAM). They help you locate:

Stack overflows
Unexpected memory usage
Function addresses

Disassembly Files let you see the machine code generated from your source. This is critical when:

Code is heavily optimized
You’re diagnosing instruction-level failures
You’re working without source code (e.g., binary-only drivers)

This resource provides a good overview on Map files, linkers and ELF format: Tenouk’s ELF/Map/Linker Guide

Common Bug: Buffer Overflows

Buffer overflows are one of the most frequent (and dangerous) issues in embedded systems. They happen when data is written past the end of an allocated array, overwriting nearby memory and causing unpredictable behavior.

Symptoms:

Code crashes mysteriously
Data appears to “corrupt itself”
Variables change value without explanation

You can learn more in my article on Debugging Buffer Overflows, which walks through ways to debug a buffer overflow and build robust buffer code.

Embedded Forensics

Sometimes, a device fails in the field, where you can’t attach a debugger. That’s where forensics comes in:

Use watchdog timers to reset the system and log failure info
Save crash signatures to non-volatile memory (for example, EEPROM, Flash)
Implement assert handlers that log file names, line numbers, or fault types

These techniques help you reconstruct what went wrong after the device has rebooted or been recovered.

You can learn more here: Debugging Techniques for Embedded Systems – Medium.

Debugging and forensics are invaluable when something goes wrong – but a robust system should aim to catch issues before they reach deployment.

That’s where automated testing becomes essential. With embedded software increasingly powering critical applications, the ability to run consistent, repeatable tests across hardware configurations saves time, improves reliability, and enables faster development cycles.

Next, let’s explore how embedded testing works, the challenges unique to hardware, and how automation frameworks help streamline validation.

Automation and Testing in Embedded Systems

Like all other areas of software engineering, testing is essential in embedded systems. But testing embedded software comes with its own set of challenges, mainly because it interacts with hardware.

Manual testing can be time-consuming and resource-intensive, especially when tests need to be repeated for multiple firmware versions or configurations. That’s where automated testing becomes invaluable.

Why Automated Testing?

Automated testing helps:

Catch regressions early
Test edge cases consistently
Reduce human error
Scale testing across versions and hardware setups

But automating tests for embedded systems isn’t just writing test cases – it’s about setting up an infrastructure that connects your code to the physical hardware under test.

Test Architecture: Host + DUT

Most embedded test setups involve two components:

Host: Your development PC or CI test controller, which sends test commands and receives data.
DUT (Device Under Test): The microcontroller board or embedded system running the firmware.

These two communicate over a physical link, commonly USB, UART, or FTDI, which carries commands and test data between them.

Diagram (suggested structure)

You could visualize this as:

Key Components of Embedded Test Automation

1. File Management

Many automated tests rely on CSV or JSON files to define:

Input configurations
Expected outputs
Test parameters

Python makes it easy to:

Read input vectors from CSVs
Write logs or pass/fail results
Parse structured data

2. Data Communication

Maintaining a stable and reliable link between the Host and DUT is critical. This includes:

Opening and managing UART or USB connections (for example, with pyserial)
Framing test commands using opcodes or simple protocols
Handling timeouts, retries, and error recovery

Example (Python with PySerial):

import serial

ser = serial.Serial('/dev/ttyUSB0', 115200) #set Baud rate
ser.write(b'\x01')  # Send opcode for "start test"
response = ser.read(64)  # Read 64 bytes of response

3. Automation Manager (DUT-side)

A lightweight software agent runs on the embedded device. Its responsibilities:

Parse incoming commands
Trigger specific test routines
Send response data back to the host

This is often implemented using a switch-case structure in C or C++:

void automation_manager(uint8_t opcode) {
    switch(opcode) {
        case 0x01: run_sensor_test(); break;
        case 0x02: run_motor_test(); break;
        default: break;
    }
}

4. Automation Manager (Host-side)

This is the control center of your test workflow:

Sends test commands and parameters to the DUT
Waits for and logs results
Compares responses to expected output
Handles communication retries or failures

Often written in Python using:

pyserial for communication
pandas for file/data processing
unittest or pytest for test structure

Tips for Effective Automation

Use unique opcodes for each test command to avoid ambiguity
Implement timeout handling to avoid hanging scripts
Log everything, responses, errors, test timestamps
Use versioned test input files to track changes over time
Include self-tests on the DUT to validate hardware state before running full tests

Automated testing in embedded systems is not just about running scripts, it's about building a bridge between your host PC and your device, managing the flow of commands and data, and ensuring tests are consistent, repeatable, and reliable.

While this requires effort to set up, the payoff is huge: confidence in your firmware, faster development cycles, and reduced risk of bugs making it into production.

Where to Go from Here

Building your Embedded Project

After exploring the theory and tooling of embedded systems, it's time to apply what you've learned. This section walks you through the steps to create your own embedded system – from concept to code and deployment.

Use the checklist below to guide your first project, whether you're prototyping a sensor device or automating a simple process.

Project Setup Checklist:

Define the Goal
- What task does the system perform?
- Identify inputs (for example, temperature sensor) and outputs (for example, relay or LED).
Requirements Gathering
- Functional: What features must it support?
- Non-functional: Memory limits, real-time behavior, power constraints.
- Any security or safety-critical elements?
Choose Your Hardware
- Microcontroller (for example, STM32F4)
- Sensors and actuators
- Communication interfaces (UART, I2C, SPI, and so on)
Software Architecture
- Bare-metal, RTOS, or embedded OS?
- Driver abstraction: will you use HAL or custom low-level code?
- Organize code into layers: application logic, drivers, hardware init.
Toolchain Setup
- Install GCC toolchain (for example, arm-none-eabi-gcc)
- Configure Makefile and linker script
- Set up debugger and flashing tools (for example, OpenOCD, ST-Link)
Firmware Implementation
- Initialize peripherals
- Implement control logic inside main() or tasks
- Use interrupts or timers for responsiveness
Flashing and Initial Tests
- Use OpenOCD or ST-Link to flash the binary
- Test peripheral behavior and debug with UART or GDB
Debug and Profile
- Use JTAG/SWD, CoreSight, and trace logs
- Check memory layout with map/disassembly files
- Identify bottlenecks and edge cases
Security Hardening
- Disable debug interfaces post-flash
- Add firmware signing and secure boot
- Minimize surface area: disable unused features
Testing and Automation

Connect Host to DUT via UART/USB
Use Python + PySerial to send test vectors
Log, compare, and report test outcomes

Embedded firmware development is a deep and rewarding field where software meets the hardware. Whether you're controlling an LED, reading from a sensor, or orchestrating multiple tasks in real time, the embedded stack teaches you how hardware, software, timing, and efficiency all come together.

Summary:

In this guide, we walked through the essential building blocks at a high level:

What embedded systems are, and how they sense → process → act
How microcontrollers work, from memory layout to interrupts and protocols
How to design robust, scalable embedded software with clean architecture
When to choose bare-metal, RTOS, or full OS solutions
How to build drivers, write modular code, and interface with peripherals
Tools for debugging, tracing, and analyzing system behavior
Strategies for automating embedded testing using Python and host-device communication
And finally, why security matters, especially in a connected world

Whether you're preparing for embedded job interviews, building your own IoT projects, or just exploring how software drives real-world systems, this article gives you a launchpad for deeper learning.

The Logic, Philosophy, and Science of Software Testing – A Handbook for Developers

Han Qi — Tue, 17 Jun 2025 18:43:38 +0000

In an age of information overload, AI assistance, and rapid technological change, the ability to think clearly and reason soundly has never been more valuable.

This handbook takes you on a journey from fundamental logical principles to their practical applications in software development, scientific reasoning, and critical thinking.

Whether you're a high school student learning to think more clearly, a professional debugging complex systems, or simply someone curious about how sound reasoning works, this handbook provides tools for sharper, more reliable thinking.

What We’ll Cover:

Part I: Foundational Theory

We start with the bedrock of formal logic – understanding implications, truth tables, and the core rules of reasoning.

You'll learn the scaffolding for everything that follows:

How "if-then" statements actually work (spoiler: it's not always intuitive!)
The power of truth tables to map all possible scenarios
Why some arguments are valid while others are logical fallacies
The elegant relationship between Modus Ponens, Modus Tollens, and Contrapositives

Part II: Practical Applications

Here's where logic comes alive in tangible ways:

In Software Development:

How debugging mirrors logical reasoning, and why your tests might be lying to you
The logic behind Test-Driven Development and Mutation Testing

In Scientific Thinking:

Karl Popper's falsification principle and why it matters beyond academia
How Hypothesis Testing is just statistics meets Modus Tollens

In Everyday Reasoning:

Spotting logical fallacies in arguments, media, and your thinking
The art of considering multiple causal paths instead of jumping to conclusions

Part III: Philosophical Depths

The final section confronts the beautiful complexity of applying pure logic to an impure world:

Why perfect "if-and-only-if" relationships are the goal but rarely achievable
How modern software systems hide their complexity
The butterfly effect of bugs and why root cause analysis is often harder than it seems
Formal verification tools: from Prolog to Coq to TLA+

What You'll Gain

For Students:

Critical thinking superpowers: Learn to spot flawed reasoning in arguments, social media, and news
Academic advantage: These concepts appear in debates, philosophy, computer science, mathematics, and statistics

For Software Engineers:

Debugging mastery: Modus Tollens for debugging: "If the output is wrong, what could cause it?"
Testing philosophy: Move beyond "make the tests pass" to "prove the code is correct"
Problem analysis: Avoid jumping to solutions before understanding the real problem
System design: Think more rigorously about failure modes and edge cases, evaluate cause-and-effect relationships in complex systems
Communication and career growth: Present arguments more clearly and persuasively, gain logical thinking skills that separate senior engineers from juniors

For Scientists:

Experimental design: Strengthen your understanding of hypothesis testing and falsifiability
Peer review: Better evaluate the logical soundness of research claims
Grant writing: Structure arguments more persuasively using solid logical foundations

Pre-requisites

I’ll introduce code samples starting in the second half of the article, so knowing a programming language would be helpful. The concepts in this article are programming language-agnostic, but I’ve used Python throughout for readability.

No prior formal logic or philosophy background is strictly necessary, but the following will let you reap the most benefits from this article:

Experience in testing and debugging during software development.
Know what REPL (Read-Evaluate-Print-Loop) is if you want to try the Proof Assistants.
Knowledge of logical operators (NOT, AND, OR), and the fact that they take 1 or 2 boolean values as input and return a single boolean value as output.
Basic Algebraic Thinking: representing statements as variables (P, Q), the concept of NOT (¬) as an inversion of statements, and the concept that different input combinations can reach the same output.
Exposure to deductive reasoning, where inferences are made based on some facts, and fallacies, which are some ways arguments can be flawed.
Willingness to engage in conceptual back-and-forth between concrete English examples and abstract logical symbols.
Holding possibly conflicting ideas between the ideal logic world and the impure real world.
Openness to challenging intuition and following logical rules before applying your real-world experience.

An Introduction to Logic
Truth Tables: Mapping All Possibilities
Contrapositives, Modus Ponens, Modus Tollens
The Origin of P⟹Q: Science and Reality
Revisiting Argument Forms: Valid Inferences and Common Fallacies
Denying the Antecedent: A Database Example
Assigning Real-World Meanings to Logic
Applying Logic to Software Testing
A Closer Look at Testing
Revisiting the Four Statements for Coding
The Missing Ingredient - If and Only If
Mutation Testing: Testing the Tests
Toward If-and-Only-If Confidence
Real-World Challenges
Glimmers of Hope: Tools and Practices for Clarity
The Power of Falsification in Testing
Proof Assistants
Food for Thought
Q.E.D.: The Enduring Power of Logic in an Uncertain World
Resources
Glossary

An Introduction to Logic

Imagine that the following statement is True:

If you are a coding instructor, then you have a job.

Now, do these make sense?

You have no job, so you are not a coding instructor
You have a job, so you are a coding instructor
You are not a coding instructor, so you have no job

Interpretations

Based on logic:

Statement 1 is correct.
Statement 2 is wrong because you may have other jobs without being a coding instructor.
Statement 3 is wrong because you may or may not have a job, and as before, you may have other jobs without being a coding instructor.

Growing complexity

These statements grow increasingly complex due to:

Changing from 2 valid statements to 2 invalid conclusions
Moving from a clear job status (1, 2) to uncertainty about job existence or type (3).

Let’s get familiar with some notation before seeing how Truth tables help manage this complexity.

Notations

Notation	Meaning	Example (if P="It's raining", Q="The ground is wet")
P, Q	Propositions	P, Q
⟹	Implies / If...then...	P⟹Q ("If it's raining, then the ground is wet")
¬	Not	¬P ("It's not raining")
∧	And (conjunction)	P∧Q ("It's raining and the ground is wet")
∨	Or (disjunction)	P∨Q ("It's raining or the ground is wet")
⟺	If and only if (biconditional)	P⟺Q ("It's raining if and only if the ground is wet")
∴	Therefore	P ⟹ Q: If it's raining, then the ground is wet; P: It's raining; ∴ Q: Therefore, the ground is wet

Truth Tables: Mapping All Possibilities

What is a Truth Table?

A truth table is a powerful tool in logic that helps us determine the overall truth or falsity of a compound logical statement. It does this by systematically listing all possible combinations of truth values (True or False) for its individual component propositions.

For every way the "inputs" (our propositions like P and Q) can be true or false, the truth table shows you the precise "output" (the truth value of the entire logical statement, such as P⟹Q).

Why are Truth Tables Helpful?

Truth tables offer critical benefits for clear thinking:

Clarity and precision: They eliminate ambiguity by explicitly showing the outcome for every single scenario.
Systematic analysis: They ensure no possible combination is missed, which is vital for sound reasoning.
Foundation for understanding: They define how logical rules work, forming the bedrock for analyzing more complex arguments in any domain.

How to Read Our First Truth Table:

Let's examine the truth table for the implication P⟹Q ("If P then Q").

Each row represents a unique scenario, combining the truth values of P and Q to show the resulting truth value of P⟹Q.

P	Q	P⟹Q (If P then Q)	Used In
True	True	True	Modus Ponens ✅
True	False	False	Falsifiability 🚨
False	True	True	No Inference
False	False	True	Modus Tollens ✅

Let's break down each row:

P and Q Columns: These show the input truth values (True or False) for our two propositions. Since each can be one of two values, we have 2×2 = 4 unique combinations, filling all four rows.
P ⟹ Q Column: This is the output truth value of the "If P then Q" statement for each combination of inputs P and Q.
- Row 1: P is True, Q is True.
  - If P is true (you are a coding instructor) and Q is also true (you have a job), then the implication P⟹Q is True. (The "If...then..." statement holds).
  - This row is key for Modus Ponens.
- Row 2: P is True, Q is False
  - If P is true (you are a coding instructor) but Q is false (you have a job), then the implication P⟹Q is False. This is the only scenario that disproves an "if-then" statement.
  - This row is key for Falsifiability.
- Row 3: P is False, Q is True.
  - If P is False (you are not a coding instructor) but Q is True (you have a job), then the implication P⟹Q is still considered True. This can seem counter-intuitive.
  - The reason is that the implication statement only makes a claim about what happens when P is true. If P is false, the implication's claim isn't tested, so it is considered vacuously true.
- Row 4: P is False, Q is False.
  - If P is False (you are not a coding instructor) and Q is False (you have no job), then the implication P⟹Q is also considered True.
  - Similar to Row 3, since the initial condition (P) was false, the implication's truth value remains True, as it hasn't been disproven.
  - This row is key for Modus Tollens.

The "Used In" column serves as a preview of the specific logical arguments or concepts that rely on each row's behavior, which we will explore in detail later.

Understanding the Implication (P⟹Q) Deeper

Most programmers are familiar with truth tables from logical operators like AND (∧), OR (∨), and NOT (¬), where they define the output based on combinations of inputs.

The implication (P⟹Q) works similarly, its output is defined by the rules of propositional logic, not by any real-world causal relationship or your “common sense”. For any given pair of inputs for P and Q, the result of P⟹Q is fixed.

If this feels counter-intuitive, consider that mathematical logic, like any formal system, is built upon agreed-upon axioms. These basic accepted truths allow us to construct complex systems of ideas. If later found ineffective or contradictory, these axioms can be redefined, or a new system can be developed.

In formal logic, this implication is also defined as being logically equivalent to "NOT P OR Q" (¬P∨Q).

This is the fundamental logical rule that dictates why, if P is False, P⟹Q is always True, regardless of Q's truth value. You can also understand this using the NOT P OR Q form.

If P is False, that means NOT P is True.
Using the rules of Logical operation:
- True (Not P) OR True (Q) is True (NOT P OR Q)
- True (Not P) OR False (Q) is True (NOT P OR Q)
- NOT P OR Q is True regardless of what Q is.

The above explains rows 3 and 4 of the truth table from the NOT P OR Q form. As an exercise, you can apply the inputs (P, Q) from the first two rows of the truth table to NOT P OR Q to arrive at the same results defined in the P⟹Q column.

This formal definition allows us to use implication to reason in powerful ways, not just in the "forward" direction (P⟹Q, leading to Modus Ponens), but also in a crucial "backward" direction.

This backward form (Contrapositive) involves swapping and negating the propositions (¬Q⟹¬P).

For example, if "If you are a coding instructor, then you have a job" is true, then it must also be true that "If you have no job (¬Q), then you are not a coding instructor (¬P). ".

This "backward" way of reasoning, which underpins Modus Tollens, is a powerful tool for inferring conclusions from observed outcomes.

We'll explore the Contrapositive and two argument forms (Modus Ponens, Modus Tollens) in detail next.

Contrapositives, Modus Ponens, Modus Tollens

We've explored the fundamental implication (P⟹Q) and how truth tables reveal its behavior.

Now, we explore reasoning tools that build upon this foundation: Modus Ponens, Modus Tollens, and the concept of Contrapositives. These are bedrock principles of valid argument and efficient logical thought.

What is Logical Equivalence?

Before we dive into these specific concepts, let's clarify what logical equivalence means. Two statements are logically equivalent if they always have the same truth value under all possible circumstances. In simpler terms, if one statement is true, the other is always true. If one is false, the other is always false. They are, in essence, different ways of saying the same logical thing.

Understanding logical equivalence is incredibly useful. It:

Simplifies logic: It allows us to substitute one statement for another without changing the truth of an argument, which simplifies complex proofs and reasoning.
Reduces complexity: In fields like circuit design, it can lead to fewer physical gates.
Maintains software correctness: In programming, it helps maintain code's correctness during refactoring and debugging, especially when simplifying conditional statements, by ensuring the transformed code still behaves identically to the original under all conditions.

The Contrapositive: An Equivalent Implication

One of the most important logical equivalences involves the Contrapositive of an implication. The contrapositive of an "If P then Q" (P⟹Q) statement is "If not Q, then not P" (¬Q⟹¬P).

You might intuitively question how "If P then Q" could be logically the same as "If not Q then not P." Let's demonstrate this using a truth table.

We'll start with our familiar P and Q columns and the P⟹Q implication. Then, we'll add columns for ¬P (Not P) and ¬Q (Not Q), and finally, the implication for the contrapositive, ¬Q⟹¬P.

Let's look at how the truth table explicitly shows this equivalence:

Q, not P, not Q, not Q -> not P" class="image--center mx-auto" width="1042" height="325" loading="lazy">

Explanation of the table

P, Q, P ⟹ Q (Columns 1-3): These are our standard propositions and the implication we've already defined.
¬P (Column 4): This column simply shows the negation (opposite truth value) of the P column. If P is True, ¬P is False, and vice-versa.
¬Q (Column 5): Similarly, this column shows the negation of the Q column.
¬Q ⟹ ¬P (Column 6): This is the contrapositive. We apply the same rules for implication that we learned earlier, but now using ¬Q as our "if" part and ¬P as our "then" part. For example, in Row 2, ¬Q is True and ¬P is False. According to the implication rule (True ⟹ False yields False), the result for ¬Q⟹¬P is False.
The Proof of Equivalence: Now, compare Column 3 (P⟹Q) with Column 6 (¬Q⟹¬P). You'll notice that for every single row, their truth values are identical! When P⟹Q is True, ¬Q⟹¬P is also True. When P⟹Q is False, ¬Q⟹¬P is also False. This perfectly illustrates why they are logically equivalent.

So, "If you are a coding instructor, then you have a job" (P⟹Q) is logically the same as saying "If you have no job, then you are not a coding instructor" (¬Q⟹¬P). They convey the same information about the relationship between being a coding instructor and having a job.

How Modus Ponens and Modus Tollens Relate to Implication

Having defined logical equivalence and the contrapositive, we can now precisely understand two of the most fundamental and valid forms of deductive argument: Modus Ponens and Modus Tollens. Both of these argument forms rely on a core premise that an implication (P⟹Q) is true, and then use additional information to draw a valid conclusion.

Modus Ponens (Affirming the Antecedent): This is often considered the most intuitive and direct form of logical inference. It works in the "forward" direction of the implication.
- Premise 1: We are given that the implication is true: If P, then Q (P⟹Q).
- Premise 2: We are also given that the "if" part, the antecedent, is true: P is true.
- Conclusion: Therefore, we can validly infer that the "then" part, the consequent, must also be true: Q is true.

Example:

Premise 1: If it is raining (P), then the ground is wet (Q).
Premise 2: It is raining (P).
Conclusion: Therefore, the ground is wet (Q).

This directly corresponds to Row 1 (True, True) of our truth table for P⟹Q.

Modus Tollens (Denying the Consequent): This argument form works in the "backward" direction and relies directly on the logical equivalence of an implication and its contrapositive.
- Premise 1: We are given that the implication is true: If P, then Q (P⟹Q).
- Premise 2: We are also given that the "then" part, the consequent, is false: Not Q (¬Q).
- Conclusion: Therefore, we can validly infer that the "if" part, the antecedent, must also be false: Not P (¬P).

Example:

Premise 1: If it is raining (P), then the ground is wet (Q).
Premise 2: The ground is not wet (¬Q).
Conclusion: Therefore, it is not raining (¬P).

Modus Tollens is valid because if P⟹Q is true, its contrapositive (¬Q⟹¬P) must also be true. Applying Modus Ponens to this contrapositive (with ¬Q as our second premise) directly leads to the conclusion ¬P. This corresponds to Row 4 (False, False) of our original truth table for P⟹Q, where P and Q are both false but the implication is still true.

These two argument forms are central to rigorous deductive reasoning, allowing us to draw certain conclusions based on the truth of implications and related facts.

The Origin of P⟹Q: Science and Reality

In science, hypotheses often take the form "If P, then Q" where P is a cause and Q is its predicted effect –for example, "If a drug is given (P), then symptoms improve (Q)."

Ideally, P is controllable, as in experimental studies, but even in observational studies, P must be clearly defined and measurable.

Each experiment yields one observation, reflecting one of four possible truth-value combinations of P and Q.

The Falsifying Case in Science and Logic

Each experiment produces a single observation – one of the four possible combinations of P and Q.

If P=True, Q=False is observed (row 2 of the truth table), the hypothesis is falsified
In all other cases, the hypothesis is not falsified (yet)

Thus:

If all observations fall in the 3 truth-preserving rows, the hypothesis remains viable.
If at least one experiment yields P=True, Q=False, we either:
- Conclude falsification, or
- Re-examine the experiment and attempt replication before accepting falsification.

The Power of the Falsifying Case

In the Logical World

The falsifying case is not useful for inference with Modus Ponens or Modus Tollens because these two argument forms require starting with P⟹Q = True. I’ll explain both arguments in detail later.

But the falsifying case is useful for showing counterexamples to disprove the implication, or proof by contradiction.

In the Real Scientific world

The falsifying case embodies Falsifiability – a crucial concept in Science.

In so far as a scientific statement speaks about reality, it must be falsifiable: and in so far as it is not falsifiable, it does not speak about reality.

— Karl R. Popper, The Logic of Scientifc Discovery

Scientific theories come about through hypotheses that are continually tested and survive attempts at falsification.

Popperian Falsification and Hypothesis Testing

These two approaches, one philosophical and one statistical, are distinct but complementary in the scientific method.

Popperian Falsification starts with a scientific hypothesis (for example, "P has an effect on Q"). Its core aim is to actively seek evidence that would disprove this hypothesis. If such disproving evidence is found, the hypothesis is falsified.
Statistical Hypothesis Testing begins with a null hypothesis (H0) (for example, "P has no effect on Q"). Its goal is to determine if the collected data provides sufficiently extreme evidence to reject this null hypothesis.

If the null hypothesis is rejected, it provides statistical support for the alternative hypothesis (that P does have an effect on Q). This statistically supported hypothesis then becomes a stronger candidate, continually subjected to further Popperian attempts at falsification through new experiments and observations.

The Nuance: Implication is Not Causality

P⟹Q does not inherently imply that P causes Q.

Consider these examples:

"If the fire alarm is sounding, then there is smoke." The alarm doesn't cause the smoke.
"If a colleague screams during code review, then the code is bad." Does the screaming cause the bad code, or merely reveal it? (Perhaps sometimes both! 😰)

Causality is a real-world concept crucial for making informed decisions, predicting outcomes, and inferring the underlying reasons for events.

It's often central to predictive modeling and supervised learning in data science, where the target variable is the effect and the predictors are proposed causes. A common pitfall here is data leakage, where predictors are inadvertently influenced by (or are themselves effects of) the target, violating the causal assumption.

Logic, however, doesn't model time, mechanisms, or interventions. It only cares about truth values and formal structure. Logic defines what is true based on premises, not what makes something true in a causal sense.

Revisiting Argument Forms: Valid Inferences and Common Fallacies

We've now established the rules of implication, understood logical equivalence, and learned about two powerful, valid argument forms: Modus Ponens and Modus Tollens. But when we try to reason using "if-then" statements, it's easy to fall into common logical traps.

In this section, we'll systematically revisit the four common ways we might try to draw conclusions from an implication P⟹Q (If you are a coding instructor, then you have a job) introduced at the start of the handbook.

Two are valid arguments (Modus Ponens and Modus Tollens), and two are common logical fallacies. Understanding the differences is crucial for sound reasoning.

First, let's quickly define the parts of an "if-then" condition:

Antecedent: The "if" part of the condition (P).
Consequent: The "then" part of the condition (Q).

Now, let's examine these four argument forms, using our knowledge of truth tables and the coding instructor example.

Affirming the Antecedent (Modus Ponens)

This is the first valid argument form we discussed. It's called "affirming the antecedent" because it asserts the truth of the "if" part (the antecedent, P) to conclude the "then" part (the consequent, Q).

Argument Form:
1. If P, then Q (P⟹Q)
2. P is true.
3. Therefore, Q is true.
Examples:
- You are a coding instructor (P), so you have a job (Q).
- You provided invalid input data (P), so the code will show an error (Q).
Interpretation: This argument directly aligns with Row 1 (P=True, Q=True) of our truth table, where the implication holds true. It's often the most intuitive form of logical deduction. In programming, it's natural to expect bad input to lead to error messages if the code is designed correctly.

Denying the Consequent (Modus Tollens)

This is the second valid argument form. It's called "denying the consequent" because it asserts the falsity of the "then" part (the consequent, ¬Q) to conclude the falsity of the "if" part (the antecedent, ¬P). As we learned, Modus Tollens derives its validity from the logical equivalence of P⟹Q and its contrapositive (¬Q⟹¬P).

Argument Form:
1. If P, then Q (P⟹Q)
2. Not Q is true (¬Q).
3. Therefore, Not P is true (¬P).
Examples:
- You have no job (¬Q), so you are not a coding instructor (¬P).
- There are no error messages (¬Q), so the input data is valid (¬P)
Interpretation: This argument corresponds to Row 4 (P=False, Q=False) of our truth table, where P⟹Q is true, and both P and Q are false. This form of reasoning is critical for skillful debugging, allowing you to infer reasonably true conclusions about the cause (P) from observations of the outcome (Q), assuming your program logic (P⟹Q) holds true.

Affirming the Consequent (Fallacy)

Now we move to the common pitfalls. This is an invalid argument form where we attempt to conclude that the antecedent (P) is true simply because the consequent (Q) is true. It's a fallacy because the truth of Q does not guarantee the truth of P, as Q could have been caused by something other than P.

Argument Form (Invalid):
1. If P, then Q (P⟹Q)
2. Q is true.
3. Therefore, P is true. (**Incorrect inference!**🚨)
Examples:
- You have a job (Q), so you are a coding instructor (P).
  - Incorrect: You could have many other jobs.
- The code showed an error (Q), so you provided invalid data (P).
  - Incorrect: Other things besides invalid data can cause errors.
Interpretation: This fallacy highlights the difference between a one-to-one and a one-to-many relationship. Looking at our truth table, when P⟹Q is True and Q is True, P could be True (Row 1) or False (Row 3). The argument mistakenly concludes that P must always be True. The uncertainty arises because observing Q as True doesn't uniquely point to P as the cause – there could be many other reasons or paths that lead to Q.
- Think of walking down a forest path, unaware that another trail has merged into yours from behind you. When retracing your steps in reverse, you encounter a split (Q) at that merge and feel disoriented, unsure which path leads back to your start point (P). Just as multiple paths can converge on the same point, multiple causes can produce the same outcome.

Denying the Antecedent (Fallacy)

This is another invalid argument form. Here, we attempt to conclude that the consequent (Q) is false simply because the antecedent (P) is false. It's a fallacy because P being false does not guarantee that Q will also be false. Q could still be true for other reasons, or the implication might not cover all scenarios where Q occurs.

Argument Form (Invalid):
1. If P, then Q (P⟹Q)
2. Not P is true (¬P).
3. Therefore, Not Q is true (¬Q). (**Incorrect inference!**🚨)
Examples:
- You are not a coding instructor (¬P), so you have no job (¬Q).
  - Incorrect: You could have a different job.
- You provided valid data (¬P), so you have no error (¬Q).
  - Incorrect: Valid data doesn't guarantee no error. Other factors like network issues, memory leaks, or non-idempotent operations can still cause errors.
Interpretation: Similar to Affirming the Consequent, this fallacy stems from incorrectly assuming a unique relationship. From our truth table, when P⟹Q is True and P is False, Q could be True (Row 3) or False (Row 4). The argument mistakenly concludes Q must always be False.

Both of these fallacies (Affirming the Consequent and Denying the Antecedent) creep into our thinking when we prematurely assume a single cause for an effect. In complex real-world systems, many factors can lead to an outcome, and narrowing your thinking too soon can lead to missed bugs or incorrect conclusions.

Fallacies and Implication: A Prerequisite

Both the fallacy of affirming the consequent and denying the antecedent assume the underlying implication (P⟹Q) is true.

If this implication is false from the start, there's no logical argument to be made, and thus, no fallacy to speak of.

Exercise: Identifying an Argument Form

Which of the 4 forms of argument is this?

Penguins can’t fly. I can’t fly. Therefore, I’m a penguin.

Hint: Rephrase the first statement into an if-then form.

Denying the Antecedent: A Database Example

We just saw that Denying the Antecedent is a logical fallacy, meaning that even if the initial implication (P⟹Q) is true, concluding ¬Q from ¬P is not a valid inference. To make this abstract concept concrete, and to illustrate why this fallacy can be particularly dangerous in real-world systems like software, let's explore a practical example involving a database.

The implication: If the database is down (P), we’ll see a connection timeout error (Q).

Now, applying the fallacy of Denying the Antecedent, we might incorrectly conclude: If the database is not down (¬P), we will not see a connection timeout error (¬Q). ❌

But even if the database itself is perfectly operational and "not down," you might still encounter a connection timeout error. This could happen due to a variety of other, independent reasons, such as:

Network problems
Firewall rules
The database is up but extremely slow
The query engine is stuck

This specific example of multiple potential causes for a "timeout" highlights a broader, critical skill in software development: thorough case analysis.

This is precisely why technical assessments, especially in areas like algorithms and system design, frequently demand that you consider exhaustive possibilities. For instance, you are often asked to handle base and recursive cases in dynamic programming, or to ensure mutually exclusive and collectively exhaustive coverage when grouping multiple scenarios in problems like interval merging.

Such strong case analysis is vital for minimizing bugs and cultivating an open-minded approach to considering multiple causal paths, driven by experience, curiosity, and a dedication to craftsmanship.

But even perfect case analysis doesn't guarantee a correct implementation. Weak language mastery or mistaken assumptions can still lead to errors, making tests a crucial last line of defense.

Before jumping into applying logic to software testing, let’s practice our agility in conceptually switching between real-world concepts in English and symbols in logic.

Assigning Real-World Meanings to Logic

We must define what P, Q, and P⟹Q refer to when applying logical theory to real-world concepts.

How we define these variables affects our truth tables.

For example:

If P means "valid input," then ¬P means "invalid input."
If P means "invalid input," then ¬P means "valid input."

Imagine we define P = "Good input" and Q = "No Error."

When testing the happy path, we are verifying that the implication P⟹Q (If input is good, then no error) holds true.
When testing the unhappy path (mutation testing, more details later), we are verifying that ¬P⟹¬Q (If input is not good, then an error occurs) holds true.

In any test, a failure indicates that the tested implication is false. This warrants investigation into whether the issue lies with the specification's interpretation, the implementation, or even the test itself.

Applying Logic to Software Testing

Software development relies on constructing systems that behave predictably. Software testing is our primary tool for validating these behaviors. At its core, testing is a process deeply rooted in logical implications, where we propose a hypothesis about our code and then run an experiment (the test) to check its truth.

A test case is carefully designed to evaluate a specific piece of code. This involves:

Setting up Preconditions and Inputs: Before executing the code under test, we meticulously establish a specific environment and provide particular inputs. This includes:
- Function/Method Arguments: The precise values passed into the code being tested.
- System State: Setting up relevant data in a database, preparing the content of a file system, configuring an object's instance variables, or dictating the responses of external services (often through "mocks" or "stubs").
- Environmental Factors: Controlling elements like the current time, specific network conditions, or user permissions relevant to the code's execution. This precise setup ensures that the code runs under defined conditions, allowing us to evaluate its behavior consistently.

Once the setup is complete, the code under test is executed, and its output or behavior is observed. This observation is then compared against an expected result.

To precisely analyze test outcomes, let's establish our specific logical mapping:

P: The code under test is correct for the specific scenario defined by the test. This refers to the actual, objective state of the code's internal logic and implementation when presented with the test's preconditions and inputs. If P is True, the code is without defect for this case. If P is False, there is a bug or deviation.
Q: The test passes. This means the actual output or behavior observed from the code precisely matches the expected outcome defined in our test case. If they do not match, the test fails.
P⟹Q: If the code under test is correct for this specific scenario, then the test will pass. In pure propositional logic, the truth value of P⟹Q is indeed defined by the truth values of P and Q. But in the context of software testing, P⟹Q represents our hypothesis or desired specification for how the code should behave. We don't directly "know" P's truth value beforehand. Instead, the test's execution provides empirical data (the actual Q) that allows us to evaluate whether this hypothesis holds true in practice, and thereby infer the actual state of P.

Understanding this mapping is vital for interpreting test results. Let's examine the different outcomes of a test run, referencing the truth table for P⟹Q:

Row 1: P is True (Code is correct), Q is True (Test passes)
- Interpretation in Testing: Ideal State/Validation
  - This is the desired outcome and strengthens our confidence that the code adheres to its specification.
  - This scenario directly confirms the truth of our hypothesis (P⟹Q).
Row 2: P is True (Code is correct), Q is False (Test fails)
- Interpretation in Testing: Logical Contradiction / Falsification of Hypothesis
  - This row means our overall hypothesis P⟹Q is false for this specific instance.
  - This demands investigation: either our initial assumption that P was True (meaning the code was correct) is wrong (i.e., there's an actual bug, so P is actually False), or the test itself is flawed (its inputs/expectations are incorrect), or the specification is wrong.
  - This is where rethinking of the P⟹Q hypothesis itself happens.
Row 3: P is False (Code is incorrect), Q is True (Test passes)
- Interpretation in Testing: False Positive / Inadequate Test
  - This is a problematic scenario. It implies the test is not robust enough to detect the defect in the code, or the test's expectation is flawed.
  - While P⟹Q remains true vacuously, this outcome is misleading and means the test is not effectively verifying code correctness.
Row 4: P is False (Code is incorrect), Q is False (Test fails)
- Interpretation in Testing: Bug Found / Confirmation of Incorrectness
  - This is a beneficial outcome, as the test has successfully identified a defect.
  - When P is truly False, P⟹Q is vacuously true.
  - This row can represent either a known, intended 'P is False' state (e.g., TDD Red phase) or the actual state discovered via deduction (explained below in Scenario 1).

Note on this Contextualized Truth Table and Probabilistic Nature

This truth table differs from a purely abstract logical truth table by being explicitly contextualized for software testing.

Specific Definitions: Unlike a generic P and Q, here they have precise meanings within the domain of code correctness and test outcomes.
"Interpretation in Testing" Column: This is the key distinguishing feature. It translates the raw logical outcomes of (P, Q, and P⟹Q) into actionable insights and common debugging/development scenarios for software engineers. It explains what it means when a particular row is observed in the context of testing.
Probabilistic Confidence: While formal logic operates in binary (True/False), real-world software testing often involves probabilistic confidence. A test doesn't provide absolute logical proof of correctness (for example, a passing test doesn't guarantee P is 100% True due to the possibility of undiscovered bugs or false positives). Instead, test results increase our confidence that the code is correct, or provide strong evidence that it is incorrect. Testing is fundamentally about reducing uncertainty and increasing the probability that our code functions as intended.

Let's now explore how these logical outcomes are interpreted in two common testing scenarios:

Scenario 1: Debugging an Unexpected Defect (Applying Modus Tollens)

This scenario occurs when a test that was previously passing, or a newly written test that we strongly trust as a precise and correct specification, unexpectedly fails. In this context, we assume the validity of the implication P⟹Q for this specific test case, treating it as an unbreakable rule for how correct code should behave.

Our Core Premise (Trusted Specification): We operate under the assumption that the implication "P⟹Q" ("If the code is correct for this scenario, then the test passes") is True for this specific test. Our confidence stems from the test's meticulous design, its history of passing, or its role in a well-established regression suite.
Test Execution and Observation: We run the test, which has its preconditions and inputs set.
- If the Test Fails (Q is False): This is the key observation. Since we trust our premise that P⟹Q is True, and we observe ¬Q (the test fails), we are logically compelled to deduce that our initial belief about P (the code being correct for this scenario) must be false.
  - Application of Modus Tollens:
    - Premise 1: If the code is correct for this scenario (P), then the test passes (Q). (P⟹Q, assumed true as a trusted specification).
    - Premise 2: The test did not pass (¬Q).
    - Conclusion: Therefore, the code is not correct for this scenario (¬P).
  - Outcome: This inference directly points us to a defect in the code. The test's failure, given its trusted nature, reveals that the actual state of the code for this scenario is P is False. This effectively places the scenario in Row 4 (P False, Q False) of our truth table, confirming the presence of a bug that needs fixing. This is typical in regression testing, where a previously correct feature suddenly breaks.

Scenario 2: Validating/Refining the Specification (Falsifying P⟹Q or Confirming Known Incorrectness)

This scenario arises when a test fails, and our primary focus is not immediately on debugging the code as if it's a regression. Instead, it's on understanding why the P⟹Q relationship (our hypothesis for this specific behavior) isn't holding, or simply confirming an expected failure. This can involve questioning the test itself, the underlying requirements, or confirming a deliberately incorrect state of the code.

Our Hypothesis (Being Challenged or Confirmed): We are either actively evaluating the validity of the implication "P⟹Q" for a specific behavior, or we are running a test against code we know is incomplete or incorrect.
Test Execution and Observation: We run the test with its defined preconditions and inputs.
If the Test Fails (Q is False): The interpretation here depends on our prior knowledge or intent about the code's state (P):
- Sub-scenario 2A: Falsifying P⟹Q and Rethinking Specification (Corresponds to Row 2: P True, Q False):
  - We observe Q is False (the test fails).
  - If we then examine the code and the requirements, and we conclude that the code should have been correct for this scenario (meaning, our expectation/belief was P is True), then the test result means the specific instance of our hypothesis "P⟹Q" is FALSE.
  - This direct falsification reveals a contradiction. We must then investigate:
    - Is our initial belief that P was True mistaken (that is, is there a genuine bug in the code that makes P actually False, moving this to a Row 4 scenario)?
    - Or, is the test itself incorrect (its inputs or expected output are wrong), meaning our P⟹Q premise needs to be re-evaluated and corrected?
    - Or, have the underlying requirements changed or been misunderstood?
  - Outcome: This critical outcome prompts us to "rethink" – either the code needs fixing, or the test needs adjusting, or the specification needs clarification. This is common in exploratory testing or when working with new/evolving features where the exact behavior is still being defined.
- Sub-scenario 2B: Confirming Known Incorrectness (Corresponds to Row 4: P False, Q False):
  - We observe Q is False (the test fails).
  - We already know or intentionally designed the code to be incorrect for this scenario (that is, we are actively developing a feature and haven't written the full code yet, or we're running a test against a known, un-fixed bug, so our expectation is P is False).
  - The test result simply confirms our prior knowledge that P is False. The test correctly highlights the missing or incorrect behavior. In this case, the P⟹Q implication is vacuously true, and the test effectively served its purpose of showing the existing defect.
  - Outcome: This is typical in Test-Driven Development (TDD) in the Red phase, where a failing test for a not-yet-implemented feature confirms the "P is False" state, guiding development to make P True. It also applies when verifying that a bug fix indeed works: the test initially fails (confirming the bug), and then passes after the fix (confirming P is now True).

A Closer Look at Testing

The Illusion of Correctness: Affirming the Consequent

Consider a common scenario where a test passes, seemingly validating our code:

def get_user_role(user_id):
    if user_id == 42:
        return "admin"
    return "guest"

# test
assert get_user_role(42) == "admin"

Here, our implicit claim (the specification) is: If the code is correct (P), then the output will match the expectation (Q).

In this example, the test passes – the output is "admin" (Q), but can we definitively conclude that the function is correct (P)? Not necessarily.

This scenario often exemplifies the logical fallacy of affirming the consequent. We see the desired outcome (Q) and mistakenly assume that our specific intended cause (P, the correctness of our specific implementation path) was the reason.

The Problem: What if the real condition for an "admin" role should be checking a database, but we have temporarily hardcoded the value for testing? The test would pass, but the correctness is illusory. If we see P as false because the code did not implement the behaviour from the full specification, this corresponds to Row 3 (P False, Q True: False Positive) in our truth table.

As I mentioned before, deliberately implementing ¬P works well if ¬Q is observed, but is not useful, or even erroneous, if Q is observed.

Even without hardcoding, the output might match by coincidence, or because of factors outside the direct logic we intended to test. This can happen due to:

Default behavior: A broader system default might produce the expected output.
Caching: A previous successful operation might have cached the result, bypassing the actual logic.
Fallback logic: An unintended fallback mechanism produces the correct output despite an error in the primary path.
Test harness bugs: Flaws in the testing setup itself might obscure real issues.

The Role and Risks of Test Doubles

The challenges highlighted above are particularly relevant when using test doubles, such as Stubs and Mocks. These are artificial components that replace real dependencies (for example, databases, external APIs, time-sensitive operations) during testing.

Stubs focus on state: they provide pre-programmed fake data or return values to get the rest of the code under test working predictably, like the get_user_role example
Mocks focus on behavior: they allow you to verify interactions, such as the number of calls made to a certain API, or how control flow flows through specific parts of the system.

Both remove external dependencies, allowing you to isolate and focus on the internal logic of the code without noise or side effects. But using them without understanding their limitations can lead to false confidence.

If a test double simulates a "correct" response, but the real dependency it replaces has a bug, or the way the main code interacts with that dependency is flawed, the test will pass (Q is True) – yet P (the code's overall correctness in a real environment) might be False, leading to a dangerous false positive.

Whether you encounter such logical fallacies in your testing depends on precisely what behavior or state you are attempting to verify, and whether you are over-interpreting the test results.

Test Scope and Interpretation

The choice of testing scope – from narrowly focused unit tests to broader integration tests, system tests, user acceptance tests (UAT), and even testing in production – represents a continuum. On this spectrum, various trade-offs are involved, especially concerning the effort-reward ratio. This effort is influenced by factors like individual developer skill, company engineering practices (for example, responsibility split between feature developer and dedicated tester roles), and industry regulations.

Generally:

Smaller-scoped tests (for example, unit tests) have fewer assumptions baked in and a shorter chain of logical implications. This translates to less risk of committing fallacies in both test implementation and test result interpretation. They are excellent for quickly verifying isolated units of code.
Larger-scoped tests (for example, end-to-end integration tests) incorporate more real-world complexities and dependencies. While providing higher confidence in the system's overall behavior, they inherently increase the potential for confounding factors that can lead to false positives or make debugging more challenging.

Being acutely aware of the assumptions implicit in each test, at every scope level, is paramount. Passing tests for the wrong reasons will inevitably cause problems down the road.

Debugging, Observability, and Mental Models

Failing tests are not failures of the testing process but are, in fact, incredibly valuable learning moments. They represent opportunities to:

Run focused debugging experiments to pinpoint the exact cause of the failure.
Refine your mental model of the code-to-outcome (P⟹Q) link. A failing test (where Q is False) tells you that your current understanding of P, or of the P⟹Q relationship, is flawed. Use this feedback to update your understanding of the code's actual behavior.
Improve both the code and the tests themselves.

Enhance system observability to better detect and confirm outcomes (Q). The more clearly, from multiple angles, and through diverse methods we can observe Q (for example, logs, metrics, tracing, output inspection), the more confident we can be in its causes and, by extension, the actual state of P.

Crucially, avoid blindly fixing tests just to make them pass. Always ensure you thoroughly understand why a test failed and update your P⟹Q model accordingly. The ultimate goal is not just to fix current bugs, but to prevent them in the future by continually strengthening both the correctness of the code and the verifiability of its behavior.

Falsifiable Tests Reveal Regressions

Beyond avoiding false positives (where the code is incorrect but the test passes), a good test must also be falsifiable. This means the test must be genuinely capable of failing under certain (incorrect) conditions. An unfalsifiable test is a broken test – it cannot serve its purpose of revealing regressions or confirming the presence of bugs.

While we strive for the implication P⟹Q to hold true for all the scenarios we care about, it may not be true for all cases due to unforeseen or mistaken assumptions, or simply because the code is incorrect. The test's ability to demonstrate this incorrectness by failing under specific, well-defined conditions makes it profoundly valuable.

Some common culprits for unfalsifiable or "bad" tests include:

Vague or Untestable Specifications: Statements like "The system should behave well under most conditions," "It shouldn't crash randomly," or "The algorithm is robust" lack clear, measurable criteria. It's impossible to design a test that definitively passes or fails against such statements, thus rendering them effectively unfalsifiable.
Broken Implementations of the Test Suite: The test code itself might be flawed, perhaps due to logical errors or control flow issues that prevent assertions from ever being reached or correctly evaluated, inadvertently taking the same passing path regardless of the code under test.
Insufficient Test Data or Edge Cases: If tests only cover "happy path" scenarios and fail to include challenging inputs or boundary conditions, they might pass for incorrect code that only breaks under specific, untested circumstances.

A robust specification clearly defines what constitutes success and failure. Correspondingly, a good test suite correctly implements that specification, making its tests both accurate and truly falsifiable.

Take a step back

Critical thinkers might observe that the application of the four fundamental logical argument forms to coding scenarios, as initially presented, could be misleading in the complexities of real-world software.

The next section shows some nuances that arise when we transition from the clear-cut rules of formal logic to the often messy reality of software development.

Specifically:

The first two points below show why the seemingly valid arguments of Modus Ponens and Modus Tollens may not always lead to reliable conclusions when applied to coding scenarios.
The last two points below show why the two common logical fallacies, Affirming the Consequent and Denying the Antecedent, may actually provide correct insights under specific real-world coding conditions.

Revisiting the Four Statements for Coding

Here are the four arguments and their associated coding examples:

Modus Ponens: If you provide invalid input data (P), the code will show an error (Q).
Modus Tollens: There are no error messages (¬Q), so the input data is valid (¬P).
Affirming the Consequent (Fallacy): The code showed an error (Q), so you provided invalid data (P).
Denying the Antecedent (Fallacy): You provided valid data (¬P), so you have no error (¬Q).

Now, let's dive into the nuances of each:

Modus Ponens

Our coding example: If you provide invalid input data (P), then the code will show an error (Q).
Why it may not always hold: This application of Modus Ponens assumes that either your code or any third-party code it relies upon will always properly detect and explicitly raise exceptions or show errors on bad data. In reality, systems might automatically fix or sanitize bad input, silence errors, or simply proceed with unexpected behavior without explicitly signaling an error, leading to a passing (or non-failing) state (¬Q) even when P (invalid input) was true.

Modus Tollens

Our coding example: There are no error messages (¬Q), so the input data is valid (¬P).
Why it may not always hold: This application of Modus Tollens assumes there are no automatic mechanisms within the system to fix or silence bad input before errors are typically displayed. If such "silent correction" or "error suppression" occurs, you might observe no error messages (¬Q), but the input data could still be invalid (P), rendering the conclusion (¬P) false despite the premise (¬Q) being true. This highlights the dangers of incomplete observability.

Affirming the Consequent (Fallacy)

Our coding example: The code showed an error (Q), so you provided invalid data (P).
Why it may actually be correct: While logically a fallacy, in specific, highly constrained real-world conditions, this inference can gain practical validity. If the error message is so uniquely and specifically defined that it can only be caused by invalid input data (P) and no other known factor, then this statement can become reliable. This is rare and typically requires meticulous error handling design where each error message maps unambiguously to a single root cause.

Denying the Antecedent (Fallacy)

Our coding example: You provided valid data (¬P), so you have no error (¬Q).
Why it may actually be correct: Although a fallacy in general logic, this inference can hold a high degree of practical confidence under certain programming paradigms (Functional Programming). If the code is sufficiently simple, purely functional (meaning outputs depend only on inputs and have no side effects), and has no external dependencies (like network or database interactions), then the absence of invalid data (¬P) can indeed make us reasonably confident that there will be no errors (¬Q). The lack of external variables and internal state makes the code's behavior highly predictable and directly tied to its inputs.

You may now be thinking: what’s the point of studying logic if it has so many loopholes and edge cases when applied to coding?

The Missing Ingredient – If and Only If

In our exploration of logical implications, we've focused primarily on the unidirectional relationship P⟹Q ("If P, then Q"). This statement tells us what happens if P is true, but it remains silent on whether Q only happens when P is true. It's like saying, "If it rains, the ground gets wet." This is true, but the ground can also get wet if a sprinkler is on, even if it's not raining.

But in many critical contexts, especially in rigorous scientific theories and robust software systems, we often seek a much stronger relationship: one where the truth of Q absolutely depends on the truth of P, and vice versa. This powerful bidirectional relationship is captured by the phrase "If and Only If" (P⟺Q).

What "If and Only If" Means: A Stronger Statement

When we assert "P⟺Q", we're making two distinct claims simultaneously:

If P, then Q (P⟹Q): P is a sufficient condition for Q. Whenever P is true, Q must also be true.
If Q, then P (Q⟹P): P is also a necessary condition for Q. Whenever Q is true, P must also be true. In other words, Q cannot be true without P being true.

Notice the significant increase in the strength of the statement. "If P, then Q" merely states a consequence. "P⟺Q" declares a definitive equivalence, where P and Q are inextricably linked. They rise and fall together – one cannot be true without the other being true, and one cannot be false without the other being false.

Bidirectional Truth Table: Unambiguous Relationships

Let's construct the truth table for P⟺Q to clearly see this strong relationship.

P⟺Q is logically equivalent to (P⟹Q)∧(Q⟹P).

Q, Q->P, P<->Q" class="image--center mx-auto" width="1226" height="323" loading="lazy">

Creating the Table (columns 4 and 5 are new):

Q⟹P (Column 4): We apply the standard implication rules, but with Q as our "if" and P as our "then." For instance, in Row 3, Q is True and P is False, so Q⟹P is False.
P⟺Q (Column 5): This is the logical AND of the P⟹Q and Q⟹P columns. For P⟺Q to be True, both component implications must be True, which explains why you see less Trues in the bidirectional implication compared to any of the unidirectional implications.

Implications for the Two Common Fallacies

The clarity provided by "If and Only If" is particularly powerful in preventing the very logical fallacies we discussed earlier: Affirming the Consequent and Denying the Antecedent. These fallacies arise from the incorrect assumption that an "if-then" statement implies an "if and only if" relationship.

Let's revisit them with the lens of P⟺Q If and Only If you provided invalid data (P), then the code will show an error (Q):

Affirming the Consequent: No More Ambiguity

The Fallacy (assuming unidirectional P⟹Q):
- If the code showed an error (Q), then you provided invalid data (P).
- Previously, when P⟹Q was True and Q was True, P could be True (Row 1) or False (Row 3). This ambiguity led to the fallacy.
With P⟺Q:
- Now, look at the P⟺Q column in the table. When P⟺Q is True and Q is True (Row 1), P is unambiguously True. The confusion from Row 3 is gone because if Q were True while P was False, P⟺Q would be False (as Q⟹P would be False), thus making that row irrelevant for valid modus ponens inference under the P⟺Q premise.
- In a system designed with P⟺Q in mind, knowing that Q is True (observing an error) would force the conclusion that P is True (invalid data is the cause), assuming the "if and only if" relationship holds true for that specific system design.

Denying the Antecedent: Unmistakable Consequences

The Fallacy (assuming unidirectional P⟹Q):
- You provided valid data (¬P), so you have no error (¬Q).
- Previously, when P⟹Q was True and P was False, Q could be True (Row 3) or False (Row 4). This ambiguity led to the fallacy.
With P⟺Q:
- Now, when P⟺Q is True and P is False (Row 4), Q is unambiguously False. The problematic scenario from Row 3 (where P was False but Q was True) is irrelevant here because P⟺Q would be False in that case (specifically, Q⟹P would be False).
- If your system genuinely adheres to "P⟺Q", then knowing that P is False (valid data provided) guarantees that Q is False (no error messages).

Practical Mitigation in Coding

The insights from "If and Only If" are more than just theoretical. Practically, both fallacies (Affirming the Consequent and Denying the Antecedent) can be mitigated by striving for conditions that approximate an "if and only if" relationship in your code and tests.

Focused Unit Tests

Design unit tests that are so granular and isolated that they effectively aim to establish an "if and only if" scenario for a tiny piece of logic. By thoroughly mocking or controlling all external dependencies and environmental factors, you reduce the impact of "other causes."

If your test for a specific input passes, you want to be as confident as possible that it passed only because the code handled that specific input correctly, and not due to some irrelevant side effect. Similarly, if it fails, you want to be sure that the failure points directly to the intended logical path.

Exception Handling and Specificity

Instead of catching broad Exception types, catch and handle specific exceptions. This helps differentiate between various "causes" (P1,P2,…) that might lead to a generic "error" (Q). The more precise your error handling, the closer you get to a scenario where "If X error, then Y specific cause," moving towards a bidirectional understanding of error conditions.

Test-Driven Development (TDD) and Mutation Testing

These methodologies inherently push towards P⟺Q thinking. TDD encourages writing a failing test first (¬Q), which then necessitates a specific code change (P) to make it pass.

Mutation testing, which we'll explore further, takes this a step further by ensuring that your tests are robust enough to fail when code is subtly altered (that is, proving that ¬P leads to ¬Q, and thus, that the original P was indeed necessary for Q).

By consciously aiming for "if and only if" relationships in your code's design and your testing strategies, you can build systems that are not only predictable but also much easier to debug and reason about, moving beyond mere correlation to a deeper understanding of cause and effect.

Callback to Mutation Testing

In the earlier section on Assigning Real-World Meanings to Logic, we discussed:

When testing the happy path, we are verifying that the implication P⟹Q (If input is good, then no error) holds true.

When testing the unhappy path (mutation testing), we are verifying that ¬P⟹¬Q (If input is not good, then an error occurs) holds true.

This dual view is key to understanding how mutation testing contributes to software correctness.

Mutation Testing: Testing the Tests

Mutation testing deliberately introduces small faults (mutations) in the code and checks whether the test suite detects them by failing. This process assesses not the code, but the tests themselves.

In a robust test suite, we strive for two ideal conditions:

All correct implementations should pass the tests.
All incorrect implementations should fail the tests.

If a mutated (wrong) version of the code is introduced and causes no test failures, that defeats the fundamental purpose of testing. It means your tests aren't sensitive enough to catch a deviation from correctness. Mutations reveal hidden assumptions or gaps in your test coverage, acting as a sensitivity probe for your test suite.

Example code mutations:

Changing an arithmetic operator (+ to -, > to >=).
Flipping a boolean condition (true to false).
Deleting or duplicating a statement.
Modifying a constant value.

Common Python mutation testing tools:

mutmut uses Python’s built-in ast module.
cosmic-ray uses parso, which provides a more complete AST.

These tools rely on abstract syntax trees to surgically mutate code.

You can even swap out underlying AST libraries for different precision or completeness: https://github.com/boxed/mutmut/issues/281

Logic Behind Mutation Testing

Let's formalize the logical mapping of mutation testing, recalling our definitions:

Let P: Code is correct.
Let Q: Tests pass.

Standard happy path testing primarily checks that P⟹Q – "if the code is correct, then tests pass."

Mutation testing focuses on the other side of the coin: we intentionally make ¬P true (by introducing a fault), and then we expect ¬Q (the tests should fail). This process rigorously checks whether the implication ¬P⟹¬Q ("if the code is not correct, then the tests fail") holds true for your test suite.

But there's a deeper, more powerful logical implication here:

As we learned earlier, the statement ¬P⟹¬Q is logically equivalent to its contrapositive, Q⟹P.

So, by successfully verifying that introducing a fault (¬P) leads to a test failure (¬Q), we are simultaneously validating the contrapositive: if tests pass (Q), then the code must be correct (P).

This is incredibly significant! It moves us much closer to establishing a bidirectional guarantee between our code and our tests: P⟺Q (code correctness is tightly coupled with test success). Mutation testing helps us confidently eliminate false positives in the test suite – situations where Q is true (the test passes) but P is false (the code is actually incorrect).

In a world where LLMs help us write and refactor code quickly, having this "if and only if" confidence in our test suite is invaluable for ensuring the generated or refactored code truly meets expectations.

Clarifying the Kinds of Failures

In software, we typically categorize errors into three main types:

Syntax errors: Violations of the language's grammatical rules (for example, missing colon, invalid keyword). These prevent the code from running at all.
Runtime errors: Errors that occur during program execution, often due to unexpected conditions (for example, TypeError, AttributeError, ZeroDivisionError).
Logic errors: The program runs without crashing, but it produces an incorrect result or behaves in a way that doesn't match the intended specification (for example, wrong algorithm, wrong return value).

Mutation testing focuses on logic errors – failures where the program runs, but produces incorrect results. These are usually caught via AssertionError in the "Assert" phase of the Arrange–Act–Assert (AAA) testing pattern.

You could argue pedantically that AssertionError is a runtime error, but in testing, we treat it as a signal for logical failure:

"The function ran, but the output didn’t match the expected behavior."

Mutation testing assumes that syntax and runtime errors are already handled. Its purpose is to validate whether the test suite reliably catches logical misbehavior.

A Deeper Falsification Perspective

Now, let's connect mutation testing back to Karl Popper's principle of falsification, which we introduced earlier in the context of scientific reasoning. Recall that Popper argued scientific theories gain strength not by being "proven," but by surviving rigorous attempts to disprove them. The core idea of falsification logic is that to disprove an implication like P⟹Q, you only need to find one instance where P is True and Q is False.

Mutation testing applies this same powerful principle, but to our test suite's effectiveness:

Instead of trying to prove directly that our tests are perfect, mutation testing takes a falsification approach to the implication ¬P⟹¬Q ("If the code is incorrect, then the tests fail"). It actively tries to falsify this crucial relationship.

If we introduce a mutation (making ¬P true, that is, the code is now incorrect) but the existing test suite still passes (meaning Q is true), then we have found an instance where:

¬P is True (the code is incorrect due to the mutation).
Q is True (the test still passes).

In this scenario, the implication ¬P⟹¬Q is falsified because we have a True antecedent (¬P) leading to a False consequent (¬Q is false, because Q is true).

And, critically, if ¬P⟹¬Q is falsified, then its logically equivalent contrapositive, Q⟹P ("If the tests pass, then the code is correct"), is also falsified. This means we can no longer trust that a passing test suite reliably indicates correct code. Our desired P⟺Q relationship is broken – the test suite is no longer fully effective at guaranteeing correctness.

By pushing for zero surviving mutants, mutation testing forces us to minimize the surface area of these "hidden assumptions" in our test suite. It demands highly sensitive and specific tests that can pinpoint even subtle logical flaws, thereby moving us closer to building truly resilient systems.

Comparing TDD (Red Phase) and Mutation Testing

Both methodologies, albeit through different means and at different stages of the development cycle, aim to establish confidence in the ¬P ⟹ ¬Q relationship.

Key Differences Summarized:

Feature	TDD (Red Phase)	Mutation Testing
Primary Goal	Drive new code development. Confirm a bug/feature.	Evaluate the quality/completeness of existing tests.
Code State	Production code is incomplete or buggy.	Production code is (assumed to be) correct.
Test State	The new test is expected to fail.	Existing tests are expected to fail (due to mutants).
Initiator	Developer wanting to add functionality/fix bug.	Tool that inserts artificial bugs into code.
"Bugs"	Actual, intended bugs or missing features.	Artificial, subtle changes to the code.

Toward If-and-Only-If Confidence

Ultimately, the goal in software development is to establish if-and-only-if relationships whenever possible, both in the code implementation and especially in the sensitivity of the test suite to the code under test.

This means if a certain condition (P) is true, then a specific outcome (Q) must occur, and if Q occurs, then P must have been the cause. Achieving this level of clarity comes from:

A deep understanding of the problem.
Aligned expectations during requirements gathering.
Logical analysis and interpretation of well-designed experiments.
Adherence to Single Responsibility Principle in SOLID
Rigorous tests with meaningful coverage.

This allows us to understand how control flow and data flow work with greater depth and confidence, leading to better inferences throughout the entire software development lifecycle.

Real-World Challenges

While striving for perfect "if-and-only-if" relationships provides a powerful logical ideal, the messy reality of modern software development presents significant hurdles. The very characteristics that make large systems powerful and scalable – their intricate interconnections and inherent dynamism – simultaneously obscure clear cause-and-effect relationships, making precise logical reasoning and debugging an ongoing battle.

A Web of Complexity

Fan-In, Fan-Out: The Nature of Modern Systems

Any reasonably large software system rarely operates through purely linear control and data flows. Fan-out and fan-in patterns – where many components are called and then their results merged – are inevitable.

For example:

In ETL pipelines, data may be ingested from multiple sources (external APIs, CSVs) and logged to multiple destinations (files, databases).
In concurrent programming, Python’s ProcessPoolExecutor splits data into chunks processed in parallel, then recombines the results.

SRP Meets Real-World Boundaries

Just as functional programming must eventually perform I/O, the Single Responsibility Principle (SRP) runs into real-world boundaries, whether conceptual or infrastructural. At some point, something must glue these isolated units together.

Orchestration logic might live in a single function, span multiple files, or even distribute across microservices and machines communicating over networks. While this decomposition enhances modularity, it also increases surface area for bugs involving:

Side effects: Unintended changes to system state outside a component's explicit outputs.
Circular dependencies: Components relying on each other in a loop, leading to difficult-to-trace behavior.
Interface drift: Changes in one component's input/output expectations not being correctly reflected elsewhere.
Race conditions: Timing-dependent bugs in concurrent operations.
Serialization issues: Problems translating data between different formats or systems.
Network unreliability: Unpredictable latency, packet loss, or disconnections in distributed systems.

The Double-Edged Sword of Abstraction

This web of dependencies is the price of progress, made manageable only through better tooling and abstractions.

If boundaries are well-designed, observable, and testable, they enable asynchronous collaboration, improve long-term maintainability, and increase developer confidence. (See GitHub Playbook in References)
If systems lack architectural coherence or fall behind evolving needs, they calcify into technical debt that demoralizes even the most motivated teams.

Clean Code Is Contextual

While abstractions and orchestration help manage complexity, overusing design patterns or creating unnecessary class layers can introduce needless indirection. This is a common counterargument to architectural purism.

Ultimately, what counts as "clean code" is context-dependent. It varies with programmer skill, the tooling at hand (linters, tests, Copilot), and whether the project is a throwaway script or a multi-year infrastructure investment. Architectural practices like SRP should evolve alongside those constraints.

The Butterfly Effect of Bugs

From SRP to Reasoning Chains

Previously, we focused on simple, direct cause-effect logic (P ⟹ Q), but real-world systems are messier.

The more we adhere to SRP through small, focused functions, the more we create longer chains of logic. This improves separation of concerns but also extends the reasoning required to debug behavior.

Debugging in a Causal Fog

A seemingly minor trigger (O) can cascade through a chain like O⟹P⟹Q⟹R, which we may not fully understand due to knowledge silos, evolving requirements, or runtime dynamism.

Even when we understand the components, precisely identifying “P” is hard, much like how redefining a research question shifts the statistical population being studied. In complex systems with feedback loops (recommender engines), there might not be a single "root cause" at all.

Short-Term Triage vs. Long-Term Insight

Finding the true origin of a bug often demands experimentation, telemetry, and broad system insight. These investigations produce robust, future-proof fixes but take time.

In on-call scenarios, however, urgency reshapes priorities. Fast mitigations and clear communication often take precedence over deep diagnosis.

Masked by Design and Debt

As systems scale, failure stops looking like a crash. Instead, it shows up as a retry spike, a slow metric drift, or silent fallback behavior.

Modern fault-tolerant systems, built with retries, failovers, circuit breakers, and autoscaling, are designed to recover quickly. This resilience often masks deeper problems, delaying detection for weeks and making root cause analysis harder.

Operating in non-deterministic environments with flaky networks, race conditions, or dynamic routing adds further ambiguity. Small symptoms become harder to link back to specific causes.

Compounding this, technical debt driven by weak technical leadership, shifting priorities or time pressure weakens the system’s observability and test coverage. Teams inherit brittle, poorly understood code, making it hard to draw clean lines between cause and effect.

Even the best engineers struggle in such conditions. When a system resists clarity, it doesn’t just block debugging. It erodes trust, slows learning, and fuels long-term burnout.

Glimmers of Hope: Tools and Practices for Clarity

Despite these challenges, several strategies and practices offer a path toward more robust and understandable software.

Leveraging Design Patterns

Design patterns offer a shared vocabulary and time-tested strategies for structuring systems. When applied well, they tame complexity, reduce technical debt, and make behavior more predictable.

They also tend to concentrate similar failure modes. The same bug might appear across companies or industries, creating a wealth of prior art and solution playbooks. Familiarity with patterns can accelerate debugging and deepen shared understanding across teams.

Nurturing Expert Mentorship

Promoting mentors based on real technical impact instead of tenure builds stronger teams and avoids the Peter Principle (people in a hierarchy tend to rise to a level of respective incompetence).

Great mentors teach more than skills – they model falsifiability, independent thinking, and an ability to reason under uncertainty.

They help others challenge assumptions, navigate tradeoffs, and grow both technically and interpersonally. In systems where root causes are murky, this kind of leadership is essential.

One of the most powerful techniques that scales from mentorship to code is falsification: the disciplined search for counterexamples. Whether applied in design reviews, debugging sessions, or automated tests, this mindset anchors reasoning in reality.

The Power of Falsification in Testing

The deliberate search for counterexamples is core to building reliable systems.

In algorithm design, testing edge cases is just falsification in disguise: finding where your logic breaks.
In code, fuzz testing (Atheris) throws diverse inputs at functions to expose falsifying examples.
Property-based testing (Hypothesis) goes further by generating inputs that satisfy certain rules, then shrinks failures to their minimal form. This greatly improves reproducibility and helps stress-test concurrency issues.

The more rigorously we attempt to falsify our assumptions, the more confidently we can reason about behavior using tools like Modus Ponens and Modus Tollens.

Assumptions are always present in software to simplify complexity. The question is whether they're explicitly codified in tests or left hidden and fragile.

Of course, no test is ever bulletproof: our assumptions could be mistaken, or the world could change. That’s why critical thinking, discerning "what should be" versus "what is", remains essential as newer generations increasingly rely on AI tools like Large Language Models.

This deliberate, falsification-driven approach is paramount for building reliable software. It underpins sophisticated testing techniques designed to expose hidden assumptions and break our logical chains.

While testing helps us uncover where our reasoning might falter, some domains demand an even higher degree of certainty. For those critical systems, we turn to the ultimate tools for logical rigor: Proof Assistants.

Proof Assistants

While traditional testing and fuzzing are powerful for finding bugs, they fundamentally cannot guarantee correctness for all possible inputs or scenarios. They can only prove the presence of bugs, not their absence.

To achieve formal, mathematically verified proofs of program behavior – providing the strongest possible guarantees – we turn to proof assistants. These tools allow us to build step-by-step logical proofs, ensuring that a program or system design adheres to its specification with absolute rigor.

Prolog

Prolog offers a relatively straightforward entry point into the world of logic programming and theorem proving. SWI-Prolog is a common interpreter (a REPL, or Read-Eval-Print Loop) for Prolog.

You interact with Prolog by providing it with a knowledge base composed of facts and rules (which are a type of logical clause called Horn clauses). You then pose queries.

Installing SWI-Prolog

You can download SWI-Prolog from its official website: https://www.swi-prolog.org/download/stable
Follow the instructions for your operating system (Windows, macOS, or Linux).

On Ubuntu/Debian, you can usually install it via:

sudo apt update
sudo apt install swi-prolog

Using Prolog: REPL vs. File

REPL (swipl) is best for: Quick, interactive tests of single facts or rules, and posing queries to an already loaded knowledge base.
A File (.pl extension) is best for: Defining your entire knowledge base (multiple facts and rules) and storing your program for reusability. This is the standard way to work with Prolog for anything beyond a few lines.

Example: A Simple Knowledge Base

Let's define a knowledge base to represent who has a job and who is a coding instructor.

1. Create a file named knowledge.pl with the following content:

% knowledge.pl
% This file defines a small knowledge base in Prolog.
% In Prolog, all statements (facts and rules) about the same predicate
% (identified by its name AND number of arguments, e.g., 'has_job' with 1 argument is 'has_job/1')
% must be written consecutively without other predicate definitions in between.

% --- Definitions for the 'has_job' predicate (takes 1 argument) ---

% Fact: Alice has a job.
has_job(alice).

% Fact: Bob has a job.
has_job(bob).

% Rule: Anyone (represented by variable X) has a job IF they are a coding instructor.
% ':-' means 'if'. 'X' is a variable (starts with uppercase).
has_job(X) :- is_coding_instructor(X).

% --- Definitions for the 'is_coding_instructor' predicate (takes 1 argument) ---

% Fact: Alice is a coding instructor.
is_coding_instructor(alice).

What each line does:

Lines starting with %: These are comments for human readability, ignored by Prolog. They explain the file's purpose and key rules like predicate grouping.
has_job(alice). / has_job(bob).: These are facts. They assert simple truths, like "Alice has a job." The . at the end is mandatory for every statement.
has_job(X) :- is_coding_instructor(X).: This is a rule. It states a conditional truth: "For any X, X has a job if X is a coding instructor." X is a variable (always starts with an uppercase letter), and :- means "if." This rule allows Prolog to deduce new information.
is_coding_instructor(alice).: Another fact, asserting "Alice is a coding instructor." It's placed after all has_job/1 clauses to satisfy Prolog's grouping rule.

2. Load and Query in the REPL:

Open your terminal and type swipl. Once at the ?- prompt, load the file and then pose your queries:

$ swipl
?- [knowledge].   % Load the 'knowledge.pl' file (omit .pl, use square brackets and a period)
% Press Enter. Prolog will confirm it loaded the file, e.g., '% knowledge.pl compiled...'
True.

?- has_job(alice). % Query: Does Alice have a job?
% Press Enter. Prolog gives you a solution, then waits.
True.              % Output: Yes, because it's a fact.
% After 'True.', you'll see the '?- ' prompt again, indicating Prolog is ready for your next query.
% If there were multiple ways to prove 'True.', Prolog would present the first 'True.' then wait for you to press ';' for alternatives, then Enter to confirm the final 'True.' or 'False.'.

?- has_job(carol). % Query: Does Carol have a job?
% Press Enter.
False.             % Output: No, Prolog cannot prove it from its knowledge.

?- has_job(X).     % Query: Who has a job? (Find values for X)
% Press Enter
X = alice ;        % Prolog finds Alice as the first solution. Type ';' and press Enter to ask for the next solution.
X = bob ;          % It finds Bob. Type ';' and press Enter for the next solution.
X = alice          % It finds Alice again (this time deduced via the rule and is_coding_instructor(alice)).
% Press Enter. This accepts the current set of solutions and stops searching for more.
False.             % Output: Indicates no more solutions found after the last 'Enter' (or if you explicitly chose not to search further).

?- halt.           % Type 'halt.' to exit the Prolog REPL cleanly.
% Alternatively, you can often use Ctrl+D (press and hold Ctrl, then D) to exit most REPLs.

The Prolog example clearly demonstrates:

"Is P(X) true for a specific X?": Shown by ?- has_job(alice). (returns True.) and ?- has_job(carol). (returns False.).
"Is there an X for which P(X) is true?": Shown by ?- has_job(X). (provides solutions like X = alice, X = bob).

Prolog Limitations

Prolog's limitations become evident when attempting to reason about falsity or non-existence. You cannot directly ask "Is there any X for which P(X) is false?"

Instead, Prolog operates on the principle of negation as failure. This means that if Prolog cannot prove a statement, it considers that statement false.

For example, if you ask ?- \+ has_job(carol). (meaning "Is it not true that Carol has a job?"), Prolog will say True, because it simply cannot find any proof that Carol has a job in its knowledge base.

This is a significant distinction: it doesn't mean Carol definitely doesn't have a job, nor does Prolog provide a formal counterexample. It merely reflects a lack of provable information.

This fundamental constraint means Prolog, while powerful for logic programming, falls short of being a full-fledged proof assistant for comprehensive formal verification.

Coq

After experimenting with Prolog and seeing its limitations, you can move on to a more powerful proof assistant like Coq. Coq is employed in safety-critical domains where absolute mathematical certainty is paramount. coqtop is the standard REPL for Coq.

A fundamental difference from Prolog is Coq's lack of a Closed World Assumption. In Coq, anything not explicitly proven is simply unknown, not automatically false.

Unlike Prolog, Coq's primary purpose isn't solving computational problems by searching a knowledge base. Its true power lies in its ability to construct and verify formal mathematical proofs and programs with absolute rigor. Its interaction involves managing a proof state (your remaining goals) and applying tactics (logical inference steps) until the proof is complete.

Installing Coq

Coq can be installed in several ways, often via package managers or a tool called opam (the OCaml package manager, as Coq is written in OCaml).

Official Downloads: Visit the Coq website for detailed instructions for your OS: https://coq.inria.fr/download
Using a system package manager (for example, Ubuntu/Debian): Bash
```
  sudo apt update
  sudo apt install coq
```

Using Coq: REPL vs. File

REPL (coqtop) is best for: Trying out single tactics, inspecting the current proof state, or learning basic syntax for very short commands.
A File (.v extension) is best for: Almost all Coq development and proof construction. This is how complex proofs and verified programs are structured and managed.

Coq's Comprehensive Question Answering

Unlike Prolog, Coq can directly address all three types of logical questions we've discussed, providing robust answers backed by formal proof:

"Is P(X) true for a specific X?": Coq allows you to define a precise statement (a theorem) like "Alice has a job." You then build a step-by-step logical proof that formally confirms whether this statement is true based on your definitions. If the proof succeeds, Coq formally verifies it: if it fails, Coq clearly shows where your logic breaks down.
"Is there an X for which P(X) is true?": Coq handles questions of existence. If you ask, "Does someone have a job?", you can construct a proof by explicitly providing an example (like "Alice") and then proving that your chosen example indeed satisfies the condition ("Alice has a job").
"Is there any X for which P(X) is false?": This is a key capability where Coq excels over Prolog. Coq allows you to formally prove that a statement is false, or that a counterexample exists. For instance, you could prove "Carol does not have a job" by showing it contradicts the definition, or prove "there exists someone who doesn't have a job" by explicitly identifying such a person and proving that they indeed lack a job. This direct ability to reason about negation and provide formal counterexamples (or prove their non-existence) is what makes Coq a full-fledged proof assistant.

While Coq's core doesn't automatically generate counterexamples when a proof fails, plugins like QuickChick can be integrated for property-based testing to find falsifying examples.

It's a Coq library that allows you to specify properties about your Coq definitions and then randomly generate inputs to try and find a counterexample that falsifies your property.

This is a powerful way to find bugs early in your formalization before you invest a lot of time trying to prove a false theorem.

TLA+, Isabelle, and Lean: A Spectrum of Formal Verification

Beyond Prolog and Coq, other powerful proof assistants and formal specification languages cater to different needs and paradigms:

TLA+: This is a formal specification language developed by Leslie Lamport. It focuses on modeling and verifying system designs (especially concurrent and distributed ones) using temporal logic, rather than proving low-level code. It helps ensure critical properties like safety (nothing bad ever happens) and liveness (something good eventually happens). Its practicality and accessibility make it popular in industry, notably at Amazon and Microsoft for robust system design.
Isabelle and Lean: These are modern, highly advanced proof assistants.
- Isabelle, grounded in higher-order logic, is widely used by researchers and institutions (for example, in projects like the seL4 verified microkernel) for formal theorem proving and software verification in academic and safety-critical domains demanding extreme rigor.
- Lean, based on dependent type theory, is favored by mathematicians for formalizing proofs in pure mathematics (for example, number theory, algebra). It's known for its powerful automation and active community.

These tools represent the pinnacle of applying formal logic to ensure the correctness and reliability of both mathematical theories and complex software systems.

Now that you have a good lay of the land in both theory and practice, here are some thought experiments to enrich your education.

Food for Thought

The journey into formal logic and its intersection with practical domains like software and science offers many avenues for deeper exploration.

Hypothesis Testing in Science and the Implication Truth Table

Statistical hypothesis testing uses a probabilistic form of Modus Tollens. We start with a null hypothesis (H0): "If H0 is true, then observing this data (or more extreme data) is likely." We then observe data that is highly unlikely/unexpected if H0 were true (that is, a small p-value). This serves as our probabilistic "not Q." Therefore, we conclude that H0 is likely not true (we reject H0). This is our probabilistic "∴¬P."

Here, the "truthiness" of P⟹Q is being tested, rather than simply assumed to be true for developing arguments, as in Modus Ponens or Modus Tollens. There's no absolute truth or anything to "prove" definitively.

Inferences are drawn from prior experiments (which inform the test data distribution) and context-specific experiment setups (which determine the significance level α), together defining the threshold (critical value) for what is considered an unlikely observation of Q.

The experiment's result is a rejection (or lack thereof) of H0, not a definitive proof that H0 is true.

Inductive Reasoning's Relationship to Deductive Arguments

Induction generates general rules (for example, "P is always followed by Q") from specific observations or cases.
Deduction then tests or applies those general rules in new situations.

If deduction leads to wrong predictions (that is, a rule is falsified), induction may need to revise the original rule, which forms a continuous feedback loop that refines our understanding.

Necessity and Sufficiency in Implication

The implication P⟹Q ("If you crossed the border, you must have had a passport") unpacks into two fundamental logical concepts:

P is sufficient for Q: Crossing the border guarantees you had a passport. (P alone is enough for Q.)
Q is necessary for P: If you didn't have a passport (¬Q), you couldn't have crossed (¬P). (Q is required for P to happen.)

Q.E.D.: The Enduring Power of Logic in an Uncertain World

Throughout this handbook, we’ve journeyed from the foundational concepts of propositional logic and truth tables to the powerful argument forms of Modus Ponens and Modus Tollens. We explored how these tools enable valid deductions and identified common logical fallacies like Affirming the Consequent and Denying the Antecedent, understanding why they lead to incorrect inferences when an "if-then" relationship isn't a strict "if and only if." We learned the profound importance of falsifiability – the ability for a statement or hypothesis to be disproven – a cornerstone of both scientific inquiry and robust software testing.

We then delved into the practical application of these logical principles in software development, mapping code correctness to test outcomes. We discovered how a failing test, when trusted, becomes a powerful application of Modus Tollens, pinpointing defects. We also confronted the "illusion of correctness" that arises from the affirming the consequent fallacy when tests pass for the wrong reasons, especially when using test doubles.

Crucially, we introduced the "If and Only If" (P⟺Q) relationship, highlighting its unparalleled power in establishing unambiguous connections between cause and effect. This bidirectional guarantee is the ideal we strive for in test suite quality, moving beyond mere correlation to a deeper understanding of causality. We saw how mutation testing rigorously pushes us towards this "if and only if" confidence by actively trying to falsify the assumption that "incorrect code leads to failing tests," thereby strengthening the inverse: "passing tests guarantee correct code."

We also acknowledged the "messy reality" of modern software. Large systems are webs of complexity, with fan-in/fan-out patterns, side effects, and unforeseen interactions that can obscure clear logical chains. Technical debt and the double-edged sword of abstraction often mask the true origins of bugs, turning debugging into a "causal fog."

Logic as Your Compass

Despite these formidable challenges, the logical principles we've explored remain your most vital tools. They provide the mental framework to navigate uncertainty.

When confronted with a bug, your ability to reason logically allows you to formulate hypotheses, design focused experiments (your tests), and interpret their outcomes with precision. Whether you're debugging a complex microservice or reasoning about a simple function, applying Modus Tollens to a failing test or designing tests that aim for P⟺Q clarity helps you cut through the noise.

We also touched upon advanced tools like Proof Assistants (Prolog, Coq, TLA+, Isabelle, Lean), which represent the pinnacle of applying formal logic to guarantee system correctness – a testament to the enduring power of logical rigor in critical domains.

In the intricate dance between theory and practice, the principles of logic stand as an unshakeable foundation. They are the "rocks" upon which you can meticulously build your understanding and your systems. The more consistently you apply this critical thinking, driven by curiosity and a commitment to rigorous validation, the clearer your path becomes.

This clarity is not just about fixing today’s bugs, it’s about continually refining your mental models, fostering trust in your codebase, and equipping yourself to build increasingly robust and predictable systems in an ever-evolving technological landscape.

If you love problem solving, critical thinking, or have experiences on how you fixed an issue that looked different from how it initially seemed, feel free to connect with me at https://linkedin.com/in/hanqi91.

Resources

Article that motivated this handbook: Classical Reasoning and Debugging
3 Formal proofs of modus tollens: https://en.wikipedia.org/wiki/Modus_tollens
Table of 24 syllogisms: https://en.wikipedia.org/wiki/Syllogism
Challenging Assumptions: Falsehoods software teams believe about user feedback
How assumptions and software evolve beyond your control: https://www.tdda.info/why-code-rusts
Relationship to Hypothesis Testing: https://sites.google.com/view/reasonedwriting/home/FRAMEWORK_FOR_SCIENTIFIC_PAPERS/HYPOTHESES/HOW_TO_TEST_HYPOTHESES/MODUS_TOLLENS
The Troubleshooting Mindset: https://www.autodidacts.io/troubleshooting/
Causal Diagrams from The Effect Book: https://theeffectbook.net/ch-CausalDiagrams.html
A systematic guide to the mindsets and practices of debugging: https://www.amazon.sg/Debug-Find-Repair-Prevent-Bugs/dp/193435628X
Constructing P in a way to ensure software correctness: https://www.hillelwayne.com/post/constructive/
Fail Fast by explicitly representing assumptions as assertions: https://www.martinfowler.com/ieeeSoftware/failFast.pdf
Deterministic Simulation Testing to tackle complex systems: https://pierrezemb.fr/posts/learn-about-dst/
GitHub’s Engineering System Success Playbook (ESSP) - Quality, Velocity, Developer Happiness on Business Outcomes: https://assets.ctfassets.net/wfutmusr1t3h/us6AUuwawrtNGTlwlT9Ac/f0fce86712054fc87f10db28b20f303b/GitHub-ESSP.pdf
Closed-world assumption: https://en.wikipedia.org/wiki/Closed-world_assumption

Glossary

Axiom: A fundamental truth or rule accepted as a starting point for a logical or mathematical system, without requiring proof.
Contrapositive: A logically equivalent form of an "if-then" statement (P⟹Q), which is ¬Q⟹¬P ("If not Q, then not P").
Deductive Reasoning: A type of logical reasoning where a conclusion is necessarily true if its premises are true.
Falsification: The principle, especially in science (from Karl Popper), that a hypothesis or theory must be capable of being proven false by empirical observation or experiment.
Formal Logic: The study of abstract systems of reasoning and arguments based on their structure, independent of content.
Hypothesis Testing: A statistical method for making inferences about a population based on sample data, typically by testing a null hypothesis (e.g., "P has no effect on Q") against an alternative hypothesis.
Logical Fallacy: A flaw in the structure or content of an argument that makes it unsound or invalid, even if its conclusion might seem plausible.
- Affirming the Consequent (Fallacy): An invalid argument form that mistakenly assumes if P⟹Q is true, and Q is true, then P must be true.
- Denying the Antecedent (Fallacy): An invalid argument form that mistakenly assumes if P⟹Q is true, and P is false, then Q must be false.
Modus Ponens: A valid argument form: If P⟹Q is true and P is true, then Q must be true.
Modus Tollens: A valid argument form: If P⟹Q is true and ¬Q is true, then ¬P must be true.
Mutation Testing: A software testing technique that involves deliberately introducing small, single-point faults (mutations) into code to assess the effectiveness and coverage of a test suite.
Propositional Logic: A branch of logic that deals with propositions and their relationships using logical operators.
Test-Driven Development (TDD): A software development methodology where tests are written before the code, guiding the development process and ensuring correctness.
Truth Table: A table that systematically lists all possible truth values for a set of propositions and shows the resulting truth value of a complex logical statement.
Vacuously True: Describes an implication (P⟹Q) that is considered true simply because its antecedent (P) is false.

How to Debug and Prevent Buffer Overflows in Embedded Systems

Soham Banerjee — Mon, 17 Mar 2025 16:34:42 +0000

Buffer overflows are one of the most serious software bugs, especially in embedded systems, where hardware limitations and real-time execution make them hard to detect and fix.

A buffer overflow happens when a program writes more data into a buffer than it was allocated, leading to memory corruption, crashes, or even security vulnerabilities. A buffer corruption occurs when unintended modifications overwrite unread data or modify memory in unexpected ways.

In safety-critical systems like cars, medical devices, and spacecraft, buffer overflows can cause life-threatening failures. Unlike simple software bugs, buffer overflows are unpredictable and depend on the state of the system, making them difficult to diagnose and debug.

To prevent these issues, it's important to understand how buffer overflows and corruptions occur, and how to detect and fix them.

Article Scope

In this article, you will learn:

What buffers, buffer overflows, and corruptions are. I’ll give you a beginner-friendly explanation with real-world examples.
How to debug buffer overflows. You’ll learn how to use tools like GDB, LLDB, and memory maps to find memory corruption.
How to prevent buffer overflows. We’ll cover some best practices like input validation, safe memory handling, and defensive programming.

I’ll also show you some hands-on code examples – simple C programs that demonstrate buffer overflow issues and how to fix them.

What this article doesn’t cover:

Security exploits and hacking techniques. We’ll focus on preventing accidental overflows, not hacking-related buffer overflows.
Operating system-specific issues. This guide is for embedded systems, not general-purpose computers or servers.
Advanced RTOS memory management. While we discuss interrupt-driven overflows, we won’t dive deep into real-time operating system (RTOS) concepts.

Now that you know what this article covers (and what it doesn’t), let’s go over the skills that will help you get the most out of it.

Prerequisites

This article is designed for developers who have some experience with C programming and want to understand how to debug and prevent buffer overflows in embedded systems. Still, beginners can follow along, as I’ll explain key concepts in a clear and structured way.

Before reading, it helps if you know:

Basic C programming.
How memory works – the difference between stack, heap, and global variables.
Basic debugging concepts – if you’ve used a debugger like GDB or LLDB, that’s a plus, but not required.
What embedded systems are – a basic idea of how microcontrollers store and manage memory.

Even if you’re not familiar with these topics, this guide will walk you through them in an easy-to-understand way.

Before you dive into buffer overflows, debugging, and prevention, let’s take a step back and understand what a buffer is and why it’s important in embedded systems. Buffers play a crucial role in managing data flow between hardware and software but when handled incorrectly, they can lead to serious software failures.

What is a Buffer, and How Does it Work?
What is a Buffer Overflow?
Common Causes of Buffer Overflows and Corruption
Consequences of Buffer Overflows
How to Debug Buffer Overflows
How to Prevent Buffer Overflows
Conclusion

What is a Buffer, and How Does it Work?

A buffer is a contiguous block of memory used to temporarily store data before it is processed. Buffers are commonly used in two scenarios:

Data accumulation: When the system needs to collect a certain amount of data before processing.
Rate matching: When the data producer generates data faster than the data consumer can process it.

Buffers are typically implemented as arrays in C, where elements are indexed from 0 to N-1 (where N is the buffer size).

Let’s look at an example of a buffer in a sensor system.

Consider a system with a sensor task that generates data at 400 Hz (400 samples per second or 1 sample every 2.5 ms). But the data processor (consumer) operates at only 100 Hz (100 samples per second or 1 sample every 10 ms). Since the consumer task is slower than the producer, we need a buffer to store incoming data until it is processed.

To determine the buffer size, we calculate:

Buffer Size = Time to consume 1 sample / Time to generate 1 sample = 10 ms/ 2.5 ms = 4

This means the buffer must hold at least 4 samples at a time to avoid data loss.

Once the buffer reaches capacity, there are several strategies to decide which data gets passed to the consumer task:

Max/min sampling: Use the maximum or minimum value in the buffer.
Averaging: Compute the average of all values in the buffer.
Random access: Pick a sample from a specific location (for example, the most recent or the first).

In real-world applications, it’s beneficial to use circular buffers or double buffering to prevent data corruption.

Circular buffer approach: A circular buffer (also called a ring buffer) continuously wraps around when it reaches the end, ensuring old data is overwritten safely without exceeding memory boundaries. The buffer size should be multiplied by 2 (4 × 2 = 8) to hold 8 samples. This allows the consumer task to process 4 samples while the next 4 samples are being filled, preventing data overwrites.
Double buffer approach: Double buffering is useful when data loss is unacceptable. It allows continuous data capture while the processor is busy handling previous data. A second buffer of the same size is added. When the first buffer is full, the write pointer switches to the second buffer, allowing the consumer task to process data from the first buffer while the second buffer is being filled. This prevents data overwrites and ensures a continuous data flow.

Buffers help manage data efficiently, but what happens when they are mismanaged? This is where buffer overflows and corruptions come into play.

What is a Buffer Overflow?

A buffer overflow occurs when a program writes more data into a buffer than it was allocated, causing unintended memory corruption. This can lead to unpredictable behavior, ranging from minor bugs to critical system failures.

To understand buffer overflow, let's use a simple analogy. Imagine a jug with a tap near the bottom. The jug represents a buffer, while the tap controls how much liquid (data) is consumed.

The jug is designed to hold a fixed amount of liquid. As long as water flows into the jug at the same rate or slower than it flows out, everything works fine. But if water flows in faster than it flows out, the jug will eventually overflow.

Similarly, in software, if data enters a buffer faster than it is processed, it exceeds the allocated memory space, causing a buffer overflow. In the case of a circular buffer, this can cause the write pointer to wrap around and overwrite unread data, leading to buffer corruption.

Buffer Overflows in Software

Unlike the jug, where water simply spills over, a buffer overflow in software overwrites adjacent memory locations. This can cause a variety of hard-to-diagnose issues, including:

Corrupting other data stored nearby.
Altering program execution, leading to crashes.
Security vulnerabilities, where attackers exploit overflows to inject malicious code.

When a buffer overflow occurs, data can overwrite variables, function pointers, or even return addresses, depending on where the buffer is allocated.

Buffer overflows can occur in different memory regions:

Buffer overflows in global/static memory (.bss / .data sections)
- These occur when global or static variables exceed their allocated size.
- The overflow can corrupt adjacent variables, leading to unexpected behavior in other modules.
- Debugging is easier because memory addresses are fixed at compile time unless the compiler optimizes them. Map files provide a memory layout of variables during the compilation and linking.
Stack-based buffer overflow (more predictable, easier to debug):
- Happens when a buffer is allocated in the stack (for example, local variables inside functions).
- Overflowing the stack can affect adjacent local variables or return addresses, potentially crashing the program.
- In embedded systems with small stack sizes, this often leads to a crash or execution of unintended code.
Heap-based buffer overflow (harder to debug):
- Happens when a buffer is dynamically allocated in the heap (for example, using malloc() in C).
- Overflowing a heap buffer can corrupt adjacent dynamically allocated objects or heap management structures.
- Debugging is harder because heap memory is allocated dynamically at runtime, causing memory locations to vary.

Buffer Overflow vs Buffer Corruption

Buffer overflow and buffer corruption are of course related, but refer to different situations.

A buffer overflow happens when data is written beyond the allocated buffer size, leading to memory corruption, unpredictable behavior, or system crashes.

A buffer corruption happens when unintended data modifications result in unexpected software failures, even if the write remains within buffer boundaries.

Both issues typically result from poor write pointer management, lack of boundary checks, and unexpected system behavior.

Now that we've covered what a buffer overflow is and how it can overwrite memory, let’s take a closer look at how these issues affect embedded systems.

In the next section, we’ll explore how buffer overflows and corruption happen in real-world embedded systems and break down common causes, including pointer mismanagement and boundary violations.

Common Causes of Buffer Overflows and Corruption

Embedded systems use buffers to store data from sensors, communication interfaces (like UART (Universal Asynchronous Receiver-Transmitter), SPI (Serial Peripheral Interface), I2C (Inter-integrated Circuit), and real-time tasks. These buffers are often statically allocated to avoid memory fragmentation, and many implementations use circular (ring) buffers to efficiently handle continuous data streams.

Here are three common scenarios where buffer overflows or corruptions occur in embedded systems:

Writing Data Larger Than the Available Space

Issue: The software writes incoming data to the buffer without checking if there is enough space.

Example: Imagine a 100-byte buffer to store sensor data. The buffer receives variable-sized packets. If an incoming packet is larger than the remaining space, it will overwrite adjacent memory, leading to corruption.

So why does this happen?

Some embedded designs increment the write pointer after copying data, making it too late to prevent overflow.
Many low-level memory functions (memcpy, strcpy, etc.) do not check buffer boundaries, leading to unintended writes.
Without proper bound checking, a large write can exceed the buffer size and corrupt nearby memory.

Here’s a code sample to demonstrate buffer overflow in a .bss / .data section:

  #include 
  #include 
  #include 

  #define BUFFER_SIZE 300

  static uint16_t sample_count = 0;
  static uint8_t buffer[BUFFER_SIZE] = {0};

  // Function to simulate a buffer overflow scenario
  void updateBufferWithData(uint8_t *data, uint16_t size)
  {
      // Simulating a buffer overflow: No boundary check!
      printf("Attempting to write %d bytes at position %d...\n", size, sample_count);

      // Deliberate buffer overflow for demonstration
      if (sample_count + size > BUFFER_SIZE)
      {
          printf("WARNING: Buffer Overflow Occurred! Writing beyond allocated memory!\n");
      }

      // Copy data (unsafe, can cause overflow)
      memcpy(&buffer[sample_count], data, size);

      // Increment sample count (incorrectly, leading to wraparound issues)
      sample_count += size;
  }

  int main()
  {   
      // Save 1 byte to buffer
      uint8_t data_to_buffer = 10;
      updateBufferWithData(&data_to_buffer, 1);

      // Save an array of 20 bytes to buffer
      uint8_t data_to_buffer_1[20] = {5};
      updateBufferWithData(data_to_buffer_1, sizeof(data_to_buffer_1));

      // Intentional buffer overflow: Save an array of 50 x 8 bytes (400 bytes)
      uint64_t data_to_buffer_2[50] = {7};
      updateBufferWithData((uint8_t*)data_to_buffer_2, sizeof(data_to_buffer_2));

      return 0;
  }

Interrupt-Driven Overflows (Real-time Systems)

Issue: The interrupt service routine (ISR) may write data faster than the main task can process, leading to buffer corruption or buffer overflow if the write pointer is not properly managed.

Example: Imagine a sensor ISR that writes incoming data into a buffer every time a new reading arrives. Meanwhile, a low-priority processing task reads and processes the data.

What can go wrong?

If the ISR triggers too frequently (due to a misbehaving sensor or high interrupt priority), the buffer may fill up faster than the processing task can keep up.
This can result in one of two failures:
1. Buffer Corruption: The ISR overwrites unread data, leading to loss of information.
2. Buffer Overflow: The ISR exceeds buffer boundaries, causing memory corruption or system crashes.

So why does this happen?

In real-time embedded systems, ISR execution preempts lower-priority tasks.
If the processing task doesn't not get enough CPU time, the buffer may become overwritten or overflow beyond its allocated scope.

System State Changes & Buffer Corruption

Issue: The system may unexpectedly reset, enter low-power mode, or changes operating state, leaving the buffer write pointers in an inconsistent state. This can result in buffer corruption (stale or incorrect data) or buffer overflow (writing past the buffer’s limits.

Example Scenarios:

Low-power wake-up issue (Buffer Overflow risk): Some embedded systems enter deep sleep to conserve energy. Upon waking up, if the buffer write pointer is not correctly reinitialized, it may point outside buffer boundaries, leading to buffer overflow and unintended memory corruption.
Unexpected mode transitions: If a sensor task is writing data and the system suddenly switches modes, the buffer states and pointers may not be cleaned up. The next time the sensor task runs, it may continue writing without clearing previous data. This can cause undefined behavior due to presence of stale data.

Now that you understand how buffer overflows and corruptions happen, let’s examine their consequences in embedded systems ranging from incorrect sensor readings to complete system failures, making debugging and prevention critical.

Consequences of Buffer Overflows

Buffer overflows can be catastrophic in embedded systems, leading to system crashes, data corruption, and unpredictable behavior. Unlike general-purpose computers, many embedded devices lack memory protection, making them particularly vulnerable to buffer overflows.

A buffer overflow can corrupt two critical types of memory:

1. Data Variables Corruption

A buffer overflow can overwrite data variables, corrupting the inputs for other software modules. This can cause unexpected behavior or even system crashes if critical parameters are modified.

For example, a buffer overflow could accidentally overwrite a sensor calibration value stored in memory. As a result, the system would start using incorrect sensor readings, leading to faulty operation and potentially unsafe conditions.

2. Function Pointer Corruption

In embedded systems, function pointers are often used for interrupt handlers, callback functions, and RTOS task scheduling. If a buffer overflow corrupts a function pointer, the system may execute unintended instructions, leading to a crash or unexpected behavior.

As an example, a function pointer controlling motor speed regulation could be overwritten. Instead of executing the correct function, the system would jump to a random memory address, causing a system fault or erratic motor behavior.

Buffer overflows are among the hardest bugs to identify and fix because their effects depend on which data is corrupted and the values it contains. A buffer overflow can affect memory in different ways:

If a buffer overflow corrupts unused memory, the system may seem fine during testing, making the issue harder to detect.
if a buffer overflow alters critical data variables, it can cause hidden logic errors that cause unpredictable behavior.
If a buffer overflow corrupts function pointers, it may crash immediately, making the problem easier to identify.

During development, if tests focus only on detecting crashes, they may overlook silent memory corruption caused by a buffer overflow. In real-world deployments, new use cases not covered in testing can trigger previously undetected buffer overflow issues, leading to unpredictable failures.

Buffer overflows can cause a chain reaction, where one overflow leads to another overflow or buffer corruption, resulting in widespread system failures. So how does this happen?

A buffer overflow corrupts a critical variable (for example, a timer interval).
The corrupted variable disrupts another module (for example, triggers the timer interrupt too frequently, causing it to push more data into a buffer than intended.).
This increased interrupt frequency forces a sensor task to write data faster than intended, eventually causing another buffer overflow or corruption by overwriting unread data.

This chain reaction can spread across multiple software modules, making debugging nearly impossible. In real-word applications, buffer overflows in embedded systems can be life-threatening:

In cars: A buffer overflow in an ECU (Electronic Control Unit) could cause brake failure or unintended acceleration.
In a spacecraft: A memory corruption issue could disable navigation systems, leading to mission failure.

Now that we’ve seen how buffer overflows can corrupt memory, disrupt system behavior, and even cause critical failures, the next step is understanding how to detect and fix them before they lead to serious issues.

How to Debug Buffer Overflows

Debugging buffer overflows in embedded systems can be complex, as their effects range from immediate crashes to silent data corruption, making them difficult to trace. A buffer overflow can cause either:

A system crash, which is easier to detect since it halts execution or forces a system reboot.
Unexpected behavior, which is much harder to debug as it requires tracing how corrupted data affects different modules.

This section focuses on embedded system debugging techniques using memory map files, debuggers (GDB/LLDB), and a structured debugging approach. Let’s look into the debuggers and memory map files.

Memory Map File (.map file)

A memory map file is generated during the linking process. It provides a memory layout of global/static variables, function addresses, and heap/stack locations. It provides a memory layout of Flash and RAM, including:

Text section (.text): Stores executable code.
Read-only section (.rodata): Stores constants and string literals.
BSS section (.bss): Stores uninitialized global and static variables.
Data section (.data): Stores initialized global and static variables.
Heap and stack locations, depending on the linker script.

If a buffer overflow corrupts a global variable, the .map file can identify nearby variables that may also be affected, provided the compiler has not optimized the memory allocation. Similarly, if a function pointer is corrupted, the .map file can reveal where it was stored in memory.

Debuggers (GDB & LLDB)

Debugging tools like GDB (GNU Debugger) and LLDB (LLVM Debugger) allow:

Controlling execution (breakpoints, stepping through code).
Inspecting variable values and memory addresses.
Getting backtraces (viewing function calls before a crash).
Extracting core dumps from microcontrollers for post-mortem analysis.

If the system halts on a crash, a backtrace (bt command in GDB) can reveal which function was executing before failure. If the overflow affects a heap-allocated variable, GDB can inspect heap memory usage to detect corruption.

The Debugging Process

Now, let’s go through a step-by-step debugging process to identify and fix buffer overflows. Once a crash or unexpected behavior occurs, follow these techniques to trace the root cause:

Step 1: Identify the misbehaving module

If the system crashes, use GDB or LLDB backtrace (bt command) to locate the last executed function. If the system behaves unexpectedly, determine which software module controls the affected functionality.

Step 2: Analyze inputs and outputs of the module

Every function or module has inputs and outputs. Create a truth table listing expected outputs for all possible inputs. Check if the unexpected behavior matches any undefined input combination, which may indicate corruption.

Step 3: Locate memory corruption using address analysis

If a variable shows incorrect values, determine its physical memory location. Depending on where the variable is stored:

Global/static variables (.bss / .data): Look up the memory map file for nearby buffers.

Heap variables: Snapshot heap allocations using GDB.

Here’s an example of using GDB to find corrupted variables:

 (gdb) print &my_variable  # Get memory address of the variable
 $1 = (int *) 0x20001000
 (gdb) x/10x 0x20001000   # Examine memory near this address, Display 10 memory words in hexadecimal format starting from 0x20001000

Step 4: Identify the overflowing buffer

If a buffer is located just before the corrupted variable, inspect its usage in the code. Review all possible code paths that write to the buffer. Check if any design limitations could cause an overflow under a specific use cases.

Step 5: Fix the root cause

If the buffer overflow happened due to missing bounds checks, add proper input validation to prevent it. Buffer design should enforce strict memory limits. The module should implement strict boundary checks for all inputs and maintain a consistent state.

In addition to GDB/LLDB, you can also use techniques like hardware tracing and fault injection to simulate buffer overflows and observe system behavior in real-time.

While debugging helps identify and fix buffer overflows, prevention is always the best approach. Let’s explore techniques that can help avoid buffer overflows altogether.

How to Prevent Buffer Overflows

You can often prevent buffer overflows through good software design, defensive programming, hardware protections, and rigorous testing. Embedded systems, unlike general-purpose computers, often lack memory protection mechanisms, which means that buffer overflow prevention critical for system reliability and security.

Here are some key techniques to help prevent buffer overflows:

Defensive Programming

Defensive programming helps minimize buffer overflow risks by ensuring all inputs are validated and unexpected conditions are handled safely.

First, it’s crucial to validate input size before writing to a buffer. Always check the write index by adding the size of data to be written prior to writing data to make sure more data is not written than the available buffer space.

Then you’ll want to make sure you have proper error handling and fail-safe mechanisms in place. If an input is invalid, halt execution, log the error, or switch to a safe state. Also, functions should indicate success/failure with helpful error codes to prevent misuse.

Sample Code:

   #include 
   #include 
   #include 
   #include 

   #define BUFFER_SIZE 300

   static uint16_t sample_count = 0;
   static uint8_t buffer[BUFFER_SIZE] = {0};

   typedef enum
   {
       SUCCESS = 0,
       NOT_ENOUGH_SPACE = 1,
       DATA_IS_INVALID = 2,
   } buffer_err_code_e;


   buffer_err_code_e updateBufferWithData(uint8_t *data, uint16_t size)
   {
       if (data == NULL || size == 0 || size > BUFFER_SIZE)  
       {
           return DATA_IS_INVALID; // Invalid input size
       }

       uint16_t available_space = BUFFER_SIZE - sample_count;
       bool can_write = (available_space >= size) ? true : false;

       if (!can_write)  
       {
           return NOT_ENOUGH_SPACE;
       }

       // Copy data safely
       memcpy(&buffer[sample_count], data, size);
       sample_count += size;

       return SUCCESS;
   }

   int main()
   {   
       buffer_err_code_e ret;

       // Save 1 byte to buffer
       uint8_t data_to_buffer = 10;
       ret = updateBufferWithData(&data_to_buffer, sizeof(data_to_buffer));
       if (ret)  
       {
           printf("Buffer update didn't succeed, Err:%d\n", ret);
       }

       // Save an array of 20 bytes to buffer
       uint8_t data_to_buffer_1[20] = {5};
       ret = updateBufferWithData(data_to_buffer_1, sizeof(data_to_buffer_1));
       if (ret)  
       {
           printf("Buffer update didn't succeed, Err:%d\n", ret);
       }

       // Save an array of 50 x 8 bytes, Intentional buffer overflow
       uint64_t data_to_buffer_2[50] = {7};
       ret = updateBufferWithData((uint8_t*)data_to_buffer_2, sizeof(data_to_buffer_2));  
       if (ret)  
       {
           printf("Buffer update didn't succeed, Err:%d\n", ret);
       }

       return 0;
   }

Choosing the Right Buffer Design And Size

Some buffer designs handle overflow better than others. Choosing the correct buffer type and size for the application reduces the risk of corruption.

Circular Buffers (Ring Buffers) prevent out-of-bounds writes by wrapping around. They overwrite the oldest data instead of corrupting memory. These are useful for real-time streaming data (for example, UART, sensor readings). This approach is ideal for applications where data loss is unacceptable.
Ping-Pong Buffers (Double Buffers) use two buffers. One buffer fills up with data. Then, once it’s full, it switches to the second buffer while the first one is processed. This approach is beneficial for application that have strict requirements on no data loss. The buffer design should be based on the speed of write and read tasks.

Hardware Protection

Memory Protection Unit (MPU)

An MPU (Memory Protection Unit) helps detect unauthorized memory accesses, including buffer overflows, by restricting which regions of memory can be written to. It prevents buffer overflows from modifying critical memory regions and triggers a MemManage Fault if a process attemps to write outside an allowed region.

But keep in mind that, an MPU does not prevent buffer overflows – it only detects and stops execution when they occur. Not all microcontrollers have an MPU, and some low-end MCUs lack hardware protection, making software-based safeguards even more critical.

Modern C compilers provide several flags to identify memory errors at compile-time:

-Wall -Wextra: Enables useful warnings
-Warray-bounds: Detects out-of-bounds array access when the array size is known at compile-time
-Wstringop-overflow: Warns about possible overflows in string functions like memcpy and strcpy.

Testing and Validation

Testing helps detect buffer overflows before deployment, reducing the risk of field failures. Unit testing each function independently with valid inputs, boundary cases, and invalid inputs helps detect buffer-related issues early. Automated testing involves feeding random and invalid inputs into the system to uncover crashes and unexpected behavior. Static Analysis Tools like Coverity, Clang Static Analyzer help detect buffer overflows before runtime. Run real-world inputs on embedded hardware to detect issues.

Now that we've explored how to identify, debug, and prevent buffer overflows, it’s clear that these vulnerabilities pose a significant threat to embedded systems. From silent data corruption to catastrophic system failures, the consequences can be severe.

But with the right debugging tools, systematic analysis, and preventive techniques, you can effectively either prevent or mitigate buffer overflows in your systems.

Conclusion

Buffer overflows and corruption are major challenges in embedded systems, leading to crashes, unpredictable behavior, and security risks. Debugging these issues is difficult because their symptoms vary based on system state, requiring systematic analysis using memory map files, GDB/LLDB, and structured debugging approaches.

In this article, we explored:

The causes and consequences of buffer overflows and corruptions
How to debug buffer overflows using memory analysis and debugging tools
Best practices for prevention

Buffer overflow prevention requires a multi-layered approach:

Follow a structured software design process to identify risks early.
Apply defensive programming principles to validate inputs and handle errors gracefully.
Use hardware-based protections like MPUs where available.
Enable compiler flags that help identify memory errors.
Test extensively, unit testing, automated testing, and code reviews help catch vulnerabilities early.

By implementing these best practices, you can minimize the risk of buffer overflows in embedded systems, improving reliability and security.

In embedded systems, where reliability and safety are critical, preventing buffer overflows is not just a best practice, it is a necessity. A single buffer overflow can compromise an entire system. Defensive programming, rigorous testing, and hardware protections are essential for building secure and robust embedded applications.

How to Use Developer Tools to Debug JavaScript in the Browser

Kolade Chris — Wed, 30 Oct 2024 15:00:55 +0000

The console object is the number one go-to for developers when working with buggy JavaScript code.

But if you still rely heavily on the console object alone to debug your JavaScript, then you're missing out on some amazing browser developer tools features.

Let's take a look at how you can debug JavaScript with the Chrome developer tools.

The Buggy Code We're Working With

To get started, I’ve prepared some buggy code that should add four numbers and also get their average.

Here's the HTML of the code:

<label for="num1">Number 1:label>
<input type="text" id="num1" placeholder="Enter a number" />

<label for="num2">Number 2:label>
<input type="text" id="num2" placeholder="Enter a number" />

<label for="num3">Number 3:label>
<input type="text" id="num3" placeholder="Enter a number" />

<label for="num4">Number 4:label>
<input type="text" id="num4" placeholder="Enter a number" />

<button id="calculateBtn">Calculate Sum and Averagebutton>

<p id="sum">p>
<p id="average">p>

<script src="script.js">script>

Here's the very minimal CSS to push the labels to their respective lines and enlarge the input elements and button a bit:

body {
  background: #d2d2d2;
}

label {
  display: block;
  margin-top: 0.5rem;
}

button {
  display: block;
  margin-top: 1rem;
}

input,
button {
  padding: 0.2rem;
}

Here's what the HTML and CSS displays in the browser:

And here's the JavaScript in which the bug occurs:

const calculateBtn = document.getElementById('calculateBtn');
const sumText = document.getElementById('sum');
const avgText = document.getElementById('average');

function calculateTotal(a, b, c, d) {
 return a + b + c + d;
}

function calculateAverage(total, count) {
 return total / count;
}

function handleButtonClick() {
 let num1 = document.getElementById('num1').value;
 let num2 = document.getElementById('num2').value;
 let num3 = document.getElementById('num3').value;
 let num4 = document.getElementById('num4').value;

 let total = calculateTotal(num1, num2, num3, num4);
 let average = calculateAverage(total, 4);

 sumText.textContent = `The sum is ${total}`;
 avgText.textContent = `The average is: ${average}`;
}

calculateBtn.addEventListener('click', handleButtonClick);

Here's what happens if you enter the 4 numbers, say 3, 4, 2, 1, and click the Calculate Sum and Average button:

Unfortunately, the numbers just got merged and the average is calculated based on that, which means concatenation is going on instead of addition. The buggy addition leads to a buggy average calculation too.

Let's investigate what's happening with the browser developer tools.

How to Debug JavaScript Code Using Chrome Developer Tools

When such a bug occurs, you might be tempted to add a bunch of console logs.

Many times, console logs get the job done – but you have to spend a lot of time figuring things out.

The browser developer tools give you more options such as adding breakpoints, watching particular expressions, and even stepping through the code line by line to see where the bug occurs.

How to Open the Developer Tools and the Sources Tab

To start, right-click in the browser and select "inspect" to open the Chrome DevTools.

While in DevTools, head over to the "Sources" tab to see the files in the program. You can also press F12 on your keyboard and select the Sources tab.

Here's a brief anatomy of the Chrome DevTools Sources tab:

On top of the debugger tab are some greyed-out icons. When active, they let you step through your code and add or remove breakpoints.

Also in the debugger tab are:

Watch: where you can add and see the watch expressions
Breakpoints: where you can see the code of the line you add a breakpoint to
Scope: contains the local and global variables
Callstack: shows the function calls that lead to the current point of code execution

To see the contents of any file, you can click on it. After doing that, some of the icons in the debugger tab won’t be greyed out anymore.

How to Debug the Code by Adding Breakpoints

To start debugging the code, you can add a breakpoint to a line by clicking the line number.

A breakpoint is like a line marker you can set in the developer tools to pause the execution of your code before that line executes. This lets you check variable values, see if functions are called as expected, or observe the general flow of the code.

When you add a breakpoint and execute the code, a bluish icon appears on that line, indicating that execution will pause before that line.

Alternatively, you can add the debugger statement to the line where you want the execution to be paused. But let’s stick to using breakpoints.

For example, let's add a breakpoint to line 14, then enter the four numbers and click the Calculate Sum and Average button so the code will run:

At this point, you can see that the execution did not continue – that's why you see "value unavailable" for all the variables under “Local”.

From here, you can start stepping through the code line by line by pressing the step icon in the top right corner:

Once you click the step icon, the line you step out of executes.

You can see that "3" is the value for line 14. That value is surrounded by a pair of double quotes. That means it‘s a string. You need to be sure about that, though, and that’s what the watch feature lets you do. You’ll learn about that feature soon.

If you’re working with several lines of code, it might be time-consuming to step through the code line by line. So, you might have to add another breakpoint.

I will go ahead and set a breakpoint at line 23 and run the code again:

Now you can see that all the variable results apart from average appear to be strings. This takes us to the next Chrome developer tools debugger feature – watcher.

How to Use the Developer Tools Watch Feature

The developer tools watch feature lets you monitor specific variables or expressions as your code runs.

To confirm the data type of the variables, you can add a watch expression that shows their values, or more appropriately, their types.

To add a watch expression, click the plus (+) icon right beside “Watch” and hit ENTER on your keyboard.

Here are the watch expressions that confirm that num1 through num4 and total are strings – but they should be integers:

You can also verify this in the console tab by checking the types of the variables there:

This means that the numbers you enter are interpreted as strings. That’s because, in JavaScript, values from HTML elements like input fields are always retrieved as strings.

This happens because the value property of an input element returns a string, regardless of whether you enter numbers – and that’s how the bug was introduced.

Remember that JavaScript only concatenates strings even if they’re numbers. That means "3" is a string type and not an integer type.

To fix the bug, you should change the types of num1 through num4 to integers so JavaScript can correctly sum up their values.

You can then go ahead and fix this in the DevTools and press CTRL + S on Windows or CMD + S on Mac to save the code. You can also fix it inside your code editor by wrapping the variables of the numbers in parseInt().

Once you do that and run the code again, the correct data types should show in the watcher, and the correct variable values should show under Local:

You can also go ahead and implement the changes in your code editor so that everything works. Here’s the correct code:

const calculateBtn = document.getElementById('calculateBtn');
const sumText = document.getElementById('sum');
const avgText = document.getElementById('average');

function calculateTotal(a, b, c, d) {
  return a + b + c + d;
}

function calculateAverage(total, count) {
  return total / count;
}

function handleButtonClick() {
  let num1 = parseInt(document.getElementById('num1').value);
  let num2 = parseInt(document.getElementById('num2').value);
  let num3 = parseInt(document.getElementById('num3').value);
  let num4 = parseInt(document.getElementById('num4').value);

  let total = calculateTotal(num1, num2, num3, num4);
  let average = calculateAverage(total, 4);

  sumText.textContent = `The sum is ${total}`;
  avgText.textContent = `The average is: ${average}`;
}

calculateBtn.addEventListener('click', handleButtonClick);

And here’s the result in the browser:

Conclusion

That’s how you can debug JavaScript using Chrome’s developer tools. The breakpoint and watcher features, alongside the step-through buttons, are upgrades over a basic console log.

Note that every modern browser has this JavaScript debugging tool built into it, so you can use the same approach to debug JavaScript with Firefox or Edge.

How to Debug Node.js Applications Using the debugger; Statement

Yogesh Chavan — Mon, 22 Apr 2024 19:50:47 +0000

In this tutorial, you will learn the easiest and most efficient way to debug Node.js application code.

So let's get started.

Want to watch the video version of this tutorial? You can check out the video below:

How You Usually Debug Node.js Applications
How to Add a Debugger to Debug Your Code
How to Run the Application for Debugging
How to Access Variables During Debugging
How to Create a Script to Debug Node.js Apps
Quick Recap

How You Usually Debug Node.js Applications

If you want to debug a Node.js application, usually you add a console.log statement in the code that you want to debug to find out the value of any variable.

This works, but you need to keep checking the console log to see the value that you're trying to print.

And if the data printed in the console contains nested objects or if it's a lot of data, then using console.log is not feasible.

Fortunately, there's a better way.

How to Add a Debugger to Debug Your Code

Instead, you can add a debugger; statement in the code that you want to debug.

So let's say you have an Express.js API route for registering a user as shown in the below code:

// controllers/auth.js

const register = async (req, res) => {
  try {
    const { email, password } = req.body;
    const existingUser = await User.findOne({
      email,
    });
    if (existingUser) {
      return res.status(400).send('User with the provided email already exist');
    }
    // some more code
    return res.status(201).send();
  } catch (error) {
    console.log(error);
    return res
      .status(500)
      .send('Error while registering a new user. Try again later.');
  }
};

module.exports = { register };

// routes/auth.js
const { register } = require('../controllers/auth');

const Router = express.Router();

Router.post('/api/register', register);

And there's some issue while registering a user, so you want to debug the register function's code.

In that case, you can just add a debugger; statement inside the register function code like this:

const register = async (req, res) => {
  try {
    const { email, password } = req.body;
    debugger;
    const existingUser = await User.findOne({
      email,
    });
    if (existingUser) {
      return res.status(400).send('User with the provided email already exist');
    }
    // some more code
    return res.status(201).send();
  } catch (error) {
    console.log(error);
    return res
      .status(500)
      .send('Error while registering a new user. Try again later.');
  }
};

How to Run the Application for Debugging

Normally, you start your Node.js application by executing the following command:

node index.js

But instead, you can execute the following command:

node inspect index.js

Here, we just added an inspect keyword in between.

If your main application file's name is server.js, you can execute the node inspect server.js command.

Once you execute the above command, you will see the output displayed as shown below:

Debugger Attached

As you can see from the output, the debugger is attached, so now you can start debugging the code.

To do that, open the Chrome browser and enter chrome://inspect in the browser URL.

You will see the output as shown below:

Chrome Inspect Page

Since you executed the node inspect index.js command to start inspecting, you can see a new target entry displayed under the Remote Target section.

Now, if you click on the displayed blue inspect link, then you will see a new browser dev tool opened as shown below:

Debugger Paused

As you can see in the right panel in the above image, the Debugger paused message is displayed. The debugging control is at the first line of code, as you can see from the highlighted yellow line.

But you don't want to start debugging from the first line of code. Instead, you want to just debug the registration code. To do this, click on the blue triangle icon which is displayed just above the Debugger paused message as shown below:

Continuing Debugging

Now don't close this window – instead, try registering a user from the application or making an API call using Postman, so the /register route handler code that we added previously will be executed.

Registering User & Debugging Code

As you can see above, when you click on create new account button, you're automatically redirected to the code where you added the debugger; statement.

Now, you can debug the code line by line and see the values of each variable during debugging to find out and fix the issue.

How to Access Variables During Debugging

Sometimes when you mouse over any variable while debugging to see its actual value, the value might be too long (because it might be an object with many properties), so you can't see it easily by mousing over it.

In that case, while the debugger is still active, you can open the console tab and type the name of the variable whose value you want to see – as you can see in the below Gif:

Logging Variables In The Console

So that's how you can easily debug any of your Node.js application's code.

How to Create a Script to Debug Node.js Apps

If you don't want to manually type the node inspect index.js command every time in the terminal, you can create a new debug script inside the package.json file like this:

"scripts": {
    "start": "node index.js",
    "debug": "node inspect index.js",
    "dev": "nodemon index.js"
},

So now, you can execute the npm run debug command to start your application in debug mode.

Quick Recap

To debug a Node.js application, you need to follow the below steps:

Add a debugger statement inside the code that you want to debug.
Run the node inspect index.js or node inspect server.js command to start the application in debug mode.
Access the URL chrome://inspect in your Chrome browser.
Click on the inspect link under the Remote Target section.
Click on the blue triangle icon to skip debugging if you don't want to start debugging your application from the first line of the index.js or server.js file.
Make an API call or do something that will trigger the code where you added the debugger; statement. This way you can debug the code line by line and find out the issue.

Thanks for Reading

That's it for this tutorial. I hope you learned something new.

Want to watch the video version of this tutorial? You can check out this video.

If you want to master JavaScript, ES6+, React, and Node.js with easy-to-understand content, check out my YouTube channel. Don't forget to subscribe.

Want to stay up to date with regular content on JavaScript, React, and Node.js? Follow me on LinkedIn.

How to Debug Coding Problems When Building Your Own Projects

Chris Blakely — Tue, 20 Feb 2024 21:47:09 +0000

Ah, the joy of coding! There you are, cruising through your project, when suddenly – bam! – you hit a bug. It's like hitting a wall in a maze.

But fear not, fellow coder, for I bring you the trusty map to navigate the treacherous bug-infested waters of programming. Whether you're a self-learner aiming to land that dream job in tech or just tinkering for fun, here's how to become a debugging ninja.

Look for Errors in Your IDE

Your Integrated Development Environment (IDE) isn't just a fancy text editor – it's your first line of defense against bugs.

TypeScript, for example, is like that friend who points out the pothole you're about to step into – it helps catch errors early with its type-checking prowess.

Imagine you accidentally try to add a number to a string. TypeScript waves a big red flag, saving you from a facepalm moment later. It's one of the many reasons we adore TS.

Example: You declare let age: number = 'twenty';. TypeScript will frown upon this, telling you that 'twenty' is not a number. It's like having a guardian angel for your code.

Try and Isolate the Area

Before you start pulling out your hair, try to play detective and isolate where the crime scene is.

Is the bug lurking in the backend, hiding in the frontend, conspiring in the database, or chilling in the infrastructure?

When you're working locally, it's usually one of the first three suspects. And here's a hot tip: the network tab in your browser's developer tools is like your police scanner, helping you pinpoint the location of the distress call.

Example: Let's say you send out a request to GET /users and it returns a 500 status. That's the server telling you, "Mate, I've got problems." It's a backend issue. But if the call comes back with a 200 status and your UI is still playing hide and seek with the data, then the bug's hosting a party in your frontend. The network tab just handed you the address.

By narrowing down the location of your issue, you can focus your debugging efforts more efficiently. It's like knowing whether to raid the castle, the dragon's lair, or the dark forest. Happy hunting!

Look for Errors in the Browser Console

The browser console is your Sherlock Holmes magnifying glass for web projects. It uncovers clues hidden in plain sight. The console tab, on the other hand, is like tracking where the villain has been, helping you spot those pesky code misfires.

Example: Your React app isn't fetching data. A quick peek in the console tab shows an "undefined" error, and a line number. This is where your problem is at. Elementary, my dear Watson!

Add `console.log()` to Different Functions

Ah, the humble console.log(), the print statement that could. When in doubt, log it out. It's like dropping breadcrumbs through your code to see how far Little Red Riding Hood gets before she meets the Big Bad Bug.

Example: Unsure if your function is receiving the expected data? console.log('Data:', data) at the start of the function. No data? Now you know where the problem starts.

Use Try-Catch Blocks

Try-catch blocks are your safety net, allowing your code to perform daring feats without crashing your app. They let you gracefully handle errors by catching them before they wreak havoc. They also let you specify your own custom error messages for a given block of code, helping you to find the problem area.

Example: Wrap your API call in a try-catch. If the call fails, the catch block catches the error, allowing you to console.log it or display a friendly message to the user.

Heres what a try catch block looks like in JS:

  function displayUsers() {
    try {
      const users = getUsers();
    } catch (error) {
      console.log("oh crap");
    }
  }

Search Google or Use ChatGPT to Help With Error Messages

Stuck on an error message? Google and ChatGPT are your library and librarian. Just copy and paste the error into the search bar and watch a plethora of solutions unfold. It's like asking the hive mind: someone, somewhere, has had your problem before.

Example: Getting a "TypeError: Cannot read property 'map' of undefined"? A quick search reveals you might be trying to use .map() on something that's not an array. Oops!

Test Often

The mantra "test early, test often" will save you heaps of time. By testing small bits of code as you go, you catch bugs early, when they're easier to squash. It's like cleaning as you cook– it makes the final cleanup so much easier.

Example: Just added a new feature? Test it out before moving on. Does it work as expected? Great! No? Time to debug while the code is still fresh in your mind.

Try a Different Approach

If you're banging your head against the wall with a problem, maybe it's time to climb over it instead. Don't get too attached to your code. Be willing to refactor or even start from scratch if it means a cleaner, more elegant solution.

Example: If your code is more tangled than a bowl of spaghetti, stepping back and rethinking your approach might reveal a simpler, more efficient path.

Debugging is part art, part science, and entirely a test of patience. But with these strategies in your toolkit, you'll be squashing bugs with the best of them. Happy coding, and may your bug hunts be short and your code be clean!

Real Life Scenario

Let's take real life scenario. I have a React, Node, Postgres app that displays users in the browser. The code, as far as I know, should be working but I'm not seeing the users displayed on the frontend.

Step 1 – Check the Console

Let's have a nosey at the Chrome dev tools console and see what's going on.

"Look ma, errors!"

Ah, the plot thickens in the saga of "Why isn't this thing working?". Let's dive into the drama unfolding in your console and break down the breadcrumbs left behind by our mischievous friend, the bug.

First up, we have our leading clue: GET http://localhost:3000/api/users 500 (Internal Server Error). This line is the equivalent of a scream in the night in our detective story. It tells us that our backend is in distress, possibly tied to a nefarious SQL query or a rogue piece of logic in our API route.

The server's cry for help is loud and clear: "Internal Server Error." Classic move by the backend, really.

Now, our supporting cast makes their entrance with ResponseError: Response returned an error code. This is the big reveal. The issue isn't just a server having a bad day – it's a ResponseError caught red-handed by UsersApi.request, and even tells us where the error line is (UserApi.ts:83).

Step 2 – Check the backend Terminal

Our journey into investigating the bug has brought us to the backend, where we are greeted with this:

If you see this and your first instinct is to run away and hide, don't worry – that was mine too. But fear not! There are plenty of clues that point us to the issue.

When a backend error occurs, this is what's known as a stack trace – basically all the errors, info, line numbers, and so on that the compiler encountered in one big block of text. Thanks compiler!

What we do here is look for key words, recognisable files, or anything that's readable by humans. Did you spot anything? Let's dive deeper:

Digging deeper into the errors

The highlighted parts in yellow, indicate that there was an error in userController.ts, specifically at getAllUsers() function. If we read further, the highlighted parts in red point us to the error message:

Authentication failed against database server at `dpg-cn9kr28l6cac73a1a7eg-a.frankfurt-postgres.render.com`, 
the provided database credentials for `dmin` are not valid.\n\nPlease make sure to provide valid database credentials for the database server

Hurray! Now we know the error. We have spelled "admin" incorrectly in our database connection string, meaning the connection failed. Doh! After we fix this, the error is resolved:

Error resolved

Step 3: Verify the Fix

Now we've added a fix, we can verify by checking the browser to see if everything's working. In this case, checking the UI is enough to verify, but for more complex flows you can get that the API is returning the correct status code (in this case, 200)

Conclusion

I hope this article has shed some light on how you can debug your projects.

If you are looking for more debugging insights, and real industry level projects to build, you can check out by YouTube where we build and deploy full stack applications using React, Node, and some other cool tech. Hope to see you there!

Console Logging Tips – How to Debug and Understand Your Code

Joan Ayebola — Tue, 20 Feb 2024 18:09:59 +0000

Console logging is an essential tool for developers to use to debug and understand the behavior of their code.

While most developers are familiar with basic console logging using console.log(), there are many other powerful methods provided by the console object that can make debugging more efficient and effective.

In this comprehensive guide, we will explore various console logging tricks such as console.table, console.group, console.assert, and more. These tricks can help you organize your debugging process, visualize complex data structures, and catch errors early on in your development workflow.

Introduction to Console Logging
Basic Console Logging
Advanced Console Logging Tricks
– [console.table](#heading-31-consoletable)
– console.group and console.groupCollapsed
– [console.assert](#heading-33-consoleassert)
– console.count and console.countReset
– console.time and console.timeEnd
– [console.trace](#heading-36-consoletrace)
– [console.dir](#heading-37-consoledir)
– [console.clear](#heading-38-consoleclear)
Best Practices for Console Logging
Conclusion

1. Introduction to Console Logging

Console logging is a technique used by developers to output messages, variables, and other information to the browser's console. This is particularly useful for debugging purposes, as it allows developers to inspect the state of their code and track its execution flow.

The console object in JavaScript provides various methods for logging different types of information. While console.log() is the most commonly used method, there are several other methods that can be used to enhance your debugging experience.

2. Basic Console Logging

Before we dive into the advanced console logging tricks, let's start by revisiting the basics of console logging using console.log(). This method accepts any number of arguments and outputs them to the console.

const name = "Femi";

const age = 30;

console.log("Name:", name, "Age:", age);

In the above example, we are logging the name and age variables to the console using console.log(). This will output:

Name: Femi Age: 30

You can use console.log() to log strings, numbers, booleans, objects, arrays, and more.

3. Advanced Console Logging Tricks

3.1 `console.table`

The console.table() method allows you to display tabular data in the console. It takes an array or an object as input and presents it as a table.

const users = [

{ name: "Chris", age: 25 },

{ name: "Dennis", age: 15 },

{ name: "Victor", age: 17 }

];

console.table(users);

The above code will output a table in the console:

(index)  |  name  |  age
-------------------------
0    |  Chris  |   25
1    |  Dennis |   15
2    |  Victor |   17

console.table() is particularly useful when dealing with arrays of objects or other tabular data structures.

3.2 `console.group` and `console.groupCollapsed`

The console.group() and console.groupCollapsed() methods allow you to group related log messages together in the console. This can help organize your debugging output and make it easier to understand.

// Start a new console group named "Group 1"
console.group("Group 1");

// Log messages inside "Group 1"
console.log("Message 1");
console.log("Message 2");

// End "Group 1"
console.groupEnd();

// Start a new collapsed console group named "Group 2"
console.groupCollapsed("Group 2");

// Log messages inside "Group 2"
console.log("Message 3");
console.log("Message 4");

// End "Group 2"
console.groupEnd();

In the above example, we create two groups of log messages. The first group is expanded, while the second group is collapsed by default. This helps keep the console output organized and easy to navigate.

If you run this code in a browser's developer console, the output will look something like this:

Group 1
  Message 1
  Message 2
Group 2
  ▶ Message 3
  ▶ Message 4

In this example, "Group 1" is expanded by default, showing the messages inside it. On the other hand, "Group 2" is collapsed initially (indicated by the ▶ symbol), and you need to click on it to expand and reveal the messages inside. The collapsing of "Group 2" makes it visually neater in the console, especially when dealing with a large number of log messages.

3.3 `console.assert`

The console.assert() method allows you to assert whether a condition is true or false. If the condition is false, it will log an error message to the console.

const x = 5;

// Check if the condition x === 10 is true, if not, log the error message
console.assert(x === 10, "x is not equal to 10");

In this case, the condition being checked is x === 10, which compares the value of the variable x to 10. Since the value of x is 5, the condition is false. As a result, the console.assert method will log the error message to the console.

If you run this code in a browser's developer console, you would see an assertion error in the console output with the specified error message:

Assertion failed: x is not equal to 10

This is a helpful way to include runtime checks in your code and log informative messages if certain conditions are not met.

3.4 `console.count` and `console.countReset`

The console.count() method allows you to count the number of times a particular piece of code is executed. You can also reset the count using console.countReset().

function greet() {
  // Log and count the number of times "greet" is called
  console.count("greet");

  // Return a greeting message
  return "Hello!";
}

// Call greet() two times
greet();
greet();

// Reset the counter for "greet"
console.countReset("greet");

// Call greet() again
greet();

console.count("greet");: This line logs the number of times "greet" is called. The count is initially 1 when greet() is first called and increments with each subsequent call.

If you run this code in a browser's developer console, the output might look like this:

greet: 1
greet: 2
greet: 1

The first two calls to greet increment the count, and the third call, after the reset, starts the count again from 1. The count is specific to the label "greet."

3.5 `console.time` and `console.timeEnd`

The console.time() and console.timeEnd() methods allow you to measure the time taken by a block of code to execute.

console.time("timer");

for (let i = 0; i < 1000000; i++) {

// Some time-consuming operation

}

console.timeEnd("timer");

In the above example,

console.time("timer");: This starts a timer with the label "timer" when the loop
console.timeEnd("timer");: This stops the timer labeled "timer" and logs the elapsed time to the console.

If you run this code in a browser's developer console, the output will look like this:

timer: XXms

The "XX" will be replaced with the actual time taken by the loop to execute the time-consuming operation. This measurement is useful for profiling and understanding the performance of a specific code block or operation.

3.6 `console.trace`

The console.trace() method outputs a stack trace to the console. This can be helpful for debugging purposes to see the call stack leading to the current execution point.

function foo() {
  // Call the bar function
  bar();
}

function bar() {
  // Log a trace of the call stack
  console.trace("Trace:");
}

// Call the foo function
foo();

foo function: Calls the bar function.
bar function: Logs a trace of the call stack using console.trace.
foo is called: This triggers the call to bar, and the trace is logged.

If you run this code in a browser's developer console, the output might look something like this:

Trace:
bar @ (index):8
foo @ (index):3
(anonymous) @ (index):12

The output shows the call stack at the time console.trace was called. It includes information about the functions in the stack, such as the function names and their respective locations in the code. In this example, the call stack is displayed in reverse order, with the most recent function call at the top.

3.7 `console.dir`

The console.dir() method allows you to display an interactive listing of the properties of a JavaScript object.

const obj = { name: "Chris", age: 25 };

// Display an interactive listing of the properties of the object
console.dir(obj);

The console.dir method is commonly used to log an interactive representation of an object to the console. If you run this code in a browser's developer console, the output might look something like this:

Object
  age: 25
  name: "Chris"
  __proto__: Object

This output provides a visual representation of the object's properties, including their names and values. It also shows the prototype of the object (__proto__). The console.dir method is particularly useful when dealing with complex objects or nested structures, as it allows you to explore the object's properties in a more interactive way than console.log.

3.8 `console.clear`

The console.clear() method clears the console of all previous log messages.

console.log("Message 1");

console.clear();

console.log("Message 2");

In the above example, console.clear() will clear the console before logging "Message 2".

4. Best Practices for Console Logging

While console logging can be a powerful debugging tool, it's important to use it judiciously and follow best practices:

Avoid Excessive Logging: Too many log messages can clutter the console and make it difficult to find relevant information. Only log what is necessary for debugging.
Use Descriptive Messages: When logging messages, use descriptive labels to make it clear what each message represents.
Use Console Methods Wisely: Choose the appropriate console method (log, table, group, and so on) based on the type of data you are logging and how you want it to be displayed.
Remove Debugging Code in Production: Remember to remove or disable console logging statements in your production code to avoid unnecessary overhead.

5. Conclusion

Console logging is a powerful tool for debugging JavaScript code. By leveraging advanced console logging tricks such as console.table, console.group, console.assert, and others, you can streamline your debugging process and gain deeper insights into your code's behavior.

In this comprehensive guide, we covered various console logging tricks, along with examples demonstrating how to use them effectively. By incorporating these techniques into your development workflow and following best practices, you can become a more efficient and effective developer.

Experiment with these console logging tricks in your own projects to see how they can help you debug and understand your code better. Happy debugging!

How to Use Chrome DevTools – Simple Strategies for Smarter Web Development

Ophy Boamah — Thu, 15 Feb 2024 16:12:02 +0000

As a web developer, there are many tools out there – in addition to your code editor – that can make you more efficient.

It doesn't matter if you're just starting out or have been coding for years. Knowing how to effectively use Developer Tools (DevTools for short) can significantly boost your development process. You can edit pages on the fly, quickly spot issues, and deeply understand your site's performance.

All major browsers have their own DevTools that let you examine the code of a webpage, evaluate its metrics, and run some tests alongside. This article will discuss Chrome's DevTools, as it's the industry standard.

What is Chrome DevTools?

Chrome DevTools is a set of tools that are essential for diagnosing and solving web development challenges, directly within the Google Chrome browser.

It gives you direct access to a website's inner workings - to inspect HTML and CSS, debug JavaScript, analyze performance, and see the immediate impact of your code, all in realtime.

This direct access to a website's inner workings is crucial for diagnosing issues quickly and efficiently, ensuring your web applications are both performant and bug-free.

A grid of elements, console, performance and network panels screenshots

How to Open Chrome DevTools

To open DevTools in your Chrome browser, you can either:

Right-click on any webpage and select inspect from the list of options.
Use the shortcut (command + option + I on Mac or control + shift + I on Windows).
Click the three dot icon next to your profile picture on your Chrome browser, choose 'More Tools' and 'Developer Tools' from the second option box.

It usually opens in a split screen interface, either below your current webpage or beside it. Once open, its features line up as tabs at the top of the DevTools window. These tabs include: Elements, Console, Source, Network, Application, Security, Memory, Performance, Audits.

Use Cmd or Ctrl + Shift + C to open the Elements panel
Use Cmd or Ctrl + Shift + J to open the Console panel
Use Cmd or Ctrl + ] to move forward to the next panel
Use Cmd or Ctrl + [ to move back to the previous panel

Key Chrome DevTools Features

DevTools is packed with features essential for web developers to streamline various aspects of their workflow. Let's look at a few of them in some detail now.

Elements Panel

This panel is used for inspecting and modifying the HTML and CSS of a webpage in real-time, which is great for debugging layout issues or experimenting with new styles before applying them in your actual code. You also get to see how the DOM (Document Object Model) is structured.

Imagine fine-tuning your website's footer appearance (background color, font size) directly in your browser and seeing the results instantly.

With DevTools open, click on the Elements tab to access it.

A screenshot of Chrome DevTools' Elements panel

Console Panel

This panel serves as your interactive playground for JavaScript within the browser. Whether you're tracking down an elusive bug with a quick console.log() or experimenting with DOM elements, in the Console panel you can test snippets of JavaScript and view any logs or errors in the currently loaded webpage.

To use it, simply open DevTools and select the "Console" tab or use the shortcut (option + command + J on Mac or contrl + shift + J on Windows).

A screenshot of Chrome DevTools' Console panel

Network Panel

This panel gives you an overview of all network activity on your webpage – from tracking every resource that is loaded to how your site communicates with servers.

If you've wondered why your website takes forever to load or why some API requests seem to vanish into thin air, the Network panel is your go-to as it provides insights into the success or failure of API calls.

To access it, open DevTools and navigate to the "Network" tab.

A screenshot of Chrome DevTools' Network panel

Performance Panel

This panel is used for capturing and analyzing a website's performance metrics. It shows all the activities happening when interacting with a page.

When your web app starts to crawl under heavy usage, the Performance panel can pinpoint where the performance bottlenecks lie so that you can resolve these issues, ensuring your app runs smoothly.

With DevTools open, click on the "Performance" tab to use it.

A screenshot of Chrome DevTools' Performance panel

The above are only a handful of the panels available, but they're by far the most popular and must-knows. Using them properly will make your development processes more intuitive and rewarding.

Practical DevTools Use Cases

In the following interactive examples, I intentionally created the mini project in Codepen with issues to simulate real-world debugging scenarios using Chrome DevTools.

I figured it'd be a great way to highlight the practical uses of certain DevTools panels and features in identifying bugs and troubleshooting right in the browser.

Prerequisites

Chrome browser (Click this link to download)
A basic understanding of HTML, CSS, and JavaScript
Codepen

See the Pen Modal Window by Ophy Boamah (@ophyboamah) on CodePen.

How to Debug HTML and CSS with the Elements Panel

Our mini project contains a modal that, upon clicking, should display a modal window with some important information. But there's a bug preventing this from happening.

This situation sets the stage for a practical demonstration of how you can use the Elements Panel to troubleshoot and resolve styling and structural issues.


  <button class="show-modal">Click me to learn a secret 🤫button>

  <div class="modal hidden">
    <button class="close-modal">×button>
    <h1>Hey Ophy here 👋🏾h1>
    <p>
      I lead Women Who Code Frontend, a global remote community of 3,000+ women frontend devs and enthusiasts. Find us on beacons.ai/wwcodefrontend
    p>
  div>
  <div class="overlay hidden">div>

  <script src="script.js">script>

.hidden {
  display: none;
}

.modal {
  position: absolute;
  left: 50%;
  transform: translate(-50%);
  width: 70%;

  background-color: white;
  padding: 6rem;
  border-radius: 5px;
  box-shadow: 0 3rem 5rem rgba(0, 0, 0, 0.3);
  z-index: 10;
}

In our modal's HTML code above, we've added the class name 'modal hidden' which has a corresponding styling with the CSS property of display:none that is set to hide the modal when the page is loaded initially and only display it when the button is clicked.

✅ Step 1 - Initial inspection:

Attempt to trigger the modal by clicking on the 'Click me to learn a secret' button. Since we've set that up not to work, right-click on the area where the modal should appear and choose "Inspect" to open DevTools' Elements Panel.

✅ Step 2 - Diagnose visibility issues:

In the Elements Panel, locate the modal in the DOM to see that the modal is present but not visible. This confirms that the bug is caused within our CSS code display: hidden.

As soon as you click on the modal in the DOM, any corresponding CSS classes will be pulled up within Styles at the bottom section of the Elements panel. You can toggle some properties on and off or type others to see the effects in real-time.

Manually change the class name from modal hidden to modal block to trigger the right properties that'll cause the modal to show.

A screenshot of debugging the modal's HTML, CSS in Elements panel

Now the modal is visible, but it's displayed at the top – which is different from where we'd like it to be (that is, in the center of the page).

To change this, modify the transform property to translate(-50%, -50%) by adding the second -50% and ensure that top: 50%, and left: 50% are correctly set to center the modal on the screen.

✅ Step 4 - Enhance the appearance:

You can go further to refine the modal's appearance by tweaking its background-color, padding, or other stylistic properties directly within the Styles to achieve the desired look and feel.

A GIF fixing the modal in Chrome DevTools' Elements panel

Debug JavaScript with the Sources Panel

I added a bug in the JavaScript code of our modal mini project to prevent it from opening when the button is clicked.

In the real world, this would cause neither the open nor close commands to trigger any action, which would leave users unable to interact with the content and frustrated as a result. Let's troubleshoot and debug this issue in the Sources Panel.

In the code below, the openModal function is set to remove the indicated classes. However, this doesn't work because we deliberately misspelled hidden.

// Introducing a bug: Incorrectly spelling 'hidden' as 'hiddn'
const openModal = function () {
  modal.classList.remove("hiddn"); // Intentional bug
  overlay.classList.remove("hidden");

  // Fetch data from a real API and display in the modal
};

✅ Step 1 - Set up breakpoints:

Open Chrome DevTools and navigate to the Sources Panel. Here, find the JavaScript file that includes the modal functionality (in our example its pen.js).

The openModal function contains the logic for displaying the modal on the screen. This function will include a line where the modal element's class is manipulated to remove a "hidden" class.

Click on the number next to this code line in DevTools. A blue (or sometimes red, depending on the theme) icon appears next to the line number, indicating that a breakpoint has been set. This breakpoint will pause the execution of our JavaScript code as soon as it reaches this line.

A screenshot of setting breakpoints the modal's JS in Sources panel

Breakpoints pause code execution at critical points, allowing you to inspect the current state of variables and understand the flow of execution. This step is crucial for identifying where the code deviates from expected behaviour.

✅ Step 2 - Examine the code execution flow:

With our breakpoint in place, try to open the modal by clicking on its button. Execution of our JavaScript code now pauses at our breakpoint, which enables us to step through the code line by line.

This is an opportunity to observe variables, function calls, look for anomalies such as misnamed functions, incorrect logic, or uncaught exceptions that could explain why the modal isn't working.

In our case it's because we intentionally misspelled the class name hidden as hiddn. Fix that in the code to get the modal working again.

A GIF troubleshooting the modal bug in Chrome DevTools' Elements panel

Optimize Performance with the Network Panel

Here I've added a fetch function that makes an API call to a live endpoint (https://jsonplaceholder.typicode.com/posts/1). This is an excellent opportunity to explore the Network Panel's capabilities in diagnosing and understanding network-related problems.

From the code below, you can see that the openModal function doesn't only open the modal but also makes an API call to the jsonplaceholder endpoint to fetch some data.

const openModal = function () {
  fetch('https://jsonplaceholder.typicode.com/posts/1')
    .then(response => response.json())
    .then(json => document.getElementById('modal-content').innerText = json.title)
    .catch(error => console.error('Error loading the content:', error));
};

✅ Step 1 - Initiate the API call:

On the modal project UI, click on the 'Click me to learn a secret' button. Though the modal does not visibly activate, because of the fetch logic within the openModal function, an API call will be made.

✅ Step 2 - Network Panel Inspection:

Ideally, your Network Panel should be open before clicking the button, but you can also reverse the steps. Detailed insights on your API request such as the request's method, status code, response and the time it took to complete, will be available under headers, preview, response, initiator and timing tabs respectively.

A screenshot overview of API request in Network panel

✅ Step 3 - Simulating Network Conditions:

Use the Network Panel's throttling feature to mimic various network speeds like offline or slow 3G to see how the API request behaves under constrained conditions.

From this you can compare how different network speeds can affect application performance. This will teach you the importance of optimizing data loading strategies to enhance user experience, especially on slower connections.

A GIF observing API requests and responses in Chrome DevTools' Network panel

Conclusion

Bringing Chrome DevTools into your web development routine is not just about fixing bugs. It's about streamlining your workflow, making your sites more accessible, and boosting their performance.

Through our modal window mini-project, we've seen firsthand how DevTools can address a wide array of development challenges, but that’s merely scratching the surface of what it can do.

As you continue to explore its capabilities and familiarize yourself with its features, you'll find it's an invaluable companion on your web development journey – designed to make your development process not just faster, but also more rewarding.

debugging - freeCodeCamp.org

From Symptoms to Root Cause: How to Use the 5 Whys Technique

Here's What We'll Cover:

What is the 5 Whys Technique?

Origins of the 5 Whys Method

How to Conduct an Effective 5 Whys Analysis

Step 1: Define the Problem Clearly

Step 2: Ask "Why" Iteratively

Step 3: Validate Each Answer with Evidence

Step 4: Identify the Root Cause

Step 5: Define Corrective Actions

Real-World Example: Applying 5 Whys in an Engineering Scenario

The Problem:

Applying the 5 Whys

1. Why did the API return a 500 error?

2. Why did the query time out?

3. Why was the connection pool exhausted?

4. Why were some queries slow?

5. Why was an unoptimized query deployed to production?

Root Cause

What a Shallow Fix Would Look Like

What a Strong Fix Looks Like

Why This Example Matters

When to Use (and When Not to Use) 5 Whys

When to Use 5 Whys

When Not to Use 5 Whys

Benefits of the 5 Whys Technique

Simple and Easy to Apply

Encourages Deeper Thinking

Promotes System-Level Improvements

Works Well in Team Settings

Helps Prevent Recurring Issues

Common Pitfalls and Limitations

Stopping Too Early

Treating Assumptions as Facts

Focusing on Individuals Instead of Systems

Oversimplifying Complex Problems

Treating It as a Rigid Formula

Not a Replacement for Deeper Analysis

Tips for Using 5 Whys Effectively

Start with a Clear, Specific Problem

Base Every Step on Evidence

Keep the Chain Logical and Connected

Focus on Systems, Not Individuals

Do Not Force Exactly Five Steps

Involve the Right People

Turn Insights into Actions

Summary

How to Fix a Failing GitHub PR: Debugging CI, Lint Errors, and Build Errors Step by Step

Table of Contents:

Prerequisites

Understanding the CI Pipeline (What’s Actually Happening)

How a CI Pipeline Processes Your Pull Request

A Practical Debugging Workflow

Step 1: Fix Authentication and Permission Issues

Step 2: Run Lint Checks Locally

Step 3: Fix Common Markdown Lint Errors

1. Non-descriptive links

2. Line length violations

3. List indentation issues

Step 4: Fix YAML Inside Markdown Code Blocks

Step 5: Fix Build Errors After Lint Passes

Step 6: Debug Cascading CI Failures

Step 7: Handle Git Issues During CI Debugging

Option 1: Rebase (clean history)

Option 2: Merge (safer)

Pushing your changes safely

Key Takeaways

Conclusion

How to Troubleshoot Ghost CMS: Fixing WSL, Docker, and ActivityPub Errors

Error 1: SQLite Installation Failures on Windows

The Symptom

The Cause

How to Fix it:

How to Verify:

Error 2: Docker Container Exiting with Code 137

The Symptom:

The Cause:

How to Fix it:

How to Verify:

My Solution: `createDebugSetter`

Best Practices for Using `createDebugSetter`

Use `createDebugSetter` Only in Dev Mode

Use `createDebugSetter` with React DevTools

Place `createDebugSetter` in `utils`

Bonus: How to Convert `createDebugSetter` to a Hook