Omer Rosenbaum - freeCodeCamp.org

Stop Staring at a Blank Deck: How I Use Claude Code + Marp to Think Through Presentations

Omer Rosenbaum — Thu, 19 Mar 2026 17:36:41 +0000

The hard part of building a presentation is figuring out the story. What are you trying to say? What’s the structure? Which sections build on which? Where does the data go, table or bullets? Before the comparison or after?

What would help is having something to react to. Starting from zero is hard. Reacting to a draft is fast. “Move this before that” is way easier than “what should I say?”

That’s the workflow I want to show you. I use Claude Code + Marp to think through presentations. Claude helps me brainstorm the story, gives me a first draft to react to, and then I iterate, either through “conversation” or by editing the Markdown directly. The whole thing is a text file. 🎉

(I used a deck to think through this post. You can find it here.)

We'll cover:

The Workflow
- Brainstorm
- React
- Iterate
- Export
- Editable PPTX
Get Started in 5 Minutes
Use Case: This Very Article
Under the Hood
Wrapping Up

The Workflow

The process has four stages: Brainstorm, React, Iterate, Export. Let me walk through each one.

Brainstorm

You kick things off with a slash command (I will provide you with this command later):

/create-marp-deck API rate limiting

Claude starts by interviewing you, asks about the goal, audience, key points, any data you want to include. This partforces you to articulate the story before a single slide exists.

~/projects/my-app $ claude
> /create-marp-deck API rate limiting
─────────────────────────────────────────────

Claude: What's this presentation about? What should the audience
        walk away knowing?

User:   How our rate limiting works, which algorithm we chose and
        why, and what they need to know to configure it for their
        services.

Claude: Who's the audience?

User:   Platform engineering team — they know our stack but haven't
        seen the rate limiting internals.

Claude: What are the key points or sections you want to cover?

User:   1. Why we need rate limiting (the incident last month)
        2. Token bucket vs sliding window — our decision
        3. How to configure it per-service
        4. Monitoring dashboard walkthrough

Think of it as a lightweight brainstorm: you talk through what you’re trying to say, and Claude helps you structure it.

React

Once you’ve aligned on the structure, Claude generates the full Marp Markdown file and exports it. You get a solid first draft you can react to and reshape.

That title slide came from this Markdown:

# API Rate Limiting
## A Technical Deep Dive

**Team**: Platform Engineering
**Date**: February 2026

Is it perfect? Probably not. But now you have something concrete, with sections, structure, and a story, that you can push around. That’s so much faster than starting from a blank canvas.

When you go through the slides, you feel if the story is coherent and clear.

Iterate

While reviewing the draft, it'll inevitably spark ideas: “oh, I should add a comparison table here,” “this section is too dense, maybe split it into two,” “move this summary up to the top.”

One way to make such edits is to ask Claude Code to do that:

"Slide 6 is too dense. Split the algorithm comparison into
two slides, one for token bucket, one for sliding window."

> Slide 6 is too dense. Split the algorithm comparison into two
  slides — one for token bucket, one for sliding window.

Claude: I'll split slide 6 into two separate slides, one per algorithm.

  Edit presentations/api-rate-limiting.md
  ───────────────────────────────────────
  - # Algorithm Comparison
  - | Feature | Token Bucket | Sliding Window |
  + # Token Bucket
  + Tokens refill at a steady rate...
  + ---
  + # Sliding Window
  + Track exact timestamp of every request...

> Add a "Why we chose token bucket" callout to that first slide

Claude: Added a blockquote callout explaining the decision.

  Edit presentations/api-rate-limiting.md
  ───────────────────────────────────────
  + > We chose token bucket because it handles bursty traffic
  + > from our mobile clients without penalizing steady callers

You can also edit in VS Code with the Marp extension for live preview. Open the .md file, hit Ctrl+Shift+V, and you get the source on the left with rendered slides on the right. Claude Code edits the file, VS Code detects the change, and the preview updates automatically. (I keep both open side by side and it just works.)

Export

When you’re done, you get three files:

.md – the source (version-controlled, diffable)
.html – open in any browser, share via Slack
.pptx – open in PowerPoint, present anywhere

$ marp --no-stdin deck.md -o deck.html
[  INFO ] Converting 1 markdown...
[  INFO ] deck.md => deck.html

$ marp --no-stdin --pptx deck.md -o deck.pptx
[  INFO ] Converting 1 markdown...
[  INFO ] deck.md => deck.pptx

$ ls presentations/
api-rate-limiting.md
api-rate-limiting.html   ✓ open in browser, share via Slack
api-rate-limiting.pptx   ✓ open in PowerPoint, present anywhere

The skill runs the export commands automatically after generating the deck. A 15-slide deck converts in about 2 seconds.

Editable PPTX

The standard PPTX export renders each slide as an image – pixel-perfect, but you can’t edit the text in PowerPoint or Google Slides. If you need editable text, Marp has a --pptx-editable flag that uses LibreOffice under the hood to produce real text boxes.

The catch: LibreOffice creates text boxes that are too narrow, so text wraps and overlaps. The skill includes a python-pptx post-processing script that automatically widens the text boxes to fix this. Just ask for “editable PPTX” and the skill handles the rest – the LibreOffice conversion, the text box fix, everything.

Get Started in 5 Minutes

OK, are you ready? Here’s everything you need:

1. Install Marp CLI:

npm install -g @marp-team/marp-cli

2. Install the skill (via skills.sh):

npx skills add Omerr/claude-skills

This works with Claude Code, Cursor, GitHub Copilot, and other AI agents. You can also install manually ( see the repo for details).

3. Run it:

/create-marp-deck your topic here

4. Iterate:

React to the draft, refine through conversation or VS Code, and export.

That’s it. Four steps. Fork the repo and customize the conventions to match your style.

Use Case: This Very Article

Want to see this workflow in practice? You’re looking at it.

I wrote this article by first creating a slide deck using exactly the process I described above. I ran /create-marp-deck, answered the interview questions, got a first draft, and iterated until the story felt right. You can see the deck here.

Why start with slides? Because a deck forces you to be concise and to go through the story. If the story doesn’t flow across 15 slides, it won’t flow across 1,500 words either. The deck became my outline, and once I had a coherent structure there, writing the article was much easier.

So if you’re ever staring at a blank doc thinking “I should write a blog post about X,” try making a deck first. You might be surprised how much faster the writing goes when the story is already figured out. 😎

Under the Hood

If you’re curious about what makes this work, read on. If not, you’re all set. 🙌🏻

Marp: Markdown to Slides

Marp (Markdown Presentation Ecosystem) converts .md files into slides. Your deck starts with frontmatter:

---
marp: true
theme: default
paginate: true
size: 16:9
---

Four lines and you have widescreen, paginated slides. Slide breaks are just --- in the Markdown. Your presentation is a text file: version-controlled, diffable, and AI-editable.

The Skill File

You could just ask Claude Code to “make me a Marp presentation” every time. But you’d spend half the conversation explaining your preferred format, color palette, and slide structure.

Instead, I created a Claude Code skill (see it here), a reusable set of instructions that Claude follows whenever you invoke it. It has two parts:

An interview phase that gathers context before generating anything (the 5 questions from the brainstorm step)
A generation phase with the full Marp conventions: CSS palette, slide structure, breadcrumb pattern, formatting rules, and export commands

The full skill is about 200 lines. That sounds like a lot, but you write it once and then every deck you create follows the same polished conventions automatically.

Section Dividers

Each section of the deck gets its own gradient background. So when you’re presenting, the audience intuitively knows when you’ve moved to a new topic:

Applied via CSS classes in the skill:


# Part 1: The Problem

This is my favorite part of the whole setup.

Every content slide has a breadcrumb header at the top that shows where you are in the deck:

Algorithms > Implementation” with the current section highlighted in blue" style="display:block;margin:0 auto" width="600" height="400" loading="lazy">

See that header? “The Problem > Algorithms > Implementation”, with “Algorithms” highlighted in blue.

In Marp, this is done with a simple HTML comment:

The **bold** text renders in blue (via CSS header strong { color: #2563eb; }), while the rest stays gray. You set it once per section and it persists until you change it.

How often have you sat through a presentation wondering “wait, where are we?” 🤔

Wrapping Up

The hard part of presentations is telling a coherent story. Get yourself a first draft to react to, iterate until it flows, and export. That’s it.

If you want to try it: npm install -g @marp-team/marp-cli, run npx skills add Omerr/claude-skills, and then /create-marp-deck. You'll have a deck in minutes and a workflow you can reuse for every presentation after that.

About the Author

Omer Rosenbaum is the author of the Brief YouTube Channel. He’s also a cyber training expert and founder of Checkpoint Security Academy. He’s the author of Product-Led Research, Gitting Things Done (in English) and Computer Networks (in Hebrew). You can find him on Twitter.

Product-Led Research: A Practical Guide for R&D Leaders [Full Book]

Omer Rosenbaum — Wed, 18 Feb 2026 22:39:21 +0000

Your team needs to solve a problem, and there's no clear solution path. Multiple approaches might work, but you're not sure which. Success isn't guaranteed.

This is Research, not Development. And if you manage it like Development, things aren't going to go well for you.

Maybe you've seen this: A talented engineer spends three weeks trying different approaches, switching between them until the original problem is no longer relevant. Or a researcher declares after two days "it's impossible" and gives up, though later it turns out the problem was actually solvable. Or worse and most frustrating of all: Research succeeds technically but doesn't impact the product, and you realize you've been solving the wrong problem.

Here's the thing: managing Research is fundamentally different from managing Development.

Development has known solution paths, relatively predictable timelines, and measurable progress. You probably learned to manage Development, and know how to do it well. Managing Development is challenging, yet there are established best practices that are well known in the industry.

But Research? Research is inherently uncertain. The techniques that work for Development fail spectacularly for Research. Most organizations either approach Research by deploying techniques devised for managing Development, or entirely give up on managing Research and treat it as mystical work by brilliant people, where the best you can do is not interrupt.

It doesn't have to be this way.

What You'll Learn

By reading this book, you’ll gain practical tools for managing Research that create real product impact.

You’ll learn your two critical roles as a Research leader:

Ensure Research connects to product impact.
Ensure Research is done effectively.

You’ll understand why Research is different from Development. You’ll get concrete tools for managing Research execution, and learn proven heuristics for ensuring product impact.

Throughout the book, you'll see the mentioned methods applied to real challenges.

You’ll feel confident managing Research. You’ll understand when to use which tools. And you’ll know how to keep Research connected to product value while managing its inherent uncertainty.

Who Is This Book For?

Any engineering leader who manages Research, or has researchers on their team.

If you're a CTO, VP of Engineering, R&D Director, or Engineering Manager responsible for Research initiatives – this book is for you.

This book is for product-focused leaders who need their Research to ship value, not just produce interesting findings.

You’ll also notice that I use a casual style throughout the book. I believe that learning Research management should be insightful and practical. These are hard problems, and writing in an overly academic style wouldn't serve you well. This book is for you, written to help you succeed.

Who Am I?

This book is about you, and your journey managing Research. But let me tell you why I think I can contribute to that journey.

I am the CTO and co-founder of Swimm.io, a knowledge management platform for code. At Swimm, I've led multiple Research initiatives — from automatically keeping documentation in sync with code changes, to extracting business rules from legacy COBOL systems. We've faced genuine Research challenges where the path to success wasn't clear, where we didn't know if solutions were even possible.

I've managed Research in product environments where "interesting findings" aren't enough — where Research must ship and create measurable value. I've experienced the failure modes firsthand: Research that succeeded technically but didn't impact the product, teams that got stuck exploring endlessly, and the challenge of managing uncertainty while maintaining velocity.

I've also managed teams of researchers — brilliant people who needed guidance not on technical capability, but on connecting their work to product value and working systematically through uncertainty.

This book combines my experience leading product-focused Research with my background in teaching and making complex ideas practical and actionable.

The Approach of This Book

This is not an academic piece on Research methodology. When writing it, I had three principles in mind:

1. Practical: In this book, you will learn how to accomplish things in Research management. You will understand frameworks not just for the sake of understanding, but with a practical mindset. I sometimes refer to this as the "practicality principle" – which guides me in deciding whether to include certain topics, and to what extent.

2. Based on proven theory: While practical, the methods are grounded in Alan Schoenfeld's problem-solving research — a framework developed by studying how people actually solve uncertain problems. You'll see how Schoenfeld's components (knowledge, heuristics, control, beliefs) map directly to practical Research management tools.

3. Real examples: You will see these methods applied to actual Research challenges. Not toy problems, but real initiatives involving complex problems with genuine uncertainty. These examples show the methods in action, including when they're messy, when approaches fail, and how to adapt.

Structure of This Book

The book is organized in three parts:

Part 1: Foundations: Read this to understand what makes Research different and why it needs specialized management. This part is short — just enough to establish the framework that organizes everything else.

Part 2: Research Management Methods: These are tools that work for any Research.

Part 3: Ensuring Product Impact: Methods specifically for connecting Research to product value.

Why Is This Book Publicly Available?

In short, I'd like this book to get to as many people as possible.

If you would like to support this book, you are welcome to buy E-Book version, Paperback, Hardc over , or buy me a coffee. Thank you!

Feedback Is Welcome

I created this book to help you and people like you manage Research effectively and ensure it creates product impact.

From the beginning, I asked for feedback from experienced leaders and researchers to make sure the book achieves these goals. If you found something valuable, felt something was missing, or thought something needed improvement — I would love to hear from you.

Your feedback helps make this book better for everyone. Please reach out at: gitting.things@gmail.com.

Now, let's begin. In Chapter 1, you'll learn exactly what makes Research different from Development, and why managing them differently matters.

Note

This book is provided for free on freeCodeCamp as described above and according to Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International.

If you would like to support this book, you are welcome to buy E-Book version, Paperback, Hardc over , or buy me a coffee. Thank you!

Part 1: Foundations

Chapter 1 - What is Research?

To manage research effectively, you first need to understand what research is, and what it is not.

From now on, I will use Research (capital R) when referring to our specific concept of research in this book, to distinguish it from general uses of the word.

Consider this scenario: Your team needs to optimize a critical API endpoint. It's slow, users complain, and you know exactly what to do: profile the code, identify bottlenecks, apply standard optimization techniques. This is challenging work, but it's not Research.

Now consider this: Your team needs to automatically extract business rules from 40-year-old COBOL codebases, consisting of millions of lines of code where the original developers are long retired. You're not even sure if extracting these rules automatically is possible. Multiple approaches might work. Or none might. This is Research.

The Difference

The distinction isn't about difficulty or technical sophistication. It's about uncertainty of approach.

Throughout this book, I will adopt the following definition: Research is confronting problems where you don't know if solutions exist or which approaches will work.

Research confronts problems where:

You don't know if a solution exists.
Multiple approaches might work, but you don't know which.
The path to success is not immediately clear.
You may need to invent new techniques.

Development involves:

Applying known techniques to build specific features.
Following established approaches, even if complex.
Clear success criteria tied to working software.

I've found Alan Schoenfeld's model of problem solving [1] to be a useful framework for defining and analyzing research. Schoenfeld, studying how people solve mathematical problems, identified four** components that determine success when facing genuinely uncertain problems. His framework applies directly to software Research:

Schoenfeld's Framework for Problem Solving

1. Knowledge Base — What you know

Are you familiar with relevant tools, algorithms, and techniques?

For COBOL business rule extraction: Do you understand COBOL syntax? Static analysis? Program comprehension techniques?

Without the right knowledge, you will have to spend time acquiring it before making progress, and might miss options that could be obvious to someone with more background.

2. Heuristics — Strategies for approaching problems

We’ll cover heuristics in much more detail later. For now, here are some examples of effective heuristics:

"Work backwards from the desired output".
"Break the problem into smaller pieces".
"Try a simpler version first".
"List all assumptions and test each one".

For our COBOL business rule extraction case: "Start by manually extracting rules from one small program to understand what 'success' looks like".

3. Control — Monitoring and adjusting your approach

Recognizing when your current strategy isn't working. Deciding when to pivot to a different approach. Managing your time and resources effectively.

This is what separates experienced researchers from novices: it's not just what you know, and the heuristics that you may deploy, but when and how you use them. If you choose one approach, reflect on its effectiveness, and decide to try something different when needed, that's an example of control.

4. Beliefs and Attitudes — Your mindset toward the problem

Schoenfeld found that successful problem solvers held certain beliefs that helped them persist through challenges. Examples include:

"I can figure this out" vs. "I'm not good at this kind of thing".
"Problems have multiple solutions" vs. "There's one right answer".
"I should write things down and work systematically" vs. "I should solve this in my head".

These beliefs profoundly affect your ability to persist and succeed.

Why This Matters for R&D Leaders

You've likely seen this: A capable engineer spends three weeks on a Research task, trying approach after approach, until the original problem is no longer relevant. Or they declare after two days "it's impossible" and give up, though it turns out later that the problem was actually solvable.

The issue isn't (necessarily) capability. It's that Research requires different management, and different skills, than Development:

Knowledge base: Did they have the right background, or access to people who did?
Heuristics: Were they using effective problem-solving strategies, or just "trying things"?
Control: Did they know when to pivot? When to ask for help? When to break the problem down differently?
Beliefs: Did they believe the problem was solvable? That systematic approaches work better than random attempts?

The good news: All four components can be improved. People get better at Research through practice, exposure to effective heuristics, and environments that support good Control and healthy Beliefs.

The rest of this book provides concrete tools, like using a Research Tree, drawing backwards, and time-boxing methods, that put Schoenfeld's framework into action in a Product-led environment. These tools help you and your team apply better heuristics, maintain effective control, and build the beliefs that sustain successful Research.

But first, let's make sure we're clear on when you actually need these tools. The next chapter dives deeper into distinguishing Research from Development work.

Notes

** Actually, Schoenfeld (1992) described five components (which he terms "categories"), but I focused on four of them. For the curious reader – the one I skipped is called "Practices" – the habits and cultural norms of the mathematical environment that shape how a student approaches a problem. I chose to skip it as applying it to Research felt artificial.

References

[1] Schoenfeld, A. H. (1992). Learning to think mathematically: Problem solving, metacognition, and sense-making in mathematics. In D. Grouws (Ed.), Handbook for Research on Mathematics Teaching and Learning (pp. 334-370). New York: MacMillan.

Chapter 2 - Research and Development

Most R&D departments have plenty of Development work. Everyone agrees on what Development is. But Research? That's murkier.

Some claim every development task involves "research" – you have to test your code, try different things, read documentation. Is this Research?

Let's be precise.

Defining Research vs. Development

We established in chapter 1 the core distinction: Research involves fundamental uncertainty about whether solutions exist and which approaches will work, while Development applies known techniques to build specific features.

With this foundation, let's explore how to identify Research in practice.

A Quick Test

You're asked to reverse-engineer a specific compiled function: disassemble it and provide the equivalent code in C language. You know assembly, you know C, you have a disassembler. Is this Research?

No. You know how to proceed. It might take three days of careful work, especially if the function is complex, but it's not Research. You're applying known techniques, and know how to progress to a solution.

But if you need to understand how an entire program operates, and one possible approach is reverse engineering its compiled form, and you're not sure if that approach is even feasible time-wise or whether it will yield the insights you need? That's Research.

Key Indicators You're Doing Research

1. Fundamental Uncertainty About Solution Viability

You're asking "Can this even be done?" rather than "How should we do this?" This isn't about implementation details – rather, it's about whether the approach itself is viable.

2. Multiple Competing Approaches Without Clear Superiority

Research often means exploring several paths simultaneously, knowing that many will fail, to discover which approach (if any) can solve the problem.

3. Need for New Fundamental Techniques

You may need to invent new methods rather than adapting existing ones. Note: Not all Research creates new techniques, but the possibility exists.

Common Misconceptions

Misconception 1: Technical Complexity = Research

Many challenging Development tasks involve sophisticated algorithms, large-scale systems, or cutting-edge technologies without requiring Research approaches.

Building a distributed system with complex consensus algorithms? Challenging Development. Figuring out whether a distributed system can meet your latency requirements given your unusual constraints? Might be Research.

Technical complexity is not the same as Research.

Misconception 2: Using Advanced Algorithms = Research

Implementing machine learning pipelines with random forests or neural networks isn't Research – even though the underlying algorithms are sophisticated. The Research happened when those algorithms were first developed. However, if you are using those algorithms when trying to solve a problem where it's unclear if they will work at all, that could be Research.

Misconception 3: Research Can Be Managed Like Development

Perhaps the most damaging misconception. This leads to:

Demanding precise time estimates for uncertain work.
Expecting steady, measurable progress on fixed schedules.
Evaluating Research with Development metrics.

Research requires different approaches. This is exactly what the rest of this book addresses.

Misconception 4: Research Cannot Be Managed

The opposite extreme: treating Research as mystical work by brilliant people. The best a manager can do is not interrupt.

I've had the pleasure and privilege to work with many extremely skilled researchers. I can confidently say that this is simply not the case, as even the most talented researcher can benefit from skillful guidance.

Specifically, even the most talented researcher benefits from:

Clear connections between their work and product goals.
Structured approaches to exploring alternatives.
Regular checkpoints to assess direction.
Team collaboration and brainstorming.

Research is not magic, it can be managed effectively.

Your Role as Research Leader

When leading Product-led Research, your job has two parts:

1. Ensure the Research makes the biggest possible impact on the product

This is your most important responsibility. "Successful" Research that doesn't impact the product is a failed project. Your job is to maintain the connection between Research work and product value – not just at the start, but continuously.

This means:

Starting with clear product needs, not interesting technical questions.
Regularly validating that the Research still serves the product goal.
Making trade-offs between thorough exploration and shipping impact.

We'll cover this in detail in Part 3.

2. Ensure the Research is done in the most effective manner

Even brilliant researchers benefit from structured approaches. Your role is to help the team work systematically rather than randomly.

This means:

Helping identify which questions are worth answering.
Introducing better heuristics when the team is stuck ("Let's work backwards," "Let's time-box this investigation").
Preventing common failure modes like endless exploration or premature commitment.

(We'll cover this in detail in Part 2.)

Responsibility (1) is defining the right goals. Responsibility (2) is reaching these goals effectively.

The rest of this book provides concrete tools for both responsibilities.

Part 1 - Summary

Research is work where the path to success is not immediately obvious. It requires:

Knowledge of relevant domains.
Effective problem-solving strategies (heuristics) .
Monitoring and adjusting approaches (control).
Healthy beliefs and persistence.

Your role as a Research leader is to:

Ensure Research connects to product impact.
Ensure Research is done effectively.

You need different tools to manage Research versus Development – in order to make sure Research is done effectively. Part 2 provides those tools.

Part 3 will focus on connecting Research to product impact.

Part 2: Research Management Methods

Chapter 3 - Why Methodology Matters: A True Story

It was a late evening, and the classroom was filled with three dozen students. They were all sitting in front of their computers, working in silence. I was leading a cybersecurity training focused on reverse engineering. Each exercise included a single compiled program without its source code, with one goal: "understand how this program works." The output would be either a detailed document, an equivalent program implemented in a high-level language, or both.

This particular evening, the students were furiously working on reverse-engineering a game. The instruction was: "understand the game's rules, and document them thoroughly." The game had a user interface with two dimensions and could be played against the computer. Moving behind the students, I could see their screens with various reverse engineering tools open.

At some point we asked them to stop, turn around and look at the instructor. The instructor then provided a guided solution – this was a technique we used quite frequently, showing the students the "right" way to approach a problem they had spent some time tackling. The instructor opened the game, looked at the screen, opened the "File" menu, clicked on "Help" – and there it was, the entire description of the game rules.

The Lesson

The room erupted in nervous laughter. Some students looked embarrassed. Others seemed frustrated. But everyone understood the point.

These were capable people. They had the relevant knowledge base, in Schoenfeld's terms, as presented in chapter 1. That is, they had the relevant technical skills – they knew both assembly and C, they knew how to use disassemblers, debuggers, and all the sophisticated tools of reverse engineering. Yet they had missed something fundamental: checking if there was a simpler solution first.

This happens all the time in Research work.

A team spends weeks diving deep into a complex technical approach, when a simpler path existed that they never explored. Or they try one thing, then another, then another – never systematically evaluating which approaches make sense before starting.

The problem isn't capability. It's approach.

This is exactly why you need structured methods for Research management. Without them, even your most talented people will waste time, miss obvious solutions, and burn out trying random approaches.

As a reminder, in chapter 2, we discussed your role as a Research leader:

Focusing on responsibility (2), the illustration here shows how the same peak can be reached via difficult climbing (reverse engineering) or by using a hot air balloon (reading the Help menu). Part of your job as a Research leader is to help your team find the easiest path to the goal.

What This Part Covers

The rest of Part 2 provides concrete tools to prevent these problems. You'll learn:

The Research Tree – A visual framework for systematically exploring solution paths.
Time-boxing methods – How to limit exploration without killing creativity.

These aren't abstract concepts. They're battle-tested techniques that directly address the failure modes you've probably seen: teams spinning their wheels, giving up too early, or getting stuck on the wrong approach.

Let's get started.

Chapter 4 - The Research Tree

You've probably seen this: An engineer/researcher starts down one path, hits an obstacle, tries something else, hits another obstacle, then tries a third approach. Three weeks later, they're still stuck – or worse, they've built something that technically works but doesn't solve the actual problem.

The issue isn't persistence. It's that they never mapped out the solution space. They never visualized which approaches might work, which questions need answering, and how everything relates to each other.

The Research Tree solves this problem.

It’s a way to visualize and manage the Control component of Schoenfeld's framework from chapter 1 – monitoring and adjusting your approach. This is where most Research efforts struggle.

What Is a Research Tree?

A Research Tree is a living visual representation of your Research journey. It captures three things:

Possible solution paths – different approaches you might take.
Open questions – what you need to learn.
Closed questions – what you've already discovered.

Unlike a static plan, the Research Tree grows and changes as you learn (which is one reason I like the name "tree" 😇). You start with what you know, then update it continuously as you investigate. Dead ends get marked. New branches appear. Questions get answered and new questions emerge.

Think of it like this: You're exploring a cave system. You don't have a map – you're creating the map as you explore. You mark passages you've tried. You note which ones are dead ends. You write down questions: "Does this passage connect to the main chamber?" "Is there water down this route?" As you explore, you answer some questions and discover new ones you hadn't considered. You write down the answers you found, and the experiments you conducted to find them ("I tried going left, hit a wall after 50 feet").

Research works the same way. The Research Tree is both your map and your log.

Your First Research Tree

Let's build one together. Consider a common engineering challenge:

Goal: Reduce API response time from 800ms to under 100ms

(Note: despite not being a real Research task, I chose this example for its simplicity to illustrate the process of creating a Research Tree. It can also show you how Research Trees can be useful in a variety of scenarios.)

You start with a fundamental question that needs answering. What's the first thing you need to know?

Take a moment to think about this before reading on.

The first question is usually: Where is the bottleneck?

Until you answer this, you don't know which approaches make sense. Let's draw this:

You can use a simple pen and paper, a whiteboard, or a digital tool to draw this out. When creating your very first tree, I highly recommend doing it by hand – the physical act of drawing will help you feel comfortable with the process.

Now, how can you answer this question? What approaches might tell you where the bottleneck is?

You might identify:

Profile the application with a performance monitoring tool.
Add detailed logging to measure each operation.
Use database query analysis tools.

Let's add these as branches:

(Note: the Brown status means "uncertain, needs investigation" – more on this later.)

Each approach is an investigation you could run to answer the question.

The Power of Stopping to Think

Before we go further, notice what just happened. You stopped.

Instead of immediately jumping into "Let's add more logging!" or "I bet it's the database, let's check queries," you identified multiple possible approaches. You're looking at three different ways to answer the same question.

This is already valuable. Most engineers would have jumped straight into whichever approach came to mind first. Maybe you've done this yourself: spent two days adding detailed logging, only to discover later that a profiler would have given you the answer in 30 minutes.

By creating this tree, you've avoided that trap. You can see all the approaches before committing to any of them. It doesn't guarantee you will choose the "right" path - you can't always do that in advance – but it will minimize the chances of you omitting it completely.

Remember the reverse engineering students from chapter 3? They never created this tree. They jumped straight to the first approach they knew: disassemblers and debuggers. They didn't stop to think: "What are all the ways we could understand this game's rules?" If they had, they would have listed approaches like: reverse engineer the binary, check the Help menu, just play the game, examine config files, watch network traffic. And if they'd evaluated those approaches using the framework you're about to learn, "check the Help menu" would have scored perfectly: fastest feedback (30 seconds), lowest cost (zero), best coverage (complete rules). Instead, they spent hours on complex reverse engineering when a simple menu click would have worked. Don't be those students.

Now comes the critical question: Which branch do you try first?

Consider our tree again:

Choosing Your First Path

You're looking at three approaches: Profile, Logging, DB Analysis. Each is a heuristic (in Schoenfeld's terms, as presented in chapter 1) – a problem-solving strategy that might work. How do you decide which one to try?

Sometimes, the answer is obvious (I wouldn't really use a framework to check if there's a "Help" menu). Let me show you the framework I use when it's not so clear which path to choose. Ask yourself these questions:

1. Which gives the fastest feedback?

How long until you get an answer?

Profile: Can run in 10 minutes. Setup is probably 5 minutes.
Logging: Need to add logging code, deploy, wait for traffic – maybe 4 hours.
DB Analysis: Need to enable slow query log, wait for data – maybe 2 hours.

Fastest feedback wins. You want to learn quickly.

2. Which has the lowest cost?

What do you need to set up or change?

Profile: Just attach a profiler – no code changes.
Logging: Need to modify code, test, deploy.
DB Analysis: Need database permissions, might need config changes.

Lower cost wins. Why spend hours adding logging if you can get the answer without changing any code?

(In your environment, it may be different. Perhaps logging is really easy, while profiling is super hard to set up. I am not claiming that profiling is a better heuristic than logging – it depends on your circumstances.)

3. Which answers the most questions?

Some approaches answer not just your immediate question, but related questions, too.

Profile: Shows you CPU, memory, database, network – a complete picture.
Logging: Only shows what you logged.
DB Analysis: Only shows database queries.

Broader coverage wins. A profiler might show you that 70% of time is database queries and that 20% is network latency – information you wouldn't get from narrow approaches.

So this is an easy one. Profiling wins on all three criteria. That's your first approach to try.

Update your tree:

This doesn't mean the other approaches are bad, or that this one will necessarily turn out to be the best. It means that profiling is the best starting point given what you know right now.

What If Your First Choice Doesn't Work?

Let's say you try profiling and hit a problem: Your profiler can't attach to the production environment due to security restrictions.

This is valuable information. Mark Profile as Red and move to your next best option:

Now you try Logging. But notice: You didn't waste days trying Profile in production. You tried it, hit a blocker, immediately pivoted to Logging. The tree helped you move quickly.

Choosing When Everything Seems Equal

Sometimes multiple approaches look equally good. Profile and DB Analysis might both be fast, low-cost, and have decent coverage. How do you choose?

Pick one and move on. Don't spend an hour analyzing which approach to try for 30 minutes. The meta-work (deciding) shouldn't take longer than the actual work (trying it).

When approaches seem equal, ask: "Which one am I more familiar with?" or "Which one does the team have experience with?" Use your judgment, make a choice, and start learning.

The worst decision is no decision.

In general, this approach might feel like overkill. Should you really sketch out trees and compare branches before actually doing something?

The surprising answer is that while almost always it feels like overkill, almost every single time, it turns out to be worth it. Try it a few times and you will see for yourself.

Heuristics Can Be Combined

Sometimes you don't have to choose just one. You might run Profile and enable DB Analysis simultaneously. If they don't conflict and you have the time, parallel investigations can be powerful.

But be careful: Don't try to do everything at once. Start with your best option. If that doesn't fully answer your question, then add another approach.

How Answers Lead to New Questions

After marking "Profile" as red, you moved on to "Logging". You added a few indicative log messages and let the system run for a day. You discover: 70% of response time is database queries.

This answer eliminates the need for other approaches (you don't need DB Analysis now – you found the answer). But more importantly, it reveals new questions:

See how the tree grows? One answered question spawns two new questions. Each of these new questions will have its own approaches for answering them.

And you'll apply the same framework to choose which question to answer first: Which gives fastest feedback? Which has lowest cost? Which answers the most questions?

Let's expand "Which queries are slowest?":

Again, you'd evaluate: Which approach gives fastest feedback? Enabling slow query log is probably fastest, as you’d just flip a config flag and wait a few minutes.

You decide to enable the slow query log. After investigating, you discover: User profile queries are slowest: they make 15 separate database calls (N+1 problem).

(What is the N+1 problem in this context? It means that when fetching user profiles, the code first queries for a list of users (1 query), then for each user, it makes an additional query to fetch related data (N queries). If there are 15 users, that's 16 queries total. This is inefficient and slows down response time.)

This answer leads to a new question:

Now you have three solution approaches. Again, evaluate them:

Rewrite queries with JOINs: Fast to implement, proven pattern.
Add Eager Loading: Depends on your ORM, might be quick.
DataLoader Pattern: Requires learning new pattern, takes longer.

Rewrite queries with JOINs probably gives the fastest feedback if your team knows SQL well.

The Complete Picture

Let's see how the full tree looks after a few days of investigation:

(Note: for the "How many queries per request?" node – I skipped the approaches for brevity.)

Reading this tree:

We started with "Where is the bottleneck?" and chose profiling (fastest feedback).
We found out the profiler can't attach to the production environment due to security restrictions, so we pivoted to logging.
Logging revealed that 70% of response time is database queries.
That answer led to two new questions about specific queries.
For "Which queries are slowest?", we chose slow query log (fastest to enable).
Answering it led to another question about fixing the N+1 problem.
For "How can we fix N+1?", we evaluated the approaches and chose "Rewrite queries with JOINs" (team knows SQL, fastest to implement).
Meanwhile, "Can we reduce query count?" is still open and has its own approaches to investigate.

Color-Coding Status

When you're creating your Research Tree, you'll mark both questions and approaches with a particular status. Of course, the following specific colors are just suggestions – the important thing is to keep something consistently so you can quickly see the status at a glance.

For Questions:

Open: Not yet answered.
Closed: Answered (show the answer).
Blocking: Must answer before proceeding with an approach.

For Approaches:

Green: Viable, worth pursuing.
Brown: Uncertain, needs investigation.
Red: Dead end or not viable.

In our example above:

"Rewrite with Joins" is Green because we've identified that it addresses the specific N+1 problem and that the team is confident in implementing it.
"Redesign API" is Red because it would take too long for this project.
Other approaches are Brown because we haven't investigated them yet.

Additional Tips

Keeping the tree clean and simple is important, and obsessing over its looks and details really misses the point. That said, some readers will find benefits by adding a few more details to the tree, specifically:

Order: add a number next to a specific branch when tackling it, so it is easy to track which direction you tried first, which one followed and so on.
Pivot Explanations: if you chose to pivot from one branch to another, write why. This might help when you revise your decisions later, or when reviewing with your team (as described later in this chapter).

The Research Tree Prevents Common Pitfalls

The Research Tree with this decision-making framework addresses five critical failure modes:

1. Jumping on First Idea: Without a tree, people implement the first approach they think of. The tree forces you to identify alternatives and evaluate them systematically before starting.

2. Tunnel Vision: Even when considering alternatives in the beginning, people tend to lock onto one approach and not pivot from it even when it turns out to be the wrong choice. The tree makes alternatives visible and helps you not only choose the best starting point, but also reevaluate continuously.

3. Inefficient Learning: Teams might try expensive, slow approaches first when faster, cheaper ones exist. The decision framework helps you learn quickly.

4. Answering Questions You Don't Need To: Teams waste time investigating interesting but irrelevant questions. The tree shows how questions connect – you only need to answer questions that lead to your goal.

Time to Practice

Open your favorite drawing tool, or just grab a piece of paper. Think of a Research problem you're currently facing or recently faced.

Now answer these questions:

What's your goal? Write it at the top.
What's the first question you need to answer? Write it below the goal.
What are 2-4 approaches to answer that question? Draw them as branches.
For each approach, evaluate:
- How fast is the feedback?
- What's the cost?
- How much does it answer?
Which approach scores best? Mark it "TRY FIRST".

Actually do this. Don't just read and think "I understand." Drawing the tree and evaluating approaches forces you to be explicit, and you'll immediately see gaps in your thinking.

I'll wait.

...

Don't worry about me, I'm enjoying some really great coffee in the meanwhile.

...

Done? Good. You now have your first Research Tree with a clear starting point.

Using the Tree with Your Team

Research Trees become even more powerful when shared with a team. It actually provides you, the Lead, with a way to see what directions the team is executing upon, and why. Your job here is to make sure the framework is used. Help your team stop and ask: are we asking the right questions? Are there approaches that we missed? Are we choosing the right approach?

During Planning:

Draw the tree together as a group.
Brainstorm questions and approaches.
Evaluate approaches using the framework (fastest feedback, lowest cost, best coverage).
Everyone sees why you're trying a particular approach first.

During Execution:

Update the tree as you learn.
When stuck, revisit the tree to identify alternative approaches.
Make sure you consider whether you are asking all of the important questions, and whether you are considering all relevant approaches.
If you pivoted from a branch, explain your reason and ask if someone can challenge your logic.
Conduct regular tree reviews (weekly or bi-weekly).

Note that the Research Tree is also useful for one-on-one sessions: you can review the tree with individual team members to understand their progress and help them choose next steps. It actually makes the Control component of Schoenfeld's framework much easier to manage - as you see the variolus questions and approaches laid out visually.

Tools

Pen and paper (seriously, this works great).
Whiteboard (for team sessions).
Miro, Mural, or similar digital whiteboards.
Mind mapping software (XMind, MindNode, and so on).
Even a simple text file with indentation.

The tool doesn't matter. What matters is that the tree exists, is visible, and gets updated.

Pro Tips

Start with the most important question

Don't try to list all possible questions upfront. Start with the one question that, if answered, would most clarify your path forward. Answer it, then see what new questions emerge. More on finding the questions to start from in chapter 7.

Show how answers lead to new questions

When you close a question, immediately ask: "What new questions does this answer reveal?" Draw those as branches from the answer.

Update questions weekly

In your weekly check-ins, explicitly review: Which questions did we close? What did we learn? Which new questions emerged? Which questions are blocking progress?

Re-evaluate when context changes

If you learn something new that changes the evaluation (maybe a tool you thought was fast turns out to be slow), re-evaluate your approaches. The tree is living – update it.

Recap - The Research Tree

The Research Tree is a living visual framework that:

Shows questions you need to answer to reach your goal.
Maps approaches for answering each question.
Helps you choose the best starting point for each question.
Prevents jumping on the first idea without considering alternatives.
Captures how answers lead to new questions.
Tracks status of questions (open/closed/blocking) and approaches (green/brown/red).
Documents the investigation path so the team understands why decisions were made.
Evolves as you learn – questions get answered, new questions emerge.

Key structure:

Goal → Question → Approaches to answer it → Answer → New questions

Decision framework for choosing approaches:

Which gives fastest feedback?
Which has lowest cost?
Which answers the most questions?

In the next chapter, you'll learn how to manage execution using time-boxing and decision points to keep your Research moving forward without getting stuck.

Chapter 5 - Time-Boxing Research Explorations

In the previous chapter, you learned about the Research Tree, a powerful tool for visualizing and managing Research efforts. The tree helps you systematically explore different solution paths and keep track of open questions. It also helps you decide which path to try first.

But once you've chosen a branch, how long should you pursue it before stepping back to reconsider?

The Problem: Research Without Time Limits

A researcher investigates whether a machine learning model can predict code complexity. Day one goes well. Day two, they need more features. Day three, a different architecture might work better. They switch. Day four, the new architecture needs different preprocessing.

Two weeks later, they're still on this path. When you ask about trying alternatives, they say "I'm close. Just need a few more days."

Three weeks in, they admit this approach isn't viable. Meanwhile, a simpler approach sat unexplored on the Research Tree.

Without a defined checkpoint, there's no natural moment to ask: "Given what I've learned, is this still the best path?" The sunk cost fallacy (continuing an approach because you've already invested time, rather than because it's still the best option) takes over. This is extremely common by Researchers, who tend to be dedicated, brilliant people who get fixated on problems they try to solve.

Time-Boxing: Creating Decision Points

Let's face it: it's hard, even impossible, to estimate how long a Research task will take. But you should still provide time limits based on how long you're willing to invest before reconsidering.

Time-boxing provides mandatory decision points.

Note that we are not talking about deadlines, but decision points. These are moments where you stop and evaluate: What did I learn? Is this still the most promising path?

As a rule, for every task longer than a day, define a time limit. For example: "I'll spend three days investigating whether we can cluster files into logical folders based on their I/O operations."

Three things can happen:

Early success: The researcher figures it out in a few hours. Done early, move to the next question.
Early blocker: After one hour, they discover they don't have access to filenames. They can immediately reconsider: "Without filenames, is this viable?"
Time box expires: Three days pass with partial progress. Now comes the mandatory decision point.

The Decision Point

When your time box expires, stop. Pull out your Research Tree (yes, you already love that Tree). Ask three questions:

1. What was your goal in pursuing this direction?

Which question were you trying to answer? Why did you choose this approach over alternatives?

2. What did you learn?

Document your discoveries, even if incomplete:

"The approach works but needs more sophisticated algorithms than we thought."
"We need data we don't currently have."
"This is harder than expected – would take at least 2 more weeks."
"We're 70% there – just need to handle edge cases."

3. Given what you learned, is this still the most promising path?

Look at your Research Tree. You have other branches. Given what you now know, is continuing the best use of time?

You have three options:

You can continue with a new time box: "I've solved the core challenge. One more day for edge cases." Define the new time box. The key: you consciously decided to continue based on what you learned, not inertia.

You can pivot: "This would take two more weeks, and I'm not confident it'll work. There's a simpler approach on my tree." Mark this branch Red. Move to a different branch.

Or you can reconsider the question: "Files in this codebase don't have clear I/O patterns. Maybe I should try clustering by function dependencies instead." Go back to your tree and identify a different question.

Example: Detecting God Objects

You're detecting God Objects (classes that do too much) using static analysis. Your Research Tree shows three approaches: complexity metrics, method naming patterns, or machine learning on AST features.

You choose complexity metrics (fastest feedback, lowest cost) and set a 2-day time box.

Day 1-2: You implement cyclomatic complexity and SLOC metrics. Results: 65% accuracy, but 40% false positives.

Decision point: You stop. Looking at your tree, method naming patterns might provide complementary information. You decide to spend 1 day exploring whether combining both approaches improves accuracy.

Day 3: Combined approach: 78% accuracy, 25% false positives. Promising.

New decision: Time-box 3 more days to refine and test on larger codebases.

Notice what happened: The 2-day time box forced assessment before perfecting the first approach. You learned combining approaches was better. Without time-boxing, you might have spent a week perfecting complexity metrics alone.

How to Set Time Boxes

When choosing a branch on your Research Tree, ask: Is this shorter than a day? A few days? Longer than a week?

Shorter than a day: Just do it. Don't overthink time-boxing for tasks this short.

A few days (2-7 days): Time-box for 2-5 days. I usually time-box for slightly less than my estimate – if I think 4 days, I set 3 days. This forces reflection based on learning, not arbitrary completion.

Longer than a week: Time-box for one week maximum. Even if you're making progress, a week is long enough that stepping back is valuable.

Don't overthink the exact duration. The goal is creating natural stopping points. The specific number matters less than having some checkpoint.

Critical: Time-boxing is not about pressure. You're not failing if the time box expires without solving the problem. That's expected in Research. What time-boxing does is force moments of reflection that prevent weeks spent on directions you'd have abandoned if you'd stopped to reconsider.

Using Time-Boxing with Your Team

When managing researchers, set time boxes together during planning. Make sure they understand it's a decision point, not a deadline. Schedule the review explicitly.

At the review, look at the Research Tree together. Ask the three questions. Make the decision collaboratively. This prevents researchers from getting stuck without asking for help, and makes pivoting feel like a positive decision rather than failure.

Integration with the Research Tree

Time-boxing combines naturally with the Research Tree (chapter 4): each branch gets a time box, and when it expires, the tree shows your alternatives.

Time-boxing creates moments to ask "given what I've learned, what should I do next?" The Research Tree helps you answer that question.

Recap

For tasks longer than a day, set a time limit (2-5 days for medium tasks, max 1 week). When time expires, stop and evaluate using your Research Tree: What was my goal? What did I learn? Is this still the best path?

Time boxes acknowledge that research estimation is hard. They prevent sunk cost fallacy and endless exploration. They're not about pressure or finishing "on time" – they're about forcing reflection instead of momentum-driven continuation.

Combined with the Research Tree, time-boxing gives you control over research execution while respecting its inherent uncertainty.

Part 2 Summary

Effective Research requires structured approaches, not just technical skill:

The Research Tree visualizes solution paths, open questions, and closed questions.
Time-boxing prevents endless exploration while preserving flexibility.
Systematic evaluation helps you choose the best path forward.

In chapter 2, I argued that your role as a Research leader is to:

Ensure Research connects to product impact.
Ensure Research is done effectively.

This part handled the latter part of your role: how to ensure Research is done effectively.

Within this part, your role is to:

Ensure the team uses these methods consistently.
Help identify when to pivot, when to persist, and which questions matter.

These tools prevent common failure modes: jumping on the first idea, tunnel vision, inefficient learning, and wasting time answering irrelevant questions.

Part 3 shows how to connect Research to product impact.

Part 3: Ensuring Product Impact

In chapter 2, I argued that your role as a Research leader is to:

Ensure Research connects to product impact.
Ensure Research is done effectively.

Part 2 addressed (2) above – ensuring Research is done effectively – with tools that work in ANY research context. This part provides the complete answer to (1) – ensuring Research connects to product impact:

First, choose research that matters (chapter 6).
Then, work backwards from product value (chapter 7).
Continuously validate with end-to-end iterations (chapter 8).

Chapter 6 - How to Choose Research Initiatives

The very first step in making sure your Research impacts the product is choosing the right thing to research. And, just as important, avoiding Research that won't impact the product.

Research initiatives can start from two different places:

From a problem the product is facing.
From a technological opportunity.

1. Starting From a Concrete Problem

The most promising way to find research initiatives that will have a big impact on the product is to start from an acute problem that the product is facing.

At Swimm, we allowed users to write documents about their code, but inevitably, the code changed, and the documentation became outdated. This made writing documentation not worth the effort in the first place. We needed to find a way to make sure the documentation stayed up to date automatically, with a good user experience. This was a clear problem we faced, and we didn't know if it was even possible to solve.

Consider a different example: a medical company that wants to diagnose a disease based on a few blood samples. Currently, they have an algorithm in place, but it's not very accurate. Specifically, it yields too many false positives. They need to find a way to improve their prediction accuracy. This is a clear problem the product is facing, and it doesn't have a clear technological solution.

In both cases, the problem is clear, and its impact on the product or company is clear. At the same time, the solution is not clear, and it's not certain that a solution will be technologically feasible.

2. Starting From a Technological Opportunity

When generative AI became popular, many companies started exploring how to leverage it to improve their products. This is an example of an emerging technology that can enable new product features.

The same can happen with smaller, more specific technologies, and not necessarily new technologies – sometimes technologies that the relevant teams just familiarized themselves with. For example, if a researcher reads a paper about a new way to parse source code, that researcher might have an idea for a new product feature that can leverage this technology.

While many good ideas come from technological opportunities, it's important to remember that the real impact of Research is determined by the product, not the technology. It's far more risky to pursue a technological opportunity than a concrete problem. If you do start a Research based on a technological opportunity, your responsibility is to make sure that the technological opportunity, if pursued successfully, will indeed have a big impact on the product.

Problem-Driven vs. Opportunity-Driven: A Comparison

Aspect	Problem-Driven Research	Opportunity-Driven Research
Starting Point	Customer pain point or product limitation	New technology or technique becomes available
Product Impact Clarity	High – you know exactly what problem you're solving	Low to Medium – you're searching for problems this technology can solve
Risk Level	Lower – you know there's demand if you succeed	Higher – solution might not match any important problem
Validation	Problem already validated through user feedback	Needs validation that the solution matters to users
Examples	Swimm's auto-updating docs; Reducing false positives in medical diagnosis	"How can we use LLMs in our product?"; "This new parsing technique could enable..."
Success Criteria	Did we solve the problem?	Did we find a valuable use case AND solve the problem?

Problem-driven research starts with validated demand: you know that the goal is worth pursuing. Opportunity-driven research starts with a hammer looking for nails. Sometimes you find valuable nails, but it's riskier.

Should You Pursue a Research Initiative?

Say you have a research initiative – an idea you'd like to research. To know whether you should pursue it, you should be able to answer a simple set of questions:

1. Product Impact – If the Research succeeds, how big will the impact be?

This is the most crucial question.

For problem-driven research, this is usually straightforward: "If we solve automatic doc updates, we'll retain 40% more customers who currently churn due to outdated docs."

For opportunity-driven research, you need to work harder: "If we use LLMs for code analysis, we could enable X feature, which would help Y users save Z hours per week, translating to $W in additional revenue."

If you're not convinced a successful Research result would make a huge impact on the product, it's probably not worth pursuing at the moment (until you are convinced). "Huge impact" means:

Solving a top-3 customer pain point, OR
Enabling a new product capability that significantly expands your market, OR
Reducing a major cost or risk factor

Anything less, and your resources are better spent on Development work with clearer ROI.

2. Time to Impact – How long until we see product value?

Time estimation is always hard in software. This is true for Development tasks, and even more so for Research. You learned ways to manage this uncertainty during Research in chapter 5, but even at this early pre-Research stage, you should consider the timeline.

Ask yourself two related questions:

How long for the Research itself? (days? weeks? months?)
How long from successful Research to product impact? (immediate integration? requires significant Development? needs market validation?)

The total timeline matters because:

Research that takes 6 months but delivers immediate product value might be worthwhile.
Research that takes 2 months but requires 8 more months of Development might not be worth it if your product roadmap can't accommodate that.
Research that takes 1 year with uncertain outcomes probably isn't worth pursuing unless the potential impact is transformational.

Of course, the actual timespans vary greatly by context (and specifically by the company you work for).

Consider also whether you can achieve incremental value. Can you get some product impact in 3 months even if the full solution takes 9 months? This significantly de-risks longer Research initiatives.

3. Resources – Do you have what you need?

Research requires specific resources beyond just "engineering time":

Knowledge: Do you have team members familiar with the relevant:

Technical domain (for example, NLP, compiler design, distributed systems)?
Business domain (for example, medical diagnostics, financial regulations)?
Similar problems solved elsewhere?

If not, can you acquire this knowledge in reasonable time? (Reading papers for a week: reasonable. Earning a PhD: not reasonable.)

Capacity: Can you dedicate someone (or multiple people) for the expected duration? Research requires sustained focus – splitting someone 10% on Research and 90% on urgent product work rarely succeeds.

Dependencies: Do you need:

Access to specific data or systems?
Collaboration from other teams?
External expertise or consulting?
Budget for tools, cloud resources, or datasets?

If critical resources are unavailable or expensive to obtain, the initiative may not be viable.

Pre-Research Checks

You might not have clear answers to all three questions above. In that case, it makes sense to spend time answering them before actually pursuing the Research. This phase should be limited in time – ideally a few days, at most a week or two.

In this stage you might:

Interview customers and business stakeholders to understand the real impact of solving the problem.
Read about the technological aspects and previous research done in this field to assess feasibility and timeline.
Consult with people who have faced similar challenges (internal experts, academic contacts, or practitioners in your network).
Run quick feasibility tests – not full Research, but simple checks like "Can we even access the data we'd need?" or "Does this library exist in our language?"

After pre-research checks, you should have a clear answer: "Yes, we should pursue this because the impact is X, the timeline is roughly Y, and we have (or can get) the resources we need."

If you're not confident about substantial product impact, don't start the Research. This might sound harsh, but it's crucial: unfocused Research that doesn't connect to product needs wastes your most valuable resource – talented engineers' time and attention. One way to enhance your certainty about product impact is described in chapter 7.

The next chapters assume you understand the product impact of successful Research outcomes, and will help you ensure you actually achieve this impact as quickly as possible.

How to Choose Research Initiatives - Summary

Research initiatives start from either:

Problem-driven: A clear product need (strongly preferred)
Opportunity-driven: A new technology (higher risk)

Before pursuing Research, answer three questions:

Product Impact: Will success create huge value?
Time to Impact: How long until we see product results?
Resources: Do we have the knowledge, capacity, and dependencies?

Run pre-research checks (days, not weeks) to answer these questions if unclear.

Only pursue Research when you're confident about substantial product impact.

The next chapter shows how to maintain that product connection throughout the Research process.

Chapter 7 - Drawing Backwards

So you've chosen a research initiative, and done so correctly (following chapter 6). Now, how do you start working on it?

Most teams start by diving into technical challenges: parsing COBOL, building callgraphs, implementing algorithms. But there's a more powerful approach that ensures your Research actually impacts the product: start from the end and work backwards.

This heuristic – working backwards from your goal – is one of the most valuable problem-solving strategies you can use. Let me show you why with a simple game.

The Spiral Game

Consider this game:

The rules are simple:

The pawn starts on spot 41.
On each turn, a player moves the pawn backward between 1 to 6 spots. That is, if the pawn is on spot 41, you can move it to any spot from 35 to 40.
The player who moves the pawn to spot 1 wins.

Take a moment: If you go first, how would you play to guarantee a win?

(I do encourage you to take a moment and try this for yourself first.)

Most people start thinking from the current position (spot 41) and try to calculate forward: "If I move 3 spaces, they can move 2, then I can move 4..." This quickly becomes overwhelming – too many possible moves to track.

But if you work backwards, the solution becomes clear:

Starting from the end (spot 1):

To win, you need the pawn on spot 1 on your turn.
Your opponent just moved, so the pawn could be on spots 2-7 (since they moved 1-6 backward from wherever it was).
For any spot from 2-7, you can move directly to spot 1.
Conclusion: If the pawn is on spots 2-7 at the start of your turn, you win.

Continue working backwards (from spots 2-7):

You want your opponent to land on spots 2-7.
From spot 8, no matter what they do (move 1-6), they land on spots 2-7.
Conclusion: If you get the pawn to spot 8, you guarantee a win.

Continue working backwards from spot 8:

Notice that you've just created a "new" game: you no longer need to land on spot 1 in order to win. It's enough that you land on spot 8 – as from there, you can win no matter what your opponent does.

So, how do you ensure you can land on spot 8?

Spots 9-14 all allow moving to spot 8.
So if your opponent starts their turn with pawn on a spot between 9-14, you can force it to 8.
Which means you do not want to land on spots 9-14, but you do want to land on 15 (which will force your opponent, in turn, to land somewhere between 9-14).

Again, you've just created a "new" game: where your goal is to land on spot 15. From there, you already know how to win.

You can keep going like this, drawing backwards – would you want to land on spot 16 or not? How about 17? Until, at some point...

The pattern emerges:

Safe spots for you: 1, 8, 15, 22, 29, 36
From any of these, your opponent cannot avoid giving you another safe spot
These are all numbers of the form: 1 + 7n

The winning strategy:

From spot 41, move 5 spots backward to reach spot 36 (a "safe spot").
No matter what your opponent does, you can always move to the next "safe spot".
Eventually, you reach spot 1 and win.

Notice what happened: By working backwards from the goal, you discovered the systematic solution. Working forward from the start position would have been much, much harder.

You actually deployed a second, powerful heuristic here: considering specific cases (1, 8, 15, 22, 29, 36) – and generalizing (1+7n). This heuristic is also very common in research, though not specifically when aiming to ensure product impact.

How to Apply Drawing Backwards to Product-led Research

Let's connect this to Product-led Research. When you face a complex Research challenge, the question isn't "What technical problem should I solve first?" but rather:

"If the Research succeeds, what would the result look like?"

This forces you to:

Connect to product impact: You must envision the end state that creates value.
Work systematically: Like the spiral game, you identify the chain of dependencies backward.
Validate assumptions: Before solving sub-problems, ensure they lead to your goal.

Let me show you how this worked in practice at Swimm.

Case Study: Extracting Business Rules from COBOL

At Swimm, we wanted to automatically generate documents from COBOL codebases that included all the extracted business rules.

Quick context on business rules: Business rules are the constraints, conditions, and actions embedded within software that reflect organizational policies. For example, in money transfer logic:

A customer cannot transfer more than their available balance (overdraft limits notwithstanding).
High-value transfers require additional verification.
Cross-currency transfers must apply current exchange rates.

Some sources define business rules with three elements: Event, Condition, Action:

ON 
IF 
THEN 
ELSE

Our goal was to extract all business rules from a COBOL codebase. While challenging in any codebase, it's particularly acute in legacy COBOL code. (If you're interested in the technical details, see this post.) For this book, it's sufficient to know that many research attempts over the last few decades have tried different approaches to face this challenge.

Where Do You Even Start?

Faced with this problem, you might think:

"Should I build a callgraph of all functions? That means I need to parse COBOL code..."
"Should I create a COBOL parser first?"
"Maybe I should read academic papers about program comprehension?" (Good practice during the pre-Research checks phase from chapter 6)
"Perhaps I should track COBOL variables through the codebase?"
"Should I distinguish business conditions ('if requested transfer amount > available balance') from technical conditions ('if variable not initialized, show error')?"

Each of these might require its own research effort. Where do you start?

Drawing Backwards: Start with the End Result

Drawing backwards made us ask:

"If the Research succeeds, what would the result look like?"

When we first asked ourselves this question, we weren't sure. We knew from our clients that they wanted extracted business rules, but we couldn't know what the "ideal" output would look like.

So we decided, before tackling any technical challenges, to manually create documents showing extracted business rules from sample programs. We did this completely manually: no parsing, no algorithms, just understanding COBOL code ourselves and writing documentation.

We did this for various types of applications from different codebases, and learned:

There's no single "right" way to construct such a document.
The output structure differs from one program to another.
Certain patterns appear consistently across business logic.

By creating these documents manually, we formed a hypothesis: "This is what the output should look like, which means this output would make the biggest impact on the product. This is our north star."

But was it actually the north star?

Validating the Hypothesis

Once we manually wrote the documents, it was time to verify our hypothesis. With concrete output in hand, we could:

Discuss internally within the team – get feedback from engineers who understand both COBOL and our product.
Reach out to clients – show them the actual output and ask: "Does this solve your problem?"

We deliberately refrained from solving hard technological challenges before knowing where we were aiming.

Working Backwards Through Sub-Problems

The manual documents also gave us something crucial: a concrete example to analyze backwards.

For instance, we saw that our documents listed many conditions. This made us realize we would (most probably) need to:

Find conditions in the code.
Filter out business-related conditions (vs. technical conditions).
Explain the condition in the document.

Here's where drawing backwards becomes powerful: We tackled (3) before (2), and (2) before (1).

Why? Because we needed to solve (3) to reach our goal – creating impactful documents.

This means we mocked the first two steps – we assumed we already had a way to find conditions and filter business-related ones. So we already had our list of business conditions, and now the challenge was: for each condition, explain it clearly in the output document.

This approach prevented us from spending weeks on tasks (1) and (2), only to discover that our explanation approach (3) didn't actually work. If we can't accomplish (3) assuming (1) and (2) are solved, then perhaps this entire approach isn't viable, and we need to reconsider alternatives.

The logic is clear: While solving (3) ultimately requires solving (1) and (2), if we fail at (3) even with (1) and (2) accomplished, then pursuing (1) and (2) might not be worth the effort at all.

What does mocking a dependency look like in practice? In our case, mocking steps (1) and (2) meant manually identifying a handful of business conditions from the COBOL code ourselves, rather than building automated systems to find them. We created a small, hand-crafted list of conditions that we knew were business-relevant and existed in our sample programs, and used that as input for testing our explanation approach in step (3).

Why Drawing Backwards Is So Powerful

The advantages of drawing backwards become clear from both the game examples and the COBOL case study:

1. Forces Connection to Product Impact

Like working backwards from spot 1 in the spiral game, starting with "what does successful output look like?" forces you to think about end-user value. You can't start drawing backwards without a clear goal. This prevents the common failure mode of technically interesting Research that doesn't impact the product.

2. Provides a System for Progressing

When Research seems like a huge, daunting task with endless options, working backwards gives you a systematic approach. Just as the spiral game became trivial once you worked backwards to identify the safe spots (1, 8, 15, 22...), Research becomes more manageable when you work backwards from the desired output to identify the dependencies.

3. Validates Each Step Before Investment

Working backwards lets you validate that each sub-problem actually contributes to your goal before you invest significant effort. In the COBOL example, we could verify that our explanation approach (step 3) worked before spending weeks on finding and filtering conditions (steps 1 and 2).

Practical Application: Your Research Tree

Drawing backwards integrates naturally with the Research Tree from chapter 4. When you create your tree:

Start with the end:

Root of tree: "Generate impactful business rule documentation".
First question: "What should successful output look like?"
Approach: Create manual examples.

Then work backwards:

Once you have output, ask: "What do we need to produce this?".
This reveals the actual sub-questions and their dependencies.
Each branch represents a prerequisite you need to solve.

Validate before going deeper:

Before pursuing any branch deeply, ask: "If I solve this, does it actually get me closer to the goal?"
Mock out dependencies to test approaches cheaply.
Use time-boxing (from chapter 5) to limit exploration of any branch.

Summary: Drawing Backwards

Drawing backwards is the heuristic of starting from your desired end state and working systematically toward your current position.

In Product-led Research, drawing backwards means:

Start by defining what successful output looks like (often manually or semi-manually).
Validate the output with stakeholders before technical work.
Work backwards through dependencies, solving them in reverse order.
Validate that each step contributes to the goal before major investment.

This heuristic ensures that Research connects to product impact, since you start with the product goal. It provides systematic progress even when problems seem overwhelming, and makes you validate each step before you invest heavily.

Integration with other tools:

Use with Research Tree (chapter 4) to map backwards dependencies.
Apply time-boxing (chapter 5) to limit exploration of each branch.
Combine with pre-Research checks (chapter 6) to validate product impact.

In the next chapter, you’ll learn about two limitations of drawing backwards and how to address them with continuous end-to-end iterations.

Chapter 8 - End-to-End Iterations

Your most important role is ensuring that Research impacts the product. One major risk: spending weeks on research questions that seem vital, only to discover they don't actually impact the product.

In chapter 7, I advocated for drawing backwards – starting from product impact and working your way back to research questions. This is indeed powerful, especially because it forces you to focus on the end result and validate it with users.

But drawing backwards alone has limitations. Two risks emerge:

Risk 1: Infeasibility in practice Your manually-created "ideal output" might be technically infeasible or impossibly expensive to generate. You won't discover this until you try to build it.

Risk 2: Lack of real-world validation Since you haven't completed an end-to-end process, you probably haven't run your solution on clients' actual data (assuming you can't access it during the manual phase). Continuing our COBOL example from the previous chapter, what works on carefully-chosen examples might fail on real codebases.

These two risks are why I advocate for continuous end-to-end iterations.

Drawing Backwards + End-to-End: A Combined Approach

These aren't competing approaches – they're complementary:

Heuristic	Purpose	What It Gives You	Strength	Limitation
Drawing Backwards (chapter 7)	Define target and path	1. Target output\ 2. Hypothesized chain of steps to get there\ 3. Order of dependencies	Ensures product focus; reveals what you need to build	Hypotheses may be wrong; doesn't validate feasibility on real data
End-to-End Iterations (this chapter)	Validate and build incrementally	1. Proof the chain works\ 2. Learning from real data\ 3. Prioritized improvements	Validates feasibility; discovers what actually works	Can lose direction without clear target and chain

The recommended flow:

Use drawing backwards to:
- Define your target output (manually create examples, validate with users).
- Identify the chain of intermediary steps needed to produce that output.
- Understand the order of dependencies.
Switch to end-to-end iterations to:
- Test whether your hypothesized chain actually works.
- Validate on real data (not just your manual examples).
- Incrementally build toward the target.
Throughout iterations, keep both the target AND the chain from step 1 as your guide.

Drawing backwards already outlines your end-to-end process (Principle 1 below). End-to-end iterations validate and build that process incrementally, learning what works and what doesn't.

The Five Principles of End-to-End Iterations

The end-to-end approach relies on five principles:

Outline the end-to-end process.
Get to end-to-end by simplifying.
Ship it as fast as you can.
Gradually replace steps.
Get frequent feedback on results.

Let me explain each principle using the COBOL business rules example from chapter 7. Since this book isn't about COBOL, I'll keep explanations short while focusing on the methodology.

Principle 1: Outline the End-to-End Process

Start by outlining the entire process from input to output. More accurately, outline the hypothesized end-to-end process – you can't know for certain until you test with users.

If you followed drawing backwards from chapter 7, you already have this. Drawing backwards naturally produces this chain: you started with the target output, asked "what do I need to produce this?", then "what do I need for that?", working your way back to the starting input. That chain IS your end-to-end process outline.

For the COBOL business rules example: We used drawing backwards to identify that our document needed business rule sections, which meant we needed to explain conditions, which meant we needed to filter business conditions, which meant we needed to find conditions, which meant we needed to parse COBOL. Working backwards gave us this chain:

Start with a COBOL program.
Parse the COBOL program into an Abstract Syntax Tree (AST).
Traverse the AST to find conditions.
Filter out business-related conditions (vs. technical conditions).
For every business rule, create a document section explaining the condition.
Sort document sections according to business rule dependencies.

This is your first draft: the hypothesized process that drawing backwards revealed.

How to outline:

Draw boxes on a whiteboard.
Use a flowchart if the process isn't linear.
Keep it visible throughout the research.
Update it as you learn.

The outline serves as your map – it shows you where you are and what you're building toward.

Principle 2: Get to End-to-End by Simplifying

Your goal: make the process work end-to-end. Start with input (a COBOL program), reach the output (a document with business rules), while passing through the intermediate stages.

This sounds like too much. Don't you need to complete the whole research to achieve this?

Definitely not. The trick is to find the easiest way to get end-to-end by taking shortcuts and making assumptions that are definitely too generous for production. You should fight your inner engineer who wants to "do it right" from the start. Your goal is to get an end-to-end process working, as this heuristic is far more valuable than perfecting one step.

This means that some of the steps can be completed manually, or with very simple implementations that you know won't work in production. Remember it is an intermediate milestone, not the final product.

For our COBOL example:

Start with a single, known COBOL program.
Skip parsing – just manually write a list of conditions for the next stage.
The filtering function returns true if business-related, false otherwise.
- First implementation: a simple mapping between input conditions (that you know of) and whether to return true or false.
- Alternative: always return true – yes, you'll get non-relevant rules, but that's a problem for later.
The document generation might also be manual for the first pass.

End result: A rough-looking document, generated by a combination of manual work and code that works solely on a single program. This is far from shippable, but it's an extremely important milestone for ensuring research impacts the product.

You might argue that it's overkill, and why waste this time on manual steps when you'll need to automate them eventually? Even if you don't, I promise from experience that many researchers and engineers feel that way.

From my own experience, I learned (the hard way) that this pays off. Getting to a working end-to-end makes sure you:

Validate that the entire flow works.
Identify bottlenecks and blockers early.
Have something concrete to show and get feedback on.
Prevent spending months on one component that turns out to be unnecessary.

Principle 3: Ship It as Fast as You Can

By the end of Principle 2, you have a working end-to-end process for a single case. You definitely can't ship it yet – it only works on one specific program, certainly not the client's program.

Next milestone: make it shippable.

Does "shippable" mean it works on any program? That's too hard and takes too long. You need shortcuts.

This is where creativity matters. For our COBOL example:

Information gathering:

If running on a client's program, get as much information as possible.
Perhaps assume it's less than 1,000 lines of code.
Perhaps you know which COBOL dialect the client uses, so you don't need to support others (fun fact: COBOL has more than 300 dialects. Well, it's fun for you, not for those who need to support them).

Algorithmic shortcuts:

Use regular expressions to find conditions instead of a full parser.
Yes, you'll miss some conditions – that's a problem for later.

UX shortcuts:

Skip the document generation. Just print rules to the console.
Run from the command line without a GUI.
Manual configuration file instead of user interface.

Your goal is clear: ship it.

It doesn't need to be perfect. It needs to be enough to learn from this iteration. If you don't ship, you can only learn from your intuition – a very bad idea.

Unlike in Principle 2, note that here you don't generate an output (partially) manually - you need a working software that can run on real data.

What "shippable" means:

Works on the client's actual data (even if imperfectly).
Produces output you can get real feedback on.
Doesn't require your manual intervention for each run.

What "shippable" doesn't mean:

Perfect accuracy.
Handles all edge cases.
Production-quality code.
Beautiful user interface.

You can't ship just anything, though. In our example, if you only have a solution that works on one specific test program you created, generating an irrelevant document for the client wastes time and teaches you nothing.

You can't learn from iteration without shipping. And you can't ship without a working end-to-end process. I know it sounds obvious, but many teams miss this in practice as they get into the rabbit hole of solving one step "the right way" before validating the entire flow.

Principle 4: Gradually Replace Steps, While Carefully Prioritizing

Now you have a working end-to-end process. You can start replacing various steps' implementations with better ones:

Replace manual steps with automated ones.
Remove shortcuts and add more robust implementations.
Improve accuracy and coverage

After shipping, you'll have many things you think you must replace immediately. But remember: the goal is making research impact the product. To do that, you need careful prioritization.

The Prioritization Framework

Prioritize changes based on three criteria:

1. Learned Necessity Did you learn that something doesn't work in the current implementation and must be fixed for the product to be viable?

Example: "Regular expressions miss nested conditions, and 60% of the client's business rules are in nested conditions. We must use a real parser."

2. Learning Potential Will changing this implementation help you learn more about product impact in the next iteration?

Example: "If we improve the filtering accuracy from 40% to 80%, we'll learn whether the document format is actually useful when it contains mostly-correct content."

3. Effort Estimation How much time will this change take?

Example: "Building a full parser: 3 weeks. Improving regex to handle nested conditions: 2 days. The latter gives us 80% of the value for 10% of the effort."

Continuing with our COBOL example, you may consider and prioritize these changes following the first iteration:

Changes after first iteration:

- Fix parser to handle nested conditions [Learned: Critical, 2 days] → DO FIRST
- Add GUI for document generation [Nice-to-have, 1 week] → DEFER
- Improve filtering accuracy [Learning: Critical, 3 days] → DO SECOND  
- Support additional COBOL dialects [No evidence needed, very long time...] → DEFER
- Better document formatting [Client mentioned it, 1 week] → DEFER (validate content first)

Iterate fast: change something, ship again, get feedback. Don't solve every issue you find, even issues clients mention. Ask: "What's the most important thing to change to learn something in the very next iteration?"

Principle 5: Get Frequent Feedback on Results

The trick is to be obsessed with getting as much feedback as you can, on each and every iteration.

On each iteration:

Get feedback on the end result
- Show the actual output to users.
- When applicable, don't just ask "does this work?" – also watch them try to use it.
- Identify what works, what doesn't, what's missing.
Understand the next questions to answer
- What did you learn about product impact?
- What assumptions were validated or invalidated?
- What new questions emerged?
Plan the next iteration accordingly
- Use the prioritization framework above.
- Focus on learning, not building.
Implement minimal changes to answer questions
- Don't fix everything.
- Make the smallest changes that will answer your next most important question.
- Keep the cycle fast (days to weeks, not months).

Example iteration cycle:

Iteration 1:
- Shipped: Regex-based condition finder, always-true filter, console output.
- Learned: Document structure works, but too many false positives (noise).
- Question: Is filtering accuracy the blocker to usefulness?
- Next: Improve filtering to 80% accuracy, ship again.

Iteration 2:
- Shipped: Same regex finder, smarter filtering (80% accurate), console output.
- Learned: With better filtering, users found the output useful!
- Question: Do we need a parser, or is regex sufficient?
- Next: Test on larger programs to see where regex breaks down.

Iteration 3:
- Shipped: Same pipeline, tested on 10 real programs.
- Learned: Regex fails on 4/10 programs (nested conditions).
- Question: Parser worth the investment now?
- Next: Build parser for nested conditions, ship again.

Notice: Each cycle is fast, focused on one question, and based on real learning.

In real life, you may want to tackle a few questions per iteration. If two things are clear, fix them before shipping again so you can actually gain valuable feedback rather than hearing the same complaints.

Also, when working with real clients, they might not be as receptive to trying things so many times – so you’ll need to consider that aspect as well. Regardless, the key remains the same: keep cycles short and focused on learning.

Integration with Other Tools

End-to-end iterations work best when combined with other research management tools:

Research Tree (chapter 4):

The outlined process becomes a branch in your tree.
Each iteration tests different approaches on branches.
Failed iterations mark branches red, successful ones mark green.

Time-boxing (chapter 5):

Time-box each iteration.
If you can't ship in the time box, you're building too much.

Drawing Backwards (chapter 7):

Drawing backwards (chapter 7) defines both:
- The target output.
- The hypothesized chain of intermediary steps to reach it.
End-to-end iterations test whether that chain actually works on real data.
The target from drawing backwards acts as your north star throughout iterations.
Each iteration validates or refines the steps that drawing backwards identified.
Use both: drawing backwards reveals what to build, while end-to-end iterations prove it works and let you test it incrementally.

Summary: End-to-End Iterations

End-to-end iterations ensure Research impacts the product by continuously validating feasibility and learning from real data.

The five principles:

Outline the process: Draw backwards already gives you this: the chain from input to output.
Simplify to get end-to-end: Use shortcuts and manual steps to make the whole chain work.
Ship fast: Real data teaches what theory can't.
Prioritize carefully: Use the three-criteria framework:
- Learned necessity (is it broken?)
- Learning potential (what will you learn?)
- Effort estimation (how long will it take?)
Get frequent feedback: Fast cycles (days to weeks) focused on learning.

Part 3 Summary

Back in chapter 2, you learned that your role as a Research leader is to:

Ensure Research connects to product impact.
Ensure Research is done effectively.

Part 2 handled ensuring Research is done effectively.

Part 3 handled ensuring Research connects to product impact – your most important responsibility. This part provided a complete methodology for ensuring product impact through three complementary stages:

The Three Stages of Product-Led Research

Stage 1: Choose research that matters (chapter 6)

Before starting any Research, answer three critical questions:

Product impact: Will success create huge value?
Time to impact: How long until we see product results?
Resources: Do we have the knowledge, capacity, and dependencies?

You learned to distinguish between problem-driven research (starting from validated customer pain points, which is strongly preferred) and opportunity-driven research (starting from new technologies).

Run focused pre-research checks to answer these questions. Only pursue Research when confident about substantial product impact.

Stage 2: Start from product value and work backwards (chapter 7)

Once you've chosen what to research, the drawing backwards heuristic ensures that you start right. Instead of diving into technical challenges, start from the end:

Manually create the desired output: What should successful Research produce? Create it by hand before solving any technical problems.
Validate with stakeholders or customers: Show them the output and confirm it solves their problem.
Work backwards through dependencies: From that validated output, identify what you need to produce it, then what you need for that, working your way back to your starting point.
Solve in reverse order: Tackle the final step first (with earlier steps mocked), validating that each step contributes to the goal before investing heavily in earlier dependencies.

The spiral game example showed why this works: working backwards from the goal reveals systematic solutions that working forward obscures.

Drawing backwards forces product connection because you literally start with product output. It integrates with the Research Tree (chapter 4): While drawing backwards identifies what questions matter, the Tree helps you explore approaches for answering them.

Stage 3: Validate and build iteratively (chapter 8)

Drawing backwards alone has two limitations:

Your manually-created "ideal output" might be technically infeasible to generate.
You haven't validated on real user data.

End-to-end Iterations address both limitations. These two tools aren't competing approaches – rather, they complement one another:

How the Three Stages Work Together

These chapters form a complete methodology for product-led Research:

Choose → Start from the end → Validate iteratively

Chapter 6 ensures you choose research that COULD have huge impact (strategic decision).
Chapter 7 ensures you start from product value (planning backward from validated output).
Chapter 8 ensures you continuously validate with real users.

Each stage prevents different undesired, but painfully common outcomes:

Chapter 6 prevents pursuing research that won't matter (even if successful).
Chapter 7 prevents building technically correct solutions that don't create product value.
Chapter 8 prevents building infeasible solutions or solutions that fail on real data.

You now have the complete answer to "How do I ensure Research impacts the product?":

Choose wisely: Only pursue research with clear, huge product impact.
Start from product value: Manually create and validate desired output before technical work.
Work backwards: Identify dependencies from output back to starting point.
Build end-to-end fast: Get entire chain working with shortcuts and manual steps.
Ship to real users: Validate on actual data, not just examples.
Iterate based on learning: Improve the chain incrementally, prioritizing what teaches you most.

This methodology keeps product impact central at every stage: choosing, planning, and executing. It prevents the most expensive failure: "successful" Research that doesn't affect the product.

Book Summary

You picked up this book because managing Research is different from managing Development – and you needed concrete tools to handle that difference.

What You Learned About Research

In chapter 1, you learned that Research isn't about difficulty or technical sophistication. It's about uncertainty of approach – that is, confronting problems where you don't know if a solution exists, where multiple approaches might work but you're not sure which, and where the path to success isn't immediately clear.

You saw how Alan Schoenfeld's problem-solving framework breaks down the Research process into four components:

Knowledge base – what you know.
Heuristics – strategies for approaching problems.
Control – monitoring and adjusting your approach.
Beliefs – your mindset toward the problem.

The good news: all four can be improved with the right management and methods.

In chapter 2, you explored the distinction between Research and Development more deeply. You learned that your role as a Research leader has two parts:

Ensure Research connects to product impact: "Successful" Research that doesn't affect the product is a failed project. This is the most important part in Product-led companies.
Ensure Research is done effectively: Even brilliant researchers benefit from structured approaches.

This two-part framework organized everything that followed.

How to Do Research Effectively

Part 2 gave you concrete methods for effective Research execution – tools that work in any research context (whether it's Product-led or not).

In chapter 3, I shared with you a personal reverse engineering classroom story. Students with sophisticated technical skills missed the obvious solution (checking the Help menu) because they lacked structured methodology. The story clearly showed that the problem isn't capability, it's often approach. This illustrated why you need the methods that follow.

Chapter 4 introduced the Research Tree method – a living visual framework for systematically exploring solution paths. You learned:

How to map questions you need to answer and approaches for answering them.
A decision framework for choosing which approach to try first: fastest feedback, lowest cost, best coverage.
How using the tree helps avoid common failure modes: jumping on the first idea, tunnel vision, inefficient learning, answering questions you don't need to, and lost context.

The Research Tree helps you implement Schoenfeld's "control" component – helping you monitor and adjust your approach systematically rather than randomly trying things. It is helpful for a researcher, but as a Research leader, it allows you to guide your team effectively.

In chapter 5, you learned how to manage exploration without killing creativity. Since Research estimation is inherently difficult, time-boxing provides structure by setting time limits for specific research directions. After the allocated time, you stop to reconsider: What did you learn? Is this still the most promising path? This tool acknowledges uncertainty while preventing endless exploration, or diving too deep into rabbit holes that don't necessarily help your Product goals.

How to Ensure Product Impact

Part 3 focused on your most important responsibility: ensuring Research creates product value.

Chapter 6 showed you how to choose what directions to research – and more importantly, what not to pursue. You learned the distinction between:

Problem-driven research – starting from customer pain points (strongly preferred, lower risk).
Opportunity-driven research – starting from new technologies (higher risk, needs validation).

Before pursuing any Research initiative, you need to answer three questions:

Product impact: Will success create huge value?
Time to impact: How long until we see product results?
Resources: Do you and your team have the knowledge, capacity, and dependencies needed?

You saw how to run focused pre-research checks to answer these questions.

Chapter 7 introduced a powerful heuristic for ensuring product connection: start from the end and work backwards. Through the spiral game example, you saw how working backwards reveals systematic solutions that working forward obscures.

In the COBOL business rules case study, you saw a practical application:

Start by manually creating the desired output (before solving any technical challenges).
Validate that output with stakeholders.
Work backwards through dependencies, solving them in reverse order.
Validate that each step contributes to the goal before major investment.

Drawing backwards forces connection to product impact because you must start with the product goal.

Chapter 8 showed you how to address two limitations of the drawing backwards heuristic through continuous end-to-end iterations. Your manually-created "ideal output" might be infeasible to generate, and you haven't validated it on real data. End-to-end iterations solve both problems.

You learned five principles to run effective end-to-end iterations:

Outline the end-to-end process: Drawing backwards already gives you this chain.
Get to end-to-end by simplifying: Use shortcuts and manual steps to make the whole chain work.
Ship it as fast as you can: Real data teaches what theory can't.
Gradually replace steps: Prioritize based on learned necessity, learning potential, and effort.
Get frequent feedback: Fast cycles focused on learning.

Drawing backwards reveals what to build and in what order, while end-to-end iterations prove it works and build it incrementally.

Your Toolkit for Research Management

You now have a complete toolkit:

From Part 2 (works for any research):

Research Tree – maps solution space, chooses approaches systematically.
Time-boxing – manages exploration with structure.

From Part 3 (specifically for product impact):

Choosing frameworks – decides what deserves Research effort.
Drawing backwards – forces product connection from the start.
End-to-end iterations – validates feasibility and learns from real users.

These methods work together. Drawing backwards identifies your goal and the chain of steps. The Research Tree maps approaches for each step. Time-boxing prevents getting deep into a rabbit hole when you should reconsider based on what you've learned, while acknowledging the inability to provide exact time estimates on Research tasks. End-to-end iterations validate and build incrementally.

My Message To You

Research is fundamentally uncertain work. Applying traditional Development management practices fails because it assumes known solution paths, predictable timelines, and steady progress.

But Research doesn't have to be mystical or random. With the right frameworks, you can manage it systematically while maintaining focus on what matters: creating product value. You learned to ensure that Research is done effectively (part 2) and that it connects to product impact (part 3). You have concrete tools, real examples, and a clear framework for both responsibilities.

Now go turn your team's uncertain Research into systematic progress toward measurable product impact. I am confident that you can lead Research teams to success, and I would be happy to hear about your experiences applying these methods.

If you liked this book, please share it with more people.

Acknowledgements

I am extremely lucky to have such wonderful people supporting me in this journey.

Abbey Rennemeyer has been a wonderful editor. Abbey had edited my posts for freeCodeCamp over the past few years, as well as my previous book Gitting Things Done, so I knew she was the perfect fit for this book as well. Her insights both on the content and the writing style have greatly improved this book.

Quincy Larson founded the amazing community at freeCodeCamp. I thank him for starting this incredible community, his ongoing support, and for his friendship.

Estefania Cassingena Navone designed the cover of this book. I am grateful for her professional work and her patience with my perfectionism and requests.

Beta readers who contributed their time and mind to read through an unfinished version of this book to improve it for all of you have helped me get it to its current shape. Specifically, I would like to thank Jason S. Shapiro and Omer Gull for their insights.

Dr. David Ginat introduced me first to Alan Schoenfeld's problem-solving research during my time at Tel Aviv University. His teaching inspired me to apply these ideas in practical contexts, including Research management.

I was privileged to work with many brilliant researchers and engineering leaders over the years, too many to name here.

To readers of my previous book, Gitting Things Done, who were kind enough to provide feedback and support – you are awesome. Receiving your emails and comments made me feel like there is a real reason to keep writing.

If You Wish to Support This Book

If you would like to support this book, you are welcome to buy E-Book version, Paperback, Hardc over , or buy me a coffee. Thank you!

Contact Me

If you liked something about this book, or felt that something was missing or needed improvement – I would love to hear from you. Please reach out at: gitting.things@gmail.com.

Thank you for learning and allowing me to be a part of your journey.

- Omer Rosenbaum

About the Author

Omer Rosenbaum is an established technologist and writer. He's the author of the Brief YouTube Channel, and the books Gitting Things Done and Computer Networks (in Hebrew). He's also a cyber training expert and founder of Checkpoint Security Academy.

How IPv4 Works – A Handbook for Developers

Omer Rosenbaum — Wed, 30 Apr 2025 15:52:50 +0000

The Internet Protocol version 4 (IPv4) is one of the core protocols of standards-based internetworking methods in the Internet and other packet-switched networks. IPv4 is still the most widely deployed Internet protocol. Google’s IPv6 Statistics show 44.29% of traffic to Google services on April 24, 2025 is over IPv6, implying 55.71% goes over IPv4.

This handbook will take you through every aspect of IPv4, from understanding IP addresses to examining packet headers and fragmentation. You'll learn:

How IP addresses work and their different formats
Network addressing schemes from fixed-length to CIDR
Special IPv4 addresses and their uses
The structure and purpose of every field in the IPv4 header
How IPv4 handles packet fragmentation across different networks

Whether you're a network engineer, software developer, or IT professional, understanding IPv4 is crucial for working with modern computer networks.

What we’ll cover:

Background
Understanding IP Addresses
Network ID and Host ID
How to Determine Network vs. Host Portions
- Fixed-Length Approach
- What are the disadvantages here? 🤔
Classful Addressing
- IP Address Assignment
- What are the disadvantages here? 🤔
CIDR: Classless Interdomain Routing
- Real-world Example
Subnet Masks
Interim Summary – IPv4 Addresses
Test Yourself
Special IPv4 Addresses
IPv4 Header
- The Header Structure
- IPv4 Header – Interim Summary
IPv4 Fragmentation
Summary – IPv4
About the Author
Additional References

Quick notes before we start

You can find more content about computer networks on my YouTube channel: Computer Networks Playlist
I am working on a book about Computer Networks! Are you interested in reading the initial versions and providing feedback? Send me an email: gitting.things@gmail.com

Background

IP stands for "Internet Protocol", so IPv4 is Internet Protocol version 4. It was described in RFC 791 by IETF, published in September 1981, and first deployed for production in 1982 on SATNET (the Atlantic Packet Satellite Network), which was an early satellite network that formed an initial segment of the Internet.

IPv4 is connectionless and operates in a best-effort delivery model. This means it doesn't guarantee delivery, correct ordering of packets, or the validity of the data. It's designed to be fast and flexible.

Understanding IP Addresses

IP addresses are hierarchical, logical addresses that power most internet connections today. Each consists of 4 bytes, or 32 bits. They're usually written in dotted decimal notation, for example:

Test yourself – Does the following address represent a valid IP address?

No. Since the dots separate different bytes, each value must be between 0 and 255. Since the number 392 is bigger than 255, it cannot be represented in a single byte.

Network ID and Host ID

IP addresses have two parts: a network identifier (or network ID) that belongs to all hosts in the network and a host identifier (or host ID) that identifies the specific host in this network.

The network identifier will be the same for all hosts in the network, and is also called a "prefix". For example, consider a network identifier of 201.22.3. Given that this is the network prefix, the following addresses:

201.22.3.15
201.22.3.91

Are part of the same network, as they share the same prefix. The first address belongs to host number 15 in this network, and the second belongs to host number 91.

This address has a different prefix, or a different network identifier, and thus belongs to a different network:

201.22.14.50

In the examples above, there's a network identifier consisting of 3 bytes, or 24 bits, and a host identifier consisting of 1 byte, or 8 bits.

How to Determine Network vs. Host Portions

A question arises: how do you know which bits are part of the network ID, and which are part of the host ID? Several approaches have evolved over time to address this challenge.

Fixed-Length Approach

Let's consider this solution: For every IP address, the first, most-significant byte would represent the network ID, and the remaining three, least-significant bytes would represent the host ID. This way it's really easy to read IP addresses. For example for this address:

20.12.1.92

You know that it describes network 20, and the host 12.1.92 inside that network. Any IP address that doesn't start with 20, such as 22.1.2.3, would reside in a different network, and any IP address that starts with 20, like 20.1.2.3, would be within the same network.

What are the disadvantages here? 🤔

With only one byte (8 bits) to represent the network ID, you only have 2^8, or 256, different networks. Of course, there are far more networks than that in the real world. Even in the early days of the internet, universities and large companies each needed their own network identifiers.

In general, using a fixed length for the network ID and a fixed length for the host ID is not flexible enough. If you decide that the two most-significant bytes will represent the network ID and the two least-significant bytes will represent the host ID, you can represent up to 2^16, or 65,536 networks, which is also not enough. Furthermore, some networks, such as those of large companies, might require more than 65,536 host IDs.

Classful Addressing

The solution lies in providing some flexibility. Consider another approach called "classful addressing". In this approach, the number of bits dedicated for the network ID changes from one address to another, and you can tell the network ID by looking at the first, most-significant byte of the address.

Any address starting with a number between 1 and 127 belongs to "Class A", meaning that its network ID consists of 1 byte, leaving 3 bytes for the host ID.
Any address starting with a number between 128 and 191 belongs to "Class B", which means that its network ID is 2 bytes long, and its host ID is also 2 bytes long.
Any address starting with a number between 192 and 223 belongs to "Class C", so it has 3 bytes of a network ID, and 1 byte of host ID.

You can see the full representation of this approach in the table below:

Class	First Byte Range	Network ID Size	Host ID Size
A	`1` - `127`	1 byte	3 bytes
B	`128` - `191`	2 bytes	2 bytes
C	`192` - `223`	3 bytes	1 byte
D	`224` - `239`	(multicast)
E	`240` - `255`	(reserved)

For example, what class does this address belong to?

(1) 130.12.204.5

Since it starts with 130, which is between 128 and 191, it belongs to "Class B". This means that its network ID is 130.12, and its host ID is 204.5. Let's mark it as "address number 1".

Do this address and the following address (2) belong to the same network?

(2) 130.90.2.40

No, since they have different network identifiers, they are not within the same network.

What class does the following address belong to?

(3) 200.1.1.9

It belongs to class C, as the value of its first byte, 200, is between 192 and 223. This means that its network identifier is 200.1.1, and any address starting with this prefix will reside within the same network. This specific address describes host 9 within this network.

To complete the picture, addresses starting with a value between 224 and 239 belong to "Class D" – that is, multicast addresses – addresses that belong to multiple devices. Addresses starting with a value between 240 and 255 were reserved for future use. Addresses starting with 0 are special addresses.

IP Address Assignment

In the early internet, IPv4 addresses were assigned to organizations by the Internet Assigned Numbers Authority (IANA). As the internet grew, this responsibility was distributed to five Regional Internet Registries (RIRs) that handle address allocation for different geographic regions. Large organizations would receive blocks of addresses based on their needs, with address classes determining the size of these blocks.

What are the disadvantages here? 🤔

While classful addressing allows for more flexibility compared to the fixed-length approach, even this approach isn't flexible enough.

Consider this scenario: A small startup company with just two founders needs a network identifier. Which class would they need?

Getting a class A or class B would be excessive, so they might get a class C – allowing 256 addresses. This is more than currently needed, but allows some expansion. What happens if the startup grows to more than 256 employees (and devices)?

At this point, they would need to get a class B address, giving no less than 65,536 addresses, when all they need is a bit over 256 addresses. This means wasting more than 60,000 addresses.

This became a real problem in the early 1990s as the internet was growing faster. The need for more IP addresses became apparent, and there was an impending exhaustion of the IPv4 address space. Cases where 60,000 addresses were wasted could no longer be tolerated.

CIDR: Classless Interdomain Routing

One of the measures to handle this shortage of addresses was to abandon classful addressing in 1993 and switch to another approach called CIDR – Classless Interdomain Routing. This approach is still used today.

CIDR allows for flexibility when choosing the network ID and the host ID. It lets network administrators create subnets of precisely the right size, rather than being limited to Classes A, B, or C.

Let's start with a simple example. In CIDR notation, we add a suffix indicating how many bits are used for the network portion:

(4) 200.8.3.1/16

This slash notation specifies how many bits describe the network ID. In example (4) above, the first 16 bits (or 2 bytes) are used for the network ID. So, in this case, 200.8 is the network identifier, and 3.1 is the host identifier. The fact that 200.8 is the network ID means that all addresses from 200.8.0.0 through 200.8.255.255 are in this network.

Consider these additional addresses:

(5) 200.2.13.5
(6) 200.8.21.6

Given this address prefix of 16 bits, or 2 bytes, which of these addresses belong to the same network as example (4) (200.8.3.1/16)?

The first address (5) (200.2.13.5) does not belong to this network, as its first 16 bits – 200.2, are different from the first 16 bits of the example address.

The second address (6) (200.8.21.6) does belong to the same network as that of the example address.

Real-world Example

In practice, an ISP might receive a large block like 104.16.0.0/12 from the RIR. This gives them control of all addresses from 104.16.0.0 to 104.31.255.255. The ISP can then allocate smaller subnets to customers, such as giving a small business a /24 subnet with 256 addresses, or a larger company a /20 subnet with 4,096 addresses.

Subnet Masks

Another way to express the network prefix is by using a subnet mask, like so:

255.255.0.0

When converted to binary, 255 in decimal equals eight 1s in binary – so all bits are on. So if you translate this mask into binary, you get:

11111111 11111111 00000000 00000000

In other words, 16 bits are on, which means a network prefix of 16 bits. Both conventions (CIDR notation and subnet masks) are used very frequently.

With CIDR, an address can reside in different networks given different network prefixes, or subnet masks. If you consider the same example address with a different prefix, say that of 8 bits – both additional addresses would belong to the same network, as they all share the first 8 bits – 200.

How would you present a network prefix of 8 bits as a subnet mask? You need the first 8 bits to be on, so that means 255 in decimal, and the remaining bits are off, resulting in this subnet mask:

255.0.0.0

What happens if you use a network prefix of 24 bits? First, how would you express that as a subnet mask? You need 24 bits to be on, so that is 3 times 8 bits to be on, resulting in:

255.255.255.0

Now, neither of the additional addresses reside within the same network as the example address, as they don't share its network ID of 200.8.3.

Note that network prefixes do not have to represent full bytes. For example, you can use a network prefix of 12 bits, or 11 bits, or 22 bits. When the prefix length isn't a multiple of 8, the subnet mask will have a value other than 0 or 255 in one of its positions.

This addresses the issue regarding the startup company. If a startup has 300 employees, they'd need to get a 23-bits network ID, leaving 9 bits for hosts within their networks. This means 2^9, or 512 addresses, which should be sufficient.

Interim Summary – IPv4 Addresses

In this section, you've learned about IPv4 addresses. IP addresses are hierarchical, logical addresses that consist of 4 bytes. IP addresses have two parts: a network identifier that belongs to all hosts in the network, and a host identifier which identifies the specific host in the network.

You've explored various options for determining the network identifier and the host identifier:

Fixed-length approach – too rigid and limited
Classful addressing approach – better but still wasteful
CIDR (Classless Interdomain Routing) – flexible and efficient

CIDR provides much more flexibility and helps overcome the significant problem of IPv4 address shortage. However, CIDR is only one part of addressing the shortage of IPv4 addresses, with other solutions including NAT (Network Address Translation) and eventually, IPv6.

The next section will explore special IPv4 addresses and then examine the header of IPv4 packets.

Test Yourself

Now practice the concepts you've learned and make sure you feel comfortable with them.

Take a moment to try answering the following questions before checking the answers.

Converting Between Prefix Notation and Subnet Masks

How would you represent a network prefix of 16 bits, written like this /16, as a subnet mask?

You need 16 bits that are on. When 8 bits are on you get 255 in decimal, so you'd use:

255.255.0.0

Given this network prefix, do these addresses belong to the same network?

Yes, they do, as they share the same most-significant 16 bits, or two bytes

Does this address belong to the same network as that of the previous addresses?

Yes, it does. Again, it shares the same two most-significant bytes.

What about this one? Does it belong to the same network as the previous addresses?

No, as the first two bytes are not 42.31 – this is a different network. So this address describes host 1.2, within the network 42.32.

Working Backwards with Subnet Masks

Let's try the other way around. You have this subnet mask:

255.255.255.0

How would you express it using a network prefix?

You have three occurrences of 255, which means three times 8 bits that are on, so overall you have 24 bits that are on. So you can also write /24. This means 3 bytes.

Given this subnet mask, do addresses (1) and (3) above belong to the same network?

They do, as they both have the same most-significant three bytes – network 42.31.93.

What about addresses (1) and (2)?

Given this network prefix, they don't belong to the same network. The first address belongs to network 42.31.93, and the second address belongs to network 42.31.1.

Non-Byte-Aligned Prefixes

Network prefixes do not have to align to 8 bits, or full bytes. Let's say you have a network prefix of 14 bits. How would you convert that to a subnet mask?

Well, the first byte is clear: you have 8 bits on, so the first byte is 255. What about the next one?

In binary, you'd want to have six additional 1s, and then 2 0s – so in binary you'd write:

11111100

Converting to decimal, this binary number represents 252. So your subnet mask is:

255.252.0.0

Another way to make this conversion: You know that eight 1s in binary represent 255 in decimal. You also know that 11 in binary is 3, so you can simply subtract 3 from 255 and get 252.

Next, try the other way around. You have the following subnet mask:

255.255.224.0

How many bits represent the network prefix?

The first two bytes are clear: you have 16 bits. Converting the third byte to binary: 224 in decimal is 11100000 in binary. This means you have an additional three 1s, so you can write the subnet mask above as a prefix of /19 bits – 16 bits for the two 255 bytes, and 3 additional bits for the 224 byte.

Determining Network Membership

Let's consider the following addresses:

Are they part of the same network? 🤔

It depends on the subnet mask.

If the network prefix is /8, then they are part of the same network, as they share the same network ID.

On the other hand, if the network prefix is /16, then they have different network IDs, and thus don't belong to the same network. But what happens with prefixes in between? Will they reside in the same network for a prefix of /9? /14?

The way to approach this question is to convert the second byte of these addresses to binary. For the first address, this byte is 24, which in binary is:

00011000

For the second address, the second byte is 23, which in binary is:

00010111

You can see that the most significant 4 bits within the second byte are identical. If you add the first 8 bits of the address, you see that the most significant 12 bits of these addresses are the same.

So, if you have a network prefix of /11, do these addresses belong to the same network?

Yes, they do – their most significant 11 bits are identical.

What about /13?

No, with this network prefix, they don't share the same network identifier, as their 13th bit is different.

This practice should help you feel comfortable with subnet masks and network prefixes. In the next section, you'll learn about special IP addresses and then examine the header of IP packets.

Special IPv4 Addresses

Now that you're comfortable with IP addresses and subnet masks, let's explore some IP addresses that have special meanings.

The "This Host" Address: 0.0.0.0

The address 0.0.0.0 means "this host" and is used in two scenarios:

First, when a machine boots up and doesn't yet have an IP address. IP addresses are logical addresses that need to be assigned to a machine. Prior to this assignment, a device has no IP address at all. If the device needs to communicate at this stage, it may use this special address, 0.0.0.0.

Second, when writing network applications that need to listen for incoming connections on all network interfaces. For example, if a machine has two interfaces – one with the IP address 1.1.1.1, and another with the address 2.2.2.2 – listening on the address 0.0.0.0 means accepting connections regardless of which network interface receives them.

"This Network" Addresses

Another class of special addresses are those starting with zeros, where the zeros mean "this network."

For example, if you have a machine with the address:

12.34.55.55

And a network prefix of 16 bits, this machine can send a packet to another device on the network using its full address, for example 12.34.66.66, or alternatively use the special zeros notation and send the packet to:

0.0.66.66

This means "send a packet to the host 66.66 on this network." Of course, the recipient must also know the relevant network prefix to correctly interpret this address.

Broadcast Addresses

The address 255.255.255.255, where all bits are set to 1, is the address of all hosts in the local network – the broadcast address. This is similar to the broadcast address in Ethernet (FF:FF:FF:FF:FF:FF). In both cases, all bits are set to 1.

Using a proper network identifier where the host identifier is all set to 1s can be used to send a broadcast packet to remote networks. For example, consider a network 12.34.0.0/16 and another network with the network ID of 12.35.0.0/16. If a machine at 12.34.55.55 wants to send a packet to all devices in the other network, it could use the destination address: 12.35.255.255.

Even though this is allowed according to the IP specification (RFC), in practice this feature is often disabled as it can create security vulnerabilities.

Loopback Addresses: 127.0.0.0/8

All addresses in the network 127.0.0.0/8 (that is, all addresses that start with 127) are loopback addresses. Packets sent to any of these addresses are not put onto the physical network but are processed locally within the operating system. This is extremely useful for development and debugging.

For example, when developing a simple chat program, you need two clients that exchange data. One approach would be to use two different physical computers, but this is tedious – you'd need to write a message on one computer, check the other computer to see if it was received, then write a message on the second computer, and go back to the first to validate receipt.

A much simpler approach is to use a loopback address. Both clients can run on the same machine and connect with one another. You can run two different client programs on the same physical computer and exchange messages between them without needing an additional machine.

For instance, you might use the address 127.0.0.1, with one client listening on port 1337 and the other on port 1338. When client A sends a packet to client B, this packet never leaves your network card but remains within the operating system. Client B receives the packet from the loopback interface as if it had been received from the physical network.

After debugging is complete, your client code doesn't need to change – the only difference is that they will communicate using real IP addresses instead of the loopback address.

Summary of Special IPv4 Addresses

To summarize the special IPv4 addresses you've learned about:

Special Address	Meaning	Usage
`0.0.0.0`	"This host"	Used during boot or to listen on all interfaces
Addresses starting with `0`	"This network"	Sending to hosts on the local network
`255.255.255.255`	Broadcast	Sending to all hosts on the local network
Network ID with all 1s in host part	Directed broadcast	Sending to all hosts on a specific network
`127.0.0.0/8`	Loopback	Testing and debugging without using the physical network

In the next section, you'll learn about the structure of the IPv4 header.

IPv4 Header

Now that you understand IP addresses, subnets, and special addresses, it's time to examine the IPv4 header structure in detail.

The Header Structure

The diagram above shows the header of IPv4 as defined in RFC 791. Let's examine each field:

Version (4 bits)

The header starts with the Version field, which consists of four bits. For an IPv4 packet, the version is 4, so this field will always carry the value of 4 (or 0100 in binary).

❓ Why does the header start with the Version field? 🤔

(Note – when I start a sentence with the ❓mark – it’s a question addressed at you, and I encourage you to try and answer it before reading on).

The reason is that the remaining fields may differ according to the version. If a network device reads an IP packet and the version field carries the value of 4, it will expect the remainder of the packet to follow the IPv4 structure. If it carries another value, such as 6, the remaining fields are different, as in IPv6.

Internet Header Length (IHL) (4 bits)

This field indicates the length of the header itself.

❓ Why do we need to specify the length? 🤔

Unlike Ethernet, where the header size is fixed, the IPv4 header length can vary because of optional fields. For an IP packet without special options, the header consists of 20 bytes, which is the most common case.

The IHL field doesn't specify the length in bytes directly but in units of 4-byte words. So to specify a length of 20 bytes, the value would be 5 (5 × 4 = 20). This encoding allows the field to use only 4 bits while specifying header lengths up to 60 bytes (when IHL = 15).

A common IPv4 packet therefore begins with the byte 0x45 in hexadecimal, meaning it's version 4 of the IP protocol, and the header is 20 bytes long.

Type of Service (TOS) (8 bits)

The idea behind this field is that not all packets are equally important. You may want to give priority to some packets over others.

For example, packets carrying real-time data (like voice or video conferencing) are more time-sensitive than packets carrying, say, email or file downloads. If a router is currently experiencing high load, it should ideally prioritize time-sensitive packets.

The Type of Service field allows senders to indicate the priority of their packets. However, on the public internet, this field is often ignored by routers because any sender can set any priority value. In most cases, this field carries the value of 0.

Total Length (16 bits)

This field specifies the total length of the IP packet, including both the header and the payload (data).

❓ Why is this needed to specify the length? 🤔

Unfortunately, the IP layer doesn’t necessarily know if some of the bytes in the packet are actually a padding of the second layer. I described this in detail in a previous post, where I showed that in Ethernet protocol, in some cases, the receiving Ethernet entity cannot tell which bytes belong to the payload and which bytes are simply padding. The IP layer needs to know precisely which bytes belong to the actual packet, hence the Total Length field.

❓What is the maximum size of an IPv4 packet? 🤔

Since this field is 16 bits long, an IPv4 packet may contain a maximum of 2^16-1 bytes, or 65,535 bytes, including the header. The minimum size is 20 bytes, consisting of just the header without options or payload.

Fragmentation Fields (32 bits)

The next four bytes are dedicated to fragmentation control. I’ll cover these fields in a separate section, as they involve a complex topic deserving special attention.

Time to Live (8 bits)

Despite its name, this field doesn't actually measure time but rather the maximum number of routing hops a packet can traverse before being discarded.

To understand its purpose, consider this scenario: If Machine A sends a packet to Machine B through a series of routers, but there's a routing loop where Router 2 sends to Router 3, which sends to Router 4, which sends back to Router 2, the packet could circulate indefinitely, consuming bandwidth and never reaching its destination.

The TTL field prevents this by setting a limit on how many hops a packet can take:

The sender sets an initial TTL value (often 64 or 128)
Each router that handles the packet decrements the TTL by 1
If a router receives a packet with TTL = 1, it decrements it to 0 and discards the packet
The router then sends an ICMP "Time Exceeded" message back to the original sender

This doesn't solve the underlying problem of routing loops, but it prevents packets from circulating forever.

In IPv6, this field is renamed "Hop Limit," which more accurately describes its function.

Protocol (8 bits)

This field describes the payload of the IPv4 packet. For example:

A value of 6 means the payload is TCP
A value of 17 means the payload is UDP

This helps the receiving system know which protocol handler should process the packet's contents. It's similar to the Type field in Ethernet, which specifies the protocol of the layer encapsulated within the Ethernet frame.

Header Checksum (16 bits)

This is a 16-bit checksum used to verify the validity of the header only (that is, excluding the payload). The sender computes this value based on the fields of the header, and the receiver also computes it to validate that the header was received correctly.

❓The checksum must be recalculated by each router. Why is that? 🤔

Because the TTL field changes at each hop. For example, if a packet starts with TTL = 7, each router will:

Verify the current checksum based on TTL = 7
Decrement TTL to 6
Calculate a new checksum based on TTL = 6
Forward the packet with the new checksum

If the checksum verification fails, the device drops the packet. This prevents packets with corrupted headers (which might have incorrect destination addresses, for instance) from being forwarded.

Source and Destination Addresses (32 bits each)

These fields contain the source and destination IPv4 addresses, respectively. Each is 4 bytes (32 bits) long, as you learned in the previous sections on IPv4 addressing.

Options (Variable Length)

Most IPv4 packets don't include options, but when present, they can provide additional functionality:

Record Route: Each router that handles the packet adds its own address to this option, creating a trace of the packet's path
Source Routing: Allows the sender to specify the route the packet should take:
- Strict Source Routing: The entire route must be followed exactly
- Loose Source Routing: Certain routers must be traversed, but the exact path between them is flexible

Padding

In some cases, the header ends with padding bytes (usually 0s).

❓Why does the IPv4 header have padding?🤔

As explained before, the IHL field specifies the header length in 4-byte units, so the total header length must be a multiple of 4 bytes. If options make the header length not divisible by 4, padding bytes (usually 0) are added to reach the next multiple of 4.

For example, if you have 3 bytes of options, you would need 1 byte of padding to make the total header length a multiple of 4 bytes.

IPv4 Header – Interim Summary

You've now learned about the structure of the IPv4 header, with the exception of the fragmentation fields which I’ll cover in the next section.

The IPv4 header efficiently packs all the necessary routing and control information into a compact structure, typically 20 bytes long (without options). This design allows for fast processing by routers while providing the flexibility needed for internet communication. It is amazing how prominent IPv4 is, even so many years after its publication.

In the next section, you'll learn about IPv4 fragmentation.

IPv4 Fragmentation

In the previous section, you learned about most of the IPv4 header structure, with the exception of 32 bits dedicated to fragmentation. This topic deserves special attention, as it reveals important aspects of how IP packets travel across different networks.

Why Fragmentation Is Needed

To understand what fragmentation is and why it's needed, consider the following network scenario:

In this diagram, you have two different networks where Machine A resides in one network and Machine B resides in another. A router forwards packets between these two networks.

These two networks have different Maximum Transmission Units (MTUs). MTU refers to the maximum size of a frame that can be transmitted in a network. For example:

Machine B is connected to an Ethernet network with an MTU of 1500 bytes
Machine A is connected to a different network with an MTU of 2000 bytes

Different MTUs stem from the different protocols and hardware that different networks have. Ethernet has an MTU of 1500 bytes. This maximum size was chosen because RAM was expensive back in the late 1970s when Ethernet was planned, and a receiver would need more RAM if a frame could be bigger. Other networks were devised at different times where RAM prices might have been lower, or just have other considerations that affect the MTU.

Now, consider this scenario: Machine A wants to send a packet to Machine B. This packet is 1800 bytes long. From A's perspective, there's no problem since its network supports packets of this size. Machine A transmits the packet.

When the router receives this packet, it faces a problem: it cannot simply forward the packet to B's network because the packet is too big for the network's MTU. The router must fragment the packet – splitting it into smaller chunks of up to 1500 bytes, which will then be reassembled by Machine B.

How Fragmentation Works in IP

Let's examine the scenario further. The router needs to take an IP packet of 1800 bytes and split it into two fragments, each consisting of up to 1500 bytes. If Machine A sends another packet of 1800 bytes to Machine B, the router will have to split that one too – resulting in four different fragments that will be reassembled into two separate packets.

When Machine B receives these fragments, it must ensure that it reassembles fragment #1 together with fragment #2 of packet A, and fragment #1 with fragment #2 of packet B – and not, for instance, fragment #1 of packet A with fragment #2 of packet B. It must also reassemble the fragments in the correct order – so structure a packet that consists of #1#2 and not #2#1.

Identification Field

First, focus on making sure Machine B reassembles fragments of the same packet (for example, fragment #1 and fragment #2 of packet A in the example above, rather than fragment #1 of packet A and fragment #2 of packet B). This is achieved using the identification field of IPv4. Fragments belonging to the same packet will have the same identification value. For example, both fragments of packet A might have identification set to 100, and both fragments of packet B might have identification of 200.

It's important to note that sharing identification values isn't sufficient for fragments to belong to the same packet. Fragments of the same packet must also share:

The same source IP address
The same destination IP address
The same protocol value (indicating whether the payload is TCP, UDP, and so on)

Fragment Offset

Since IP is a connectionless protocol, there's no guarantee that fragments will arrive at Machine B in the correct order. Fragment #2 of packet A may arrive before fragment #1. To handle this issue, each fragment carries an Offset field, which denotes the offset from the beginning of the original packet.

The Offset field consists of 13 bits, which means it can carry values from 0 to 8191 (2^13-1). This poses a potential problem, as the maximum size of an IP packet can be 65,535 bytes (since the Total Length field of the IP header consists of 16 bits).

To address this limitation, the value encoded in the Offset field is actually multiplied by 8 (2^3). This means the minimum size of a fragment is 8 bytes, with the exception of the last fragment.

❓Why do IP packets carry an offset in bytes divided by 8, instead of just a sequential fragment number?🤔

While using sequence numbers might seem simpler, it would create problems when packets need to be fragmented multiple times.

For example, if Computer A sends a packet to the first router, which fragments it into pieces of 1480 bytes and 320 bytes, and then these fragments are sent to another router that needs to fragment them again into even smaller pieces, how would you number them?

With byte offsets, the solution is straightforward – if the first fragment has an offset of 0 and the next one has an offset of 1480, then if we need to split them into maximum 800-byte fragments, we'd have:

First fragment: 800 bytes with offset 0
Second fragment: 680 bytes with offset 800
Third fragment: 320 bytes with offset 1480

More Fragments and Don't Fragment Flags

When Machine B receives a fragment, it needs to know whether this is an entire packet by itself or if it should expect additional fragments. For this purpose, each IP fragment carries a More Fragments (MF) bit that is set to 1 for every fragment that is not the last fragment of the packet. For the last fragment, it's set to 0.

In case the packet consists of a single fragment – the MF bit will be set to 0, and the offset field will also hold the value 0 (that is, 13 bits of 0s).

Another bit related to fragmentation is the Don't Fragment (DF) bit. When this flag is turned on, intermediate devices should not fragment the original packet, even if it exceeds the MTU. Instead, they should drop it and typically send an ICMP "Fragmentation Needed" message back to the source.

In our example, if Machine A sets the Don't Fragment bit to 1, the router would drop the packet, and notify Machine A about it.

Note that right after the identification field and before the DF flag, there is a reserved bit set to 0. This bit was reserved in case it is needed in the future, for a reason unknown to the original authors of IPv4.

Fragmentation Example

Consider again our example above – with Machine A residing in a network where the MTU is 2000, and Machine B residing in a network where the MTU is 1500. Machine A sends a packet which is 1800 bytes long.

❓Can you fill the values in these tables?

First Fragment:

Total Length
Identification
Don’t Fragment
More Fragments
Offset

Second Fragment:

Total Length
Identification
Don’t Fragment
More Fragments
Offset

For our example above, the values of the relevant fragmentation fields in IP would be as follows:

First Fragment:

Total Length: 1500 (including 20 bytes of IP header, so 1480 bytes of payload)
Identification: 1337 (arbitrary value)
Don't Fragment bit: 0 (off, to allow further fragmentation if needed)
More Fragments bit: 1 (on, as this is not the last fragment)
Offset: 0 (it's the first fragment)

Second Fragment:

Total Length: 340 (including 20 bytes of IP header, so 320 bytes of payload – together with the first fragment, we get to 1800 bytes of payload)
Identification: 1337 (same as first fragment, indicating they belong together)
Don't Fragment bit: 0 (off, to allow further fragmentation if needed)
More Fragments bit: 0 (off, as this is the last fragment)
Offset: 185 (1480/8 = 185, or 0xB9 in hexadecimal)

IPv4 Fragmentation – Summary

You've now learned about the final part of the IPv4 Header: fragmentation. Fragmentation is necessary to allow packets to travel across networks with different MTUs. The IPv4 header includes several fields specifically designed to support fragmentation:

Identification (16 bits): Identifies which fragments belong together
Flags (3 bits): Including the "More Fragments" and "Don't Fragment" flags
Fragment Offset (13 bits): Indicates where in the original packet this fragment belongs

With this knowledge, you now understand every bit and byte of the IPv4 header and how IP packets can traverse networks with different characteristics.

Summary – IPv4

In this comprehensive guide to IPv4, you've learned about the fundamental building blocks of Internet communications. Let's recap the key concepts we covered:

Addressing and Network Structure

IPv4 addresses are 32-bit numbers typically written in dotted decimal notation
Networks can be identified using various methods:
- Fixed-length approach (historically)
- Classful addressing (A, B, C, D, E classes)
- CIDR (modern approach allowing flexible network sizes)
Special addresses serve specific purposes:
- 0.0.0.0 for "this host"
- 127.0.0.0/8 for loopback
- 255.255.255.255 for broadcast

IPv4 Header Structure

The header contains crucial fields for packet routing and processing:
- Version and IHL for header interpretation
- Type of Service for traffic prioritization
- Total Length for packet size
- Various fields for fragmentation control
- TTL to prevent infinite routing loops
- Protocol to identify the encapsulated protocol
- Checksum for error detection
- Source and destination addresses

Fragmentation

Allows IPv4 packets to traverse networks with different MTUs
Uses three key fields:
- Identification to group fragments
- Flags to control fragmentation
- Fragment Offset to reassemble packets

Final Words

While IPv4 has limitations, particularly its address space constraints, its elegant design and robust features have allowed it to remain the backbone of the Internet for over four decades. Understanding IPv4 provides essential context for working with modern networks and helps in transitioning to newer protocols like IPv6.

About the Author

Omer Rosenbaum is Swimm’s Chief Technology Officer. He's the author of the Brief YouTube Channel. He's also a cyber training expert and founder of Checkpoint Security Academy. He's the author of Gitting Things Done (in English) and Computer Networks (in Hebrew). You can find him on Twitter.

Additional References

Computer Networks Playlist - on my Brief channel

Gitting Things Done – A Visual and Practical Guide to Git [Full Book]

Omer Rosenbaum — Mon, 08 Jan 2024 17:12:21 +0000

Introduction

Git is awesome.

Most software developers use Git on a daily basis. But how many truly understand Git? Do you feel like you know what's going on under the hood as you use Git to perform various tasks?

For example, what happens when you use git commit? What is stored between commits? Is it just a diff between the current and previous commit? If so, how is the diff encoded? Or is an entire snapshot of the repository stored each time?

Most people who use Git don't know the answers to the questions posed above. But does it really matter? Do you really have to know all of those things?

I'd argue that it does matter. As professionals, we should strive to understand the tools we use, especially if we use them all the time, like Git.

Even more acutely, I've found that understanding how Git actually works is useful in many scenarios — whether resolving merge conflicts, looking to conduct an interesting rebase, or even just when something goes slightly wrong.

So many times have I received questions about Git from experienced, highly skilled software engineers. I have seen wonderful developers react in fear when something happened in their commit history, and they just didn't know what to do. It doesn't have to be this way.

By reading this book, you will gain a new perspective of Git. You will feel confident when working with Git, and you will understand Git's underlying mechanisms, at least those that are useful to understand. You will Git it. You will be Gitting things done.

Introduction
Part 1 - Main Objects and Introducing Changes
Part 2 - Branching and Integrating Changes
Part 3 - Undoing Changes
Part 4 - Amazing and Useful Git Tools
Summary
Appendixes

Who Is This Book For?

Any software developer who wants to deepen their knowledge about Git.

If you are experienced with Git - I am sure you will be able to deepen your knowledge. Even if you are new to Git - I will start with an overview of the mechanisms of Git, and the terms used throughout this book.

This book is for you. I wrote it so you can learn more about Git, and also come to appreciate, or even love Git.

You will also notice that I use a casual style throughout the book. I believe that learning Git should be insightful and fun. Learning new things is always hard, and I felt like writing in a less casual style wouldn't really make a good service. And as I already mentioned - this book is for you.

Who Am I?

This book is about you, and your journey with Git. But I would like to tell you a bit about why I think I can contribute to your journey.

I am the CTO and one of the co-founders of Swimm.io, a knowledge management tool for code. Part of what we do is linking parts from code in Git repositories to parts of the documentation, and then tracking changes in the repository to update the documentation if needed.

At Swimm, I got to dissect parts of Git, understand its underlying mechanisms and also gain intuition about why Git is implemented the way it is.

Before founding Swimm I practiced teaching in many different environments - among them, managing the Cyber track of Israel Tech Challenge, founding Check Point Security Academy, and writing a full text book.

This book is my attempt to make the most of both worlds - my teaching experience as well as my in-depth hands-on experience with Git, and give you the best learning experience I can.

The Approach of This Book

This is definitely not the first book about Git. When sitting down to write it, I had three principles in mind.

Practical - in this book, you will learn how to accomplish things in Git. How to introduce changes, how to undo them, and how to fix things when they go wrong. You will understand how Git works not just for the sake of understanding, but with a practical mindset. I sometimes refer to this as the "practicality principle" - which guides me in deciding whether to include certain topics, and to what extent.
In depth - you will dive deep into Git's way of operating, to understand its mechanisms. You will build your understanding gradually, and always link your knowledge to real scenarios you might face in your work. In order to achieve an in-depth understanding, I almost always prefer the command line over graphical interfaces, so you can really see what commands are running.
Visual - as I strive to provide you with intuition, the chapters will be accompanied by visual aids.

Why Is This Book Publicly Available?

I think everyone should have access to high quality content about Git, and I'd like this book to get to as many people as possible.

If you would like to support this book, you are welcome to buy the Paperback version, an E-Book version, or buy me a coffee. Thank you!

Accompanying Videos

I have covered many topics from this book on my YouTube channel - Brief (https://www.youtube.com/@BriefVid). You are welcome to check them out as well.

Get Your Hands Dirty

Throughout this book, I will mostly use the second person singular - and directly write to you. I will ask you to get your hands dirty, run the commands yourself, so you actually get to feel what it's like to use do things with Git, not just read about it.

Git's Feelings

Throughout the book, I sometimes refer to Git with words such as "believes", "thinks", or "wants". As you may argue, Git is not a human, and it doesn't have feelings or beliefs. Well, that's true, but in order for us to enjoy playing around with Git, and to help you enjoy reading (and me writing) this book, I feel like referring to Git as more than just code makes it all so much more enjoyable.

My Setup

I will include screenshots. There's no need for your setup to match mine, but if you're curious about my setup, then:

I am using Ubuntu 20.04 (WSL).
For my terminal, I use Oh My Zsh
I also use plugins for Oh My Zsh, you can follow this tutorial on freeCodeCamp.
git-graph (my alias is gg)

Feedback Is Welcome

This book has been created to help you and people like you learn, understand Git, and apply that knowledge in real life.

Right from the beginning, I asked for feedback and was lucky to receive it from great people (see Acknowledgments) to make sure the book achieves these goals. If you liked something about this book, felt that something was missing, or that something needed improvement - I would love to hear from you. Please reach out at gitting.things@gmail.com.

Note

This book is provided for free on freeCodeCamp as described above and according to Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International.

If you would like to support this book, you are welcome to buy the Paperback version, an E-Book version, or buy me a coffee. Thank you!

Part 1 - Main Objects and Introducing Changes

Chapter 1 - Git Objects

It's time to start your journey into the depths of Git. In this chapter - starting with the basics - you will learn about the most important Git objects, and adopt a way of thinking about Git. Let's get to it!

Git as a System for Maintaining a File System

While there are different ways to use Git, I'll adopt here the way I've found to be the most clear and useful: Viewing Git as a system maintaining a file system, and specifically - snapshots of that file system over time.

A file system begins with a root directory (in UNIX-based systems, /), which usually contains other directories (for example, /usr or /bin). These directories contain other directories, and/or files (for example, /usr/1.txt). On a Windows machine, a root directory of a drive would be C:\, and a subdirectory could be C:\users. I will adopt the convention of UNIX-based systems throughout this book.

Blobs

In Git, the contents of files are stored in objects called blobs, short for binary large objects.

The difference between blobs and files is that files also contain meta-data. For example, a file "remembers" when it was created, so if you move that file from one directory into another directory, its creation time remains the same.

Blobs, in contrast, are just binary streams of data, like a file's contents. A blob does not register its creation date, its name, or anything other than its contents.

Every blob in Git is identified by its SHA-1 hash. SHA-1 hashes consist of 20 bytes, usually represented by 40 characters in hexadecimal form. Throughout this book I will sometimes show just the first characters of that hash. As hashes, and specifically SHA-1 hashes are so ubiquitous within Git, it is important you understand the basic characteristics of hashes.

Hashes

A hash is a deterministic, one-way mathematical function.

Deterministic means that the same input will provide the same output. That is - you take a stream of data, run a hash function on that stream, and you get a result.

For example, if you provide the SHA-1 hash function with the stream hello, you will get 0xaaf4c61ddcc5e8a2dabede0f3b482cd9aea9434d. If you run the SHA-1 hash function again, from a different machine, and provide it the same data (hello), you will get the same value.

Git uses SHA-1 as its hash function in order to identify objects. It relies on it being deterministic, such that an object will always have the same identifier.

A one-way function is a function that is hard to invert given an output. That is, it is impossible (or at least, very hard) to tell, given the result of the hash function (for example 0xaaf4c61ddcc5e8a2dabede0f3b482cd9aea9434d), what input yielded that result (in this example, hello).

Back to Git

Back to Git - Blobs, just like other Git objects, have SHA-1 hashes associated with them.

Blobs have corresponding SHA-1 values

As I said in the beginning, Git can be viewed as a system to maintain a file system. File systems consist of files and directories. A blob is the Git object representing the contents of a file.

Trees

In Git, the equivalent of a directory is a tree. A tree is basically a directory listing, referring to blobs, as well as other trees.

Trees are identified by their SHA-1 hashes as well. Referring to these objects, either blobs or other trees, happens via the SHA-1 hash of the objects.

A tree is a directory listing

Consider the drawing above. Note that the tree CAFE7 refers to the blob F92A0 as the file pic.png. In another tree, that same blob may have another name - but as long as the contents are the same, it will still be the same blob object, and still have the same SHA-1 value.

A tree may contain sub-trees, as well as blobs

The diagram above is equivalent to a file system with a root directory that has one file at /test.js, and a directory named /docs consisting of two files: /docs/pic.png, and /docs/1.txt.

Commits

Now it's time to take a snapshot of that file system — and store all the files that existed at that time, along with their contents.

In Git, a snapshot is a commit. A commit object includes a pointer to the main tree (the root directory of the file system), as well as other meta-data such as the committer (the user who authored the commit), a commit message, and the commit time.

In most cases, a commit also has one or more parent commits — the previous snapshot (or snapshots). Of course, commit objects are also identified by their SHA-1 hashes. These are the hashes you are probably used to seeing when you use commands such as git log.

A commit is a snapshot in time. It refers to the root tree. As this is the first commit, it has no parents

Every commit holds the entire snapshot, not just differences between itself and its parent commit or commits.

How can that work? Doesn't that mean that Git has to store a lot of data for every commit?

Examine what happens if you change the contents of a file. Say that you edit the file 1.txt, and add an exclamation mark — that is, you changed the content from HELLO WORLD, to HELLO WORLD!.

Well, this change means that Git creates a new blob object, with a new SHA-1 hash. This makes sense, as sha1("HELLO WORLD") is different from sha1("HELLO WORLD!").

Changing the blob results in a new SHA-1

Since you have a new hash, then the tree's listing should also change. After all, your tree no longer points to blob 73D8A, but rather blob 62E7A instead. Since you change the tree's contents, you also change its hash.

The tree that points to the changed blob needs to change as well

And now, since the hash of that tree is different, you also need to change the parent tree — as the latter no longer points to tree CAFE7, but rather to tree 24601. Consequently, the parent tree will also have a new hash.

The root tree also changes, and so does its hash

Almost ready to create a new commit object, and it seems like you are going to store a lot of data — the entire file system, once more! But is that really necessary?

Actually, some objects, specifically blob objects, haven't changed since the previous commit — the blob F92A0 remained intact, and so did the blob F00D1.

So this is the trick — as long as an object doesn't change, Git doesn't store it again. In this case, Git doesn't need to store blob F92A0 or blob F00D1 once more. Git can refer to them using only their hash values. You can then create your commit object.

Blobs that remained intact are referenced by their hash values

Since this commit is not the first commit, it also has a parent commit — commit A1337.

Considering Hashes

After introducing blobs, trees, and commits - consider the hashes of these objects. Assume I wrote the string Git is awesome!, and created a blob object from it. You did the same on your system. Would we have the same hash?

The answer is — Yes. Since the blobs consist of the same data, they'll have the same SHA-1 values.

What if I made a tree that references the blob of Git is awesome!, and gave it a specific name and metadata, and you did exactly the same on your system. Would we have the same hash?

Again, yes. Since the tree objects are the same, they would have the same hash.

What if I created a commit pointing to that tree with the commit message Hello, and you did the same on your system? Would we have the same hash?

In this case, the answer is — No. Even though our commit objects refer to the same tree, they have different commit details — time, committer, and so on.

How Are Objects Stored?

You now understand the purpose of blobs, trees, and commits. In the next chapters, you will also create these objects yourself. Despite being interesting, understanding how these objects are actually encoded and stored is not vital to your understanding, and for gitting things done.

Short Recap - Git Objects

To recap, in this section we introduced three Git objects:

Blob — contents of a file.
Tree — a directory listing (of blobs and trees).
Commit — a snapshot of the working tree.

In the next chapter, we will understand branches in Git.

Chapter 2 - Branches in Git

In the previous chapter, I suggested that we should view Git as a system for maintaining a file system.

One of the wonders of Git is that it enables multiple people to work on that file system, in parallel, (mostly) without interfering with each other's work. Most people would say that they are "working on branch X." But what does that actually mean?

A branch is just a named reference to a commit.

You can always reference a commit by its SHA-1 hash, but humans usually prefer other ways to name objects. A branch is one way to reference a commit, but it's really just that.

In most repositories, the main line of development is done in a branch called main. This is just a name, and it's created when you use git init, making it widely used. However, you could use any other name you'd like.

Typically, the branch points to the latest commit in the line of development you are currently working on.

A branch is just a named reference to a commit

To create another branch, you can use the git branch command. When you do that, Git creates another pointer. If you created a branch called test, by using git branch test, you would be creating another pointer that points to the same commit as the branch you are on:

Using git branch creates another pointer

How does Git know which branch you're currently on? It keeps another designated pointer, called HEAD. Usually, HEAD points to a branch, which in turns points to a commit. In the case described, HEAD might point to main, which in turn points to commit B2424. In some cases, HEAD can also point to a commit directly.

HEAD points to the branch you are currently on

To switch the active branch to be test, you can use the command git checkout test, or git switch test. Now you can already guess what this command actually does — it just changes HEAD to point to test.

git checkout test changes where HEAD points

You could also use git checkout -b test before creating the test branch, which is the equivalent of running git branch test to create the branch, and then git checkout test to move HEAD to point to the new branch.

At the point represented in the drawing above, what would happen if you made some changes and created a new commit using git commit? Which branch will the new commit be added to?

The answer is the test branch, as this is the active branch (since HEAD points to it). Afterwards, the test pointer will move to the newly added commit. Note that HEAD still points to test.

Every time we use git commit, the branch pointer moves to the newly created commit

If you go back to main by using git checkout main, Git will move HEAD to point to main again.

The resulting state after using git checkout main

Now, if you create another commit, which branch will it be added to?

That's right, it will be added to the main branch (and its parent would be commit B2424).

The resulting state after creating another commit on the main branch

Short Recap - Branches

A branch is a named reference to a commit.
When you use git commit, Git creates a commit object, and moves the branch to point to the newly created commit.
HEAD is a special pointer telling Git which branch is the active branch (in rare cases, it can point directly to a commit).

In the next chapters, you will learn how to introduce changes to Git. You will create a repository from scratch — without using git init, git add, or git commit. This will allow you to deepen your understanding of what is happening under the hood when you work with Git. You will also create new branches, switch branches, and create additional commits — all without using git branch or git checkout. I don't know about you, but I am excited already!

Chapter 3 - How to Record Changes in Git

So far, we've learned about four different entities in Git:

Blob — contents of a file.
Tree — a directory listing (of blobs and trees).
Commit — a snapshot of the working tree, with some meta-data such as the time or the commit message.
Branch — a named reference to a commit.

The first three are objects, whereas the fourth is one way to refer to objects (specifically, commits).

Now, it's time to understand how to introduce changes in Git.

When you work on your source code, you work from a working dir. A working dir(ectory) (also called "working tree") is any directory on your file system which has a repository associated with it. It contains the folders and files of your project, and also a directory called .git that we will talk more about later. Remember that we said that Git is a system to maintain a file system. The working directory is the root of the file system for Git.

After you make some changes, you might want to record them in your repository. A repository (in short: "repo") is a collection of commits, each of which is an archive of what the project's working tree looked like at a past date, whether on your machine or someone else's. That is, as I said before, a commit is a snapshot of the working tree.

A repository also includes things other than your code files, such as HEAD and branches.

A working dir alongside the repository

Note regarding the drawing conventions I use: I include .git within the working directory, to remind you that it is a folder within the project's folder on the filesystem. The .git folder actually contains the objects of the repository, as we will see in chapter 4.

There are other version control systems where changes are committed directly from the working dir to the repository. In Git, this is not the case. Instead, changes are first registered in something called the index, or the staging area.

Both of these terms refer to the same thing, and they are used often in Git's documentation. I will use these terms interchangeably throughout this book, as you should feel comfortable with both of them.

You can think of adding changes to the index as a way of "confirming" your changes, one by one, before creating a commit (which records all your approved changes at once).

When you checkout a branch, Git populates the index and the working dir with the contents of the files as they exist in the commit that branch is pointing to. When you use git commit, Git creates a new commit object based on the state of the index.

The three "states" - working dir, index, and repository

Using the index allows you to carefully prepare each commit. For example, you may have two files with changes in your working dir:

Working dir includes two files with changes

For example, assume these two files are 1.txt and 2.txt. It is possible to only add one of them (for instance, 1.txt) to the index, by using git add 1.txt:

The state after staging 1.txt

As a result, the state of the index matches the state of HEAD (in this case, "Commit 2"), with the exception of the file 1.txt, which matches the state of 1.txt in the working directory. Since you did not stage 2.txt, the index does not include the updated version of 2.txt. So the state of 2.txt in the index matches the state of 2.txt in "Commit 2".

Behind the scenes - once you stage a version of a file, Git creates a blob object with the file's contents. This blob object is then added to the index. As long as you only modify the file on the working directory, without staging it, the changes you make are not recorded in blob objects.

When considering the previous figure, note that I do not draw the staged version of the file as part of the "repository", as in this representation, the "repository" refers to a tree of commits and their references, and this blob has not been a part of any commit.

Now, you can use git commit to record the change to 1.txt only:

The state after using git commit

Using git commit performs two main operations:

It creates a new commit object. This commit object reflects the state of the index when you ran the git commit command.
Updates the active branch to point to the newly created commit. In this example, main now points to "Commit 3", the new commit object.

How to Create a Repo — The Conventional Way

Let's make sure that you understand how the terms we've introduced relate to the process of creating a new repository. This is a quick high-level view, before diving much deeper into this process.

Initialize a new repository using git init my_repo, and then change your directory to that of the repository using cd my_repo:

git init

By using tree -f .git you can see that running git init my_repo resulted in quite a few sub-directories inside .git. (The flag -f includes files in tree's output).

Note: if you're using Windows, run tree /f .git.

The output of tree -f .git after using git init

Create a file inside the my_repo directory:

Creating f.txt

This file is within your working directory. If you run git status, you'll see this file is untracked:

The result of git status

Files in your working directory can be in one of two states: tracked or untracked.

Tracked files are files that Git "knows" about. They either were in the last commit, or they are staged now (that is, they are in the staging area).

Untracked files are everything else — any files in your working directory that were not in your last commit, and are not in your staging area.

The new file (f.txt) is currently untracked, as you haven't added it to the staging area, and it hasn't been included in a previous commit.

f.txt is in the working directory (and untracked)

You can now add this file to the staging area (also referred to as staging this file) by using git add f.txt. You can verify that it has been staged by running git status:

Adding the new file to the staging area

So now the state of the index matches that of the working dir:

The state after adding the new file

You can now create a commit using git commit:

Committing an initial commit

If you run git status again, you'll see that the status is clean - that is, the state of HEAD (which points to your initial commit) equals the state of the index, and also the state of the working dir. By using git log you will see indeed that HEAD points to main which in turn points to your new commit:

The output of git log after introducing the first commit

Has something changed within the .git directory? Run tree -f .git to check:

A lot of things have changed within .git

Apparently, quite a lot has changed. It's time to dive deeper into the structure of .git and understand what is going on under the hood when you run git init, git add or git commit. That's exactly what the next chapter will cover.

Recap - How to Record Changes in Git

You learned about the three different "states" of the file system that Git maintains:

Working dir(ectory) (also called "working tree") - any directory on your file system which has a repository associated with it.
Index, or the Staging Area - a playground for the next commit.
Repository (in short: "repo") - a collection of commits, each of which is a snapshot of the working tree.

When you introduce changes in Git, you almost always follow this order:

You change the working directory first
Then you stage these changes (or some of them) to the index
And finally, you commit these changes - thereby updating the repository with a new commit. The state of this new commit matches the state of the index.

Ready to dive deeper?

Chapter 4 - How to Create a Repo From Scratch

So far we've covered some Git fundamentals, and now you should be ready to really Git going (I can't seem to get enough of that pun).

In order to deeply understand how Git works, you will create a repository, but this time — you will build it from scratch. As in other chapters, I encourage you to try out the commands alongside this chapter.

How to Set Up `.git`

Create a new directory, and run git status within it:

git status in a new directory

Alright, so Git seems unhappy as you don't yet have a .git folder. The natural thing to do would be to create that directory and try again:

git status after creating .git

Apparently, creating a .git directory is just not enough. You need to add some content to that directory.

A Git repository has two main components:

A collection of objects — blobs, trees, and commits.
A system of naming those objects — called references.

A repository may also contain other things, such as hooks, but at the very least — it must include objects and references.

Create a directory for the objects at .git/objects, and a directory for the references (in short: "refs") at .git/refs (on Windows systems — .git\ objects and .git\refs, respectively).

Considering the directory tree

One type of reference is branches. Internally, Git calls branches by the name heads. Create a directory for branches — .git/refs/heads.

The directory tree

This still doesn't change the result of git status:

git status after creating .git/refs/heads

How does Git know where to start when looking for a commit in the repository? As I explained earlier, it looks for HEAD, which points to the current active branch (or commit, in some cases).

So, you need to create HEAD, which is just a file residing at .git/HEAD. You can apply the following:

On UNIX:

echo "ref: refs/heads/main" > .git/HEAD

On Windows:

echo ref: refs/heads/main > .git\HEAD

So you now know how HEAD is implemented — it is simply a file, and its contents describe what it points to.

Following the command above, git status seems to change its mind:

HEAD is just a file

Notice that Git "believes" you are on a branch called main, even though you haven't created this branch. main is just a name. You can also make Git believe you are on a branch called banana if you wish:

Creating a branch named banana

Switch back to main, as you will keep working from (mostly) there throughout this chapter, just to adhere to the regular convention:

echo "ref: refs/heads/main" > .git/HEAD

Now that you have your .git directory ready, you can work your way to make a commit (again, without using git add or git commit).

Plumbing vs Porcelain Commands in Git

At this point, it would be helpful to make a distinction between two types of Git commands: plumbing and porcelain. The application of the terms oddly comes from toilets, traditionally made of porcelain, and the infrastructure of plumbing (pipes and drains).

The porcelain layer provides a user-friendly interface to the plumbing. Most people only deal with the porcelain. Yet, when things go (terribly) wrong, and someone wants to understand why, they would have to roll up their sleeves and deal with the plumbing.

Git uses this terminology as an analogy to separate the low-level commands that users don't usually need to use directly ("plumbing" commands) from the more user-friendly high level commands ("porcelain" commands).

So far, you have dealt with porcelain commands — git init, git add or git commit. It's time to go deeper, and get yourself acquainted with some plumbing commands.

How to Create Objects in Git

Start by creating an object and writing it into the objects database of Git, residing within .git/objects. To know the SHA-1 hash value of a blob, you can git hash-object (yes, a plumbing command), in the following way:

On UNIX:

echo "Git is awesome" | git hash-object --stdin

On Windows:

> echo Git is awesome | git hash-object --stdin

By using --stdin you are instructing git hash-object to take its input from the standard input. This will provide you with the relevant hash value:

Getting a blob's SHA-1

In order to actually write that blob into Git's object database, you can add the -w switch for git hash-object. Then, you check the contents of the .git folder, and see that they have changed:

Writing a blob to the objects' database

You can see that the hash of your blob is 7a9bd34a0244eaf2e0dda907a521f43d417d94f6. You can also see that a directory has been created under .git/objects, a directory named 7a, and within it, a file by the name of 9bd34a0244eaf2e0dda907a521f43d417d94f6.

What Git did here is take the first two characters of the SHA-1 hash, and use them as the name of a directory. The remaining characters are used as the filename for the file that actually contains the blob.

Why is that so? Consider a fairly big repository, one that has 400,000 objects (blobs, trees, and commits) in its database. Looking up a hash inside that list of 400,000 hashes might take a while. Thus, Git simply divides that problem by 256.

To look up the hash above, Git would first look for the directory named 7a inside the directory .git/objects, which may have up to 256 directories (00 through FF). Then, it will search within that directory, narrowing down the search as it goes.

Back to the process of generating a commit. You have just created an object. What is the type of that object? You can use another plumbing command, git cat-file -t (-t stands for "type"), to check that out:

_Using git cat-file -t <object_sha> reveals the type of the Git object_

Not surprisingly, this object is a blob. You can also use git cat-file -p (-p stands for "pretty-print") to see its contents:

git cat-file -p

This process of creating a blob object under .git/objects usually happens when you add something to the staging area — that is, when you use git add. So blobs are not created every time you save a file to the file system (the working dir), but only when you stage it.

Remember that Git creates a blob of the entire file that is staged. Even if a single character is modified or added, the file has a new blob with a new hash (as in the example in chapter 1 where you added ! at the end of a line).

Will there be any change to git status?

git status after creating a blob object

Apparently, no. Adding a blob object to Git's internal database does not change the status, as Git does not know of any tracked (or untracked) files at this stage.

You need to track this file — add it to the staging area. To do that, you can use another plumbing command, git update-index, like so:

git update-index --add --cacheinfo 100644

Note: The cacheinfo is a 16-bit file mode as stored by Git, following the layout of POSIX types and modes. This is not within the scope of this book, as it is really not important for you to Git things done.

Running the command above will result in a change to .git's contents:

The state of .git after updating the index

Can you spot the change? A new file by the name of index has been created. This is it — the famous index (or staging area), is basically a file that resides within .git/index.

So now that your blob has been added to the index, do you expect git status to look different?

git status after using git update-index

That's interesting! Two things happened here.

First, you can see that awesome.txt appears in green, in the "Changes to be committed" area. That is so because the index now includes awesome.txt, waiting to be committed.

Second, we can see that awesome.txt appears in red — because Git believes the file awesome.txt has been deleted, and the fact that the file has been deleted is not staged.

(Note: You may have noticed that I sometimes refer to Git with words such as "believes", "thinks", or "wants". As I explained in the introduction of this book - in order for us to enjoy playing around with Git, and reading (and writing) this book, I feel like referring to Git as more than just code makes it all so much more enjoyable.)

This happens as you added the blob with the contents Git is awesome to the objects' database, and updated the index that the file awesome.txt holds the contents of that blob, but you never actually created that file on disk.

You can easily solve this by taking the contents of the blob and writing them to our file system, to a file called awesome.txt:

echo "Git is awesome" > awesome.txt

As a result, it will no longer appear in red by git status:

git status after creating awesome.txt on disk

So now it's time to create a commit object from your staging area. As explained in chapter 1, a commit object has a reference to a tree, so you need to create a tree.

You can achieve this using the command git write-tree, which records the contents of the index in a tree object. Of course, you can use git cat-file -t to see that it's indeed a tree:

Creating a tree object with the contents of the index

And you can use git cat-file -p to see its contents:

git cat-file -p to see the tree's contents

Great, so you created a tree, and now you need to create a commit object that references this tree. To do that, you can use the command:

git commit-tree  -m

Committing using the tree object

You should now feel comfortable with the commands used to check the created object's type, and print its contents:

Creating a commit object

Note that this commit object doesn't have a parent, because it is the first commit. When you add another commit you will probably want to declare its parent — don't worry, you will do so later.

The last hash that we got — b6d05ee40344ef5d53502539772086da14ad2b07 – is a commit's hash. You should actually be used to using these hashes — you probably look at them all the time (when using git log, for instance). Note that this commit object points to a tree object, with its own hash, which you rarely specify explicitly.

Will something change in git status?

git status after creating a commit object

No, nothing has changed. Why is that?

Well, to know that your file has been committed, Git needs to know about the latest commit. How does Git do that? It goes to the HEAD:

Looking at the contents of HEAD

HEAD points to main, but what is main? You haven't really created it yet.

As we explained earlier in chapter 2, a branch is simply a named reference to a commit. And in this case, we would like main to refer to the commit object with the hash b6d05ee40344ef5d53502539772086da14ad2b07.

You can achieve this by creating a file at .git/refs/heads/main, with the contents of this hash, like so:

Creating main

In sum, a branch is just a file inside .git/refs/heads, containing a hash of the commit it refers to.

Now, finally, git status and git log seem to appreciate our efforts:

git status

git log

You have successfully created a commit without using porcelain commands! How cool is that?

Recap - How to Create a Repo From Scratch

In this chapter, you fearlessly deep-dived into Git. You stopped using porcelain commands and switched to plumbing commands.

By using echo and low-level commands such as git hash-object, you were able to create a blob, add it to the index, create a tree of the index, and create a commit object pointing to that tree.

You also learned that HEAD is a file, located in .git/HEAD. Branches are also files, located under .git/refs/heads. When you understand how Git operates, those abstract notions of HEAD or "branches" become very tangible.

The next chapter will deepen your understanding of how branches work under the hood.

Chapter 5 - How to Work with Branches in Git — Under the Hood

In the previous chapter you created a repository and a commit without using git init, git add or git commit. In this chapter, you we will create and switch between branches without using porcelain commands (git branch, git switch, or git checkout).

It's perfectly understandable if you are excited, I am too!

Continuing from the previous chapter - you only have one branch, named main. To create another one with the name test (as the equivalent of git branch test), you would need to create a file named test within .git/refs/heads, and the contents of that file would be the same commit's hash as the main branch points to.

Creating test branch

If you use git log, you can see that this is indeed the case — both main and test point to this commit:

git log after creating test branch

(Note: if you run this command and don't see a valid output, you may have written something other than the commit's hash into .git/refs/heads/test.)

Next, switch to our newly created branch (the equivalent of git checkout test). How would you do that? Try to answer for yourself before moving on to the next paragraph.

To change the active branch, you should change HEAD to point to your new branch:

Switching to branch test by changing HEAD

As you can see, git status confirms that HEAD now points to test, which is, therefore, the active branch.

You can now use the commands you have already used in the previous chapter to create another file and add it to the index:

Writing and staging another file

Following the commands above, you:

Create a blob with the content of Another File (using git hash-object).
Add it to the index by the name another_file.txt (using git update-index).
Create a corresponding file on disk with the contents of the blob (using git cat-file -p).
Create a tree object representing the index (using git write-tree).

It's now time to create a commit referencing this tree. This time, you should also specify the parent of this commit — which would be the previous commit. You specify the parent using the -p switch of git commit-tree:

Creating another commit object

We have just created a commit, with a tree as well as a parent, as you can see:

Observing the new commit object

(Note: the SHA-1 value of your commit object will be different than the one shown in the screenshot above, as it includes the timestamp of the commit, and also author's details - which would be different on your machine.)

Will git log show us the new commit?

git log after creating "Commit 2"

As you can see, git log doesn't show anything new. Why is that?

Remember that git log traces the branches to find relevant commits to show. It shows us now test and the commit it points to, and it also shows main which points to the same commit.

That's right — you need to change test to point to the new commit object. You can do that by changing the contents of .git/refs/heads/test:

echo 22267a945af8fde78b62ee7f705bbecfdd276b3d > .git/refs/heads/test

And now if you run git log:

git log after updating test branch

It worked!

git log goes to HEAD, which tells Git to go to the branch test, which points to commit 222..3d, which links back to its parent commit b6d..07.

Feel free to admire the beauty, I Git you. 😊

By inspecting your repository's folder, you can see that you have six different objects under the folder .git/objects - these are the two blobs you created (one for awesome.txt and one for file.txt), two commit objects ("Commit 1" and "Commit 2"), and the tree objects - each pointed to by one of the commit objects.

The tree listing after creating "Commit 2"

You also have .git/HEAD that points to the active branch or commit, and two branches - within .git/refs/heads.

Recap - How to Work with Branches in Git — Under the Hood

In this chapter you understood how branches actually work in Git.

The main things we covered:

A branch is a file under .git/refs/heads, where the content of the file is a SHA-1 value of a commit.
To create a new branch, Git simply creates a new file under .git/refs/heads with the name of the branch - for example, .git/refs/heads/my_branch for the branch my_branch.
To switch the active branch, Git modifies the contents of .git/HEAD to refer to the new active branch. .git/HEAD may also point to a commit object directly.
When committing using git commit, Git creates a commit object, and also moves the current branch (that is, the contents of the file under .git/refs/heads) to point to the newly created commit object.

Part 1 - Summary

This part introduced you to the internals of Git. We started by covering the basic objects — blobs, trees, and commits.

You learned that a blob holds the contents of a file. A tree is a directory-listing, containing blobs and/or sub-trees. A commit is a snapshot of our working directory, with some meta-data such as the time or the commit message.

You learned about branches, seeing that they are nothing but a named reference to a commit.

You learned the process of recording changes in Git, and that it involves the working directory, a directory that has a repository associated with it, the staging area (index) which holds the tree for the next commit, and the repository, which is a collection of commits and references.

We clarified how these terms relate to Git commands we know by creating a new repository and committing a file using the well-known git init, git add, and git commit.

Then you created a new repository from scratch, by using echo and low-level commands such as git hash-object. You created a blob, added it to the index, created a tree object representing the index, and even created a commit object pointing to that tree.

You were also able to create and switch between branches by modifying files directly. Kudos to those of you who tried this on your own!

All together, after following along through this part, you should feel that you've deepened your understanding of what is happening under the hood when working with Git.

The next part will explore different strategies for integrating changes when working in different branches in Git - specifically, merge and rebase.

Part 2 - Branching and Integrating Changes

Chapter 6 - Diffs and Patches

In Part 1 you learned how Git works under the hood, the different Git objects, and how to create a repo from scratch.

When teams work with Git, they introduce sequences of changes, usually in branches, and then they need to combine different change histories together. To really understand how this is achieved, you should learn how Git treats diffs and patches. You will then apply your knowledge to understand the process of merge and rebase.

Many of the interesting processes in Git like merging, rebasing, or even committing are based on diffs and patches. Developers work with diffs all the time, whether using Git directly or relying on the IDE's diff view. In this chapter, you will learn what Git diffs and patches are, their structure, and how to apply patches.

As a reminder from the chapter on Git Objects, a commit is a snapshot of the working tree at a certain point in time, in addition to some meta-data.

Yet, it is really hard to make sense of individual commits by looking at the entire working tree. Rather, it is more helpful to look at how different a commit is from its parent commit, that is, the diff between these commits.

So, what do I mean when I say "diff"? Let's start with some history.

Git Diff's History

Git's diff is based on the diff utility on UNIX systems. diff was developed in the early 1970's on the Unix operating system. The first released version shipped with the Fifth Edition of Unix in 1974.

git diff is a command that takes two inputs, and computes the difference between them. Inputs can be commits, but also files, and even files that have never been introduced to the repository.

Git diff takes two inputs, which can be commits or files

This is important - git diff computes the difference between two strings, which most of the time happen to consist of code, but not necessarily.

Time to Get Hands-On

As always, you are encouraged to run the commands yourself while reading this chapter. Unless noted otherwise, I will use the following repository:

https://github.com/Omerr/gitting_things_repo.git

You can clone it locally and have the same starting point I am using for this chapter.

Consider this short text file on my machine, called file.txt, which consists of 6 lines:

file.txt consists of six lines

Now, modify this file a bit. Remove the second line, and insert a new line as the fourth line. Add an exclamation mark (!) to the end of the last line, so you get this result:

After modifying file.txt, we get different six lines

Save this file with a new name, new_file.txt.

Now you can run git diff to compute the difference between the files like so:

git diff --no-index file.txt new_file.txt

(I will explain the --no-index switch of this command later. For now it's enough to understand it allows us to compare between two files that are not part of a Git repository.)

_The output of git diff --no-index file.txt new_file.txt_

The output of git diff shows quite a lot of things.

Focus on the part starting with This is a file. You can see that the added line (// new test) is preceded by a + sign. The deleted line is preceded by a - sign.

Interestingly, notice that Git views a modified line as a sequence of two changes - erasing a line and adding a new line instead. So the patch includes deleting the last line, and adding a new line that's equal to that line, with the addition of a !.

Addition lines are preceded by +, deletion lines by -, and modification lines are sequences of deletions and additions

Now would be a good time to discuss the terms "patch" and "diff". These two are often used interchangeably, although there is a distinction, at least historically.

A diff shows the differences between two files, or snapshots, and can be quite minimal in doing so. A patch is an extension of a diff, augmented with further information such as context lines and filenames, which allow it to be applied more widely. It is a text document that describes how to alter an existing file or codebase.

These days, the Unix diff program, and git diff, can produce patches of various kinds.

A patch is a compact representation of the differences between two files. It describes how to turn one file into another.

In other words, if you apply the "instructions" produced by git diff on file.txt - that is, remove the second line, insert // new test as the fourth line, remove the last line, and add instead a line with the same content and ! - you will get the content of new_file.txt.

Another important thing to note is that a patch is asymmetric: the patch from file.txt to new_file.txt is not the same as the patch for the other direction. Generating a patch between new_file.txt and file.txt, in this order, would mean exactly the opposite instructions than before - add the second line instead of removing it, and so on.

A patch consists of asymmetric instructions to get from one file to another

Try it out:

git diff --no-index new_file.txt file.txt

Running git diff in the reverse direction yields the reverse instructions - add a line instead of removing it, and so on

The patch format uses context, as well as line numbers, to locate differing file regions. This allows a patch to be applied to a somewhat earlier or later version of the first file than the one from which it was derived, as long as the applying program can still locate the context of the change. We will see exactly how these are used.

The Structure of a Diff

It's time to dive deeper.

Generate a diff from file.txt to new_file.txt again, and consider the output more carefully:

git diff --no-index file.txt new_file.txt

_The output of git diff --no-index file.txt new_file.txt_

The first line introduces the compared files. Git always gives one file the name a, and the other the name b. So in this case file.txt is called a, whereas new_file.txt is called b.

The first line in diff's output introduces the files being compared

Then the second line, starting with index, includes the blob SHAs of these files. So even though in our case they are not even stored within a Git repo, Git shows their corresponding SHA-1 values.

The third value in this line, 100644, is the "mode bits", indicating that this is a "regular" file: not executable and not a symbolic link.

The use of two dots (..) here between the blob SHAs is just as a separator (unlike other cases where it's used within Git).

The second line in diff's output includes the blob SHAs of the compared files, as well as the mode bits

Other header lines might indicate the old and new mode bits if they've changed, old and new filenames if the files were being renamed, and so on.

The blob SHAs (also called "blob IDs") are helpful if this patch is later applied by Git to the same project and there are conflicts while applying it. You will better understand what this means when you learn about the merges in the next chapter.

After the blob IDs, we have two lines: one starting with - signs, and the other starting with + signs. This is the traditional "unified diff" header, again showing the files being compared and the direction of the changes: - signs show lines in the A version that are missing from the B version, and + signs show lines missing in the A version but present in B.

If the patch were of this file being added or deleted in its entirety, then one of these would be /dev/null to signal that.

- signs show lines in the A version but missing from the B version, and + signs, lines missing in A version but present in B

Consider the case where you delete a file:

rm awesome.txt

And then use git diff:

git diff's output for a deleted file

The A version, representing the state of the index, is currently awesome.txt, compared to the working dir where this file does not exist, so it is /dev/null. All lines are preceded by - signs as they exist only in the A version.

For now, undo the deleting (more on undoing changes in Part 3):

git restore awesome.txt

Going back to the diff we started with:

_The output of git diff --no-index file.txt new_file.txt_

After this unified diff header, we get to the main part of the diff, consisting of "difference sections", also called "hunks" or "chunks" in Git. Note that these terms are used interchangeably, and you may stumble upon either of them in Git's documentation and tutorials, as well as Git's source code.

Every hunk begins with a single line, starting with two @ signs. These signs are followed by at most four numbers, and then a header for the chunk - which is an educated guess by Git. Usually, it will include the beginning of a function or a class, when possible.

In this example it doesn't include anything as this is a text file, so consider another example for a moment:

git diff --no-index example.py example_changed.py

When possible, Git includes a header for each hunk, for example a function or class definition

In the image above, the hunk's header includes the beginning of the function that includes the changed lines - def example_function(x).

Back to our previous example then:

Back to the previous diff

After the two @ signs, you'll find four numbers:

The first numbers are preceded by a - sign as they refer to file A. The first number represents the line number corresponding to the first line in file A that this hunk refers to. In the example above, it is 1, meaning that the line This is a file corresponds to line number 1 in version file A.

This number is followed by a comma (,), and then the number of lines this chunk consists of in file A. This number includes all context lines (the lines preceded with a space in the diff), or lines marked with a - sign, as they are part of file A, but not lines marked with a + sign, as they do not exist in file A.

In our example, this number is 6, counting the context line This is a file, the - line It has a nice poem:, then the three context lines, and lastly Are belong to you.

As you can see, the lines beginning with a space character are context lines, which means they appear as shown in both file A and file B.

Then, we have a + sign to mark the two numbers that refer to file B. First, there's the line number corresponding to the first line in file B, followed by the number of lines this chunk consists of in file B.

This number includes all context lines, as well as lines marked with the + sign, as they are part of file B, but not lines marked with a - sign.

These four numbers are followed by two additional @ signs.

After the header of the chunk, we get the actual lines - either context, -, or + lines.

Typically and by default, a hunk starts and ends with three context lines. For example, if you modify lines 4-5 in a file with ten lines:

Line 1 - context line (before the changed lines)
Line 2 - context line (before the changed lines)
Line 3 - context line (before the changed lines)
Line 4 - changed line
Line 5 - another changed line
Line 6 - context line (after the changed lines)
Line 7 - context line (after the changed lines)
Line 8 - context line (after the changed lines)
Line 9 - this line will not be part of the hunk

So by default, changing lines 4-5 results in a hunk consisting of lines 1-8, that is, three lines before and three lines after the modified lines.

If that file doesn't have nine lines, but rather six lines - then the hunk will contain only one context line after the changed lines, and not three. Similarly, if you change the second line of a file, then there would be only one line of context before the changed lines.

The patch format by git diff

How to Produce Diffs

The last example we considered shows a diff between two files. A single patch file can contain the differences for any number of files, and git diff produces diffs for all altered files in the repository in a single patch.

Often, you will see the output of git diff showing two versions of the same file and the difference between them.

To demonstrate, consider the state in another branch called diffs:

git checkout diffs

Again, I encourage you to run the commands with me - make sure you clone the repository from:

https://github.com/Omerr/gitting_things_repo.git

At the current state, the active directory is a Git repository, with a clean status:

git status

Take an existing file, my_file.py:

_An example file - my_file.py_

And change the second line from print('An example function!') to print('An example function! And it has been changed!'):

_The contents of my_file.py after modifying the second line_

Save your changes, but don't stage or commit them. Next, run git diff:

_The output of git diff for my_file.py after changing it_

The output of git diff shows the difference between my_file.py's version in the staging area, which in this case is the same as the last commit (HEAD), and the version in the working directory.

I covered the terms "working directory", "staging area", and "commit" in the Git objects chapter, so check it out in ccase you would like to refresh your memory. As a reminder, the terms "staging area" and "index" are interchangeable, and both are widely used.

At this state, the status of the working dir is different from the status of the index. The status of the index is the same as that of HEAD

To see the difference between the working dir and the staging area, use git diff, without any additional flags.

Without switches, git diff shows the difference between the staging area and the working directory

As you can see, git diff lists here both file A and file B pointing to my_file.py. file A here refers to the version of my_file.py in the staging area, whereas file B refers to its version in the working dir.

Note that if you modify my_file.py in a text editor, and don't save the file, then git diff will not be aware of the changes you've made. This is because they haven't been saved to the working dir.

We can provide a few switches to git diff to get the diff between the working dir and a specific commit, or between the staging area and the latest commit, or between two commits, and so on.

First create a new file, new_file.txt, and save it:

_A simple new file saved as new_file.txt_

Currently the file is in the working dir, and it is actually untracked in Git.

A new, untracked file

Now stage and commit this file:

git add new_file.txt
git commit -m "Commit 3"

Now, the state of HEAD is the same as the state of the staging area, as well as the working tree:

The state of HEAD is the same as the index and the working dir

Next, edit new_file.txt by adding a new line at the beginning and another new line at the end:

_Modifying new_file.txt by adding a line in the beginning and another in the end_

As a result, the state is as follows:

After saving, the state in the working dir is different than that of the index or HEAD

A nice trick would be to use git add -p, which allows you to split the changes even within a file, and consider which ones you'd like to stage.

In this case, add the first line to the index, but not the last line. To do that, you can split the hunk using s, then accept to stage the first hunk (using y), and not the second part (using n).

If you are not sure what each letter stands for, you can always use a ? and Git will tell you.

Using git add -p, you can stage only the first change

So now the state in HEAD is without either of those new lines. In the staging area you have the first line but not the last line, and in the working dir you have both new lines.

The state after staging only the first line

If you use git diff, what will happen?

git diff shows the difference between the index and the working dir

Well, as stated before, you get the diff between the staging area and the working tree.

What happens if you want to get the diff between HEAD and the staging area? For that, you can use git diff --cached:

git diff --cached shows the difference between HEAD and the index

And what if you want the difference between HEAD and the working tree? For that you can run git diff HEAD:

git diff HEAD shows the difference between HEAD and the working dir

To summarize the different switches for git diff we have seen so far, here's a diagram:

Different switches for git diff

As a reminder, at the beginning of this chapter you used git diff --no-index. With the --no-index switch, you can compare two files that are not part of the repository - or of any staging area.

Now, commit the changes you have in the staging area:

git commit -m "Commit 4"

To observe the diff between this commit and its parent commit, you can run the following command:

git diff HEAD~1 HEAD

The output of git diff HEAD~1 HEAD

By the way, you can omit the 1 above and write HEAD~, and get the same result. Using 1 is the explicit way to state you are referring to the first parent of the commit.

Note that writing the parent commit here, HEAD~1, first results in a diff showing how to get from the parent commit to the current commit. Of course, I could also generate the reverse diff by writing:

git diff HEAD HEAD~1

The output of git diff HEAD HEAD~1 generates the reverse patch

To summarize all the different switches for git diff we covered in this section, see this diagram:

The different switches for git diff

A short way to view the diff between a commit and its parent is by using git show, for example:

git show HEAD

git show HEAD

This is the same as writing:

git diff HEAD~ HEAD

We can now update our diagram:

git diff HEAD~ HEAD is used to show the difference between commits

You can go back to this diagram as a reference when needed.

As a reminder, Git commits are snapshots - of the entire working directory of the repository, at a certain point in time. Yet, it's sometimes not useful to regard a commit as a whole snapshot, but rather by the changes this specific commit introduced. In other words, by the diff between a parent commit to the next commit.

As you learned in the Git Objects chapter, Git stores the entire snapshots. The diff is dynamically generated from the snapshot data - by comparing the root trees of the commit and its parent.

Of course, Git can compare any two snapshots in time, not just adjacent commits, and also generate a diff of files not included in a repository.

How to Apply Patches

By using git diff you can see a patch Git generates, and you can then apply this patch using git apply.

Historical Note

Actually, sharing patches used to be the main way to share code in the early days of open source. But now - virtually all projects have moved to sharing Git commits directly through pull requests (called "merge requests" on some platforms).

The biggest problem with using patches is that it is hard to apply a patch when your working directory does not match the sender's previous commit. Losing the commit history makes it difficult to resolve conflicts. You will better understand this as you dive deeper into the process of git apply, especially in the next chapter where we cover merges.

A Simple Patch

What does it mean to apply a patch? It's time to try it out!

Take the output of git diff:

git diff HEAD~1 HEAD

And store it in a file:

git diff HEAD~1 HEAD > my_patch.patch

Use reset to undo the last commit:

git reset --hard HEAD~1

Don't worry about the last command - I'll explain it in detail in Part 3, where we discuss undoing changes. In short, it allows us to "reset" the state of where HEAD is pointing to, as well as the state of the index and of the working dir. In the example above, they are all set to the state of HEAD~1, or "Commit 3" in the diagram.

So after running the reset command, the contents of the file are as follows (the state from "Commit 3"):

nano new_file.txt

_new_file.txt_

And you will apply this patch that you've just saved:

nano my_patch.patch

The patch you are about to apply, as generated by git diff

This patch tells Git to find the lines:

This is a new file
With new content!

Those lines used to be line number 1 and line number 2 in new_file.txt, and add a line with the content START! right above them.

Run this command to apply the patch:

git apply my_patch.patch

And as a result, you get this version of your file, just like the commit you have created before:

nano new_file.txt

_The contents of new_file.txt after applying the patch_

Understanding the Context Lines

To understand the importance of context lines, consider a more advanced scenario. What happens if line numbers have changed since you created the patch file?

To test, start by creating another file:

nano test.text

Creating another file - test.txt

Stage and commit this file:

git add test.txt

git commit -m "Test file"

Now, change this file by adding a new line, and also erasing the line before the last one:

Changes to test.txt

Observe the difference between the original version of the file and the version including your changes:

git diff -- test.txt

The output for git diff -- test.txt

(Using -- test.txt tells Git to run the command diff, taking into consideration only test.txt, so you don't get the diff for other files.)

Store this diff into a patch file:

git diff -- test.txt > new_patch.patch

Now, reset your state to that before introducing the changes:

git reset --hard

If you were to apply new_patch.patch now, it would simply work.

Let's now consider a more interesting case. Modify test.txt again by adding a new line at the beginning:

Adding a new line at the beginning of test.txt

As a result, the line numbers are different from the original version where the patch has been created. Consider the patch you created before:

_new_patch.patch_

It assumes that the line With more text is the second line in test.txt, which is no longer the case. So...will git apply work?

git apply new_patch.patch

It worked!

By default, Git looks for 3 lines of context before and after each change introduced in the patch - as you can see, they are included in the patch file. If you take three lines before and after the added line, and three lines before and after the deleted line (actually only one line after, as no other lines exist) - you get to the patch file. If these lines all exist - then applying the patch works, even if the line numbers changed.

Reset the state again:

git reset --hard

What happens if you change one of the context lines? Try it out by changing the line With more text to With more text!:

Changing the line With more text to With more text!

And now:

git apply new_patch.patch

git apply doesn't apply the patch

Well, no. The patch does not apply. If you are not sure why, or just want to better understand the process Git is performing, you can add the --verbose flag to git apply, like so:

git apply --verbose new_patch.patch

git apply --verbose shows the process Git is taking to apply the patch

It seems that Git searched lines from the file, including the line "With more text", right before the line "It has some really nice lines". This sequence of lines no longer exists in the file. As Git cannot find this sequence, it cannot apply the patch.

As mentioned earlier, by default, Git looks for 3 lines of context before and after each change introduced in the patch. If the surrounding three lines do not exist, Git cannot apply the patch.

You can ask Git to rely on fewer lines of context, using the -C argument. For example, to ask Git to look for 1 line of the surrounding context, run the following command:

git apply -C1 new_patch.patch

The patch applies!

_git apply -C1 new_patch.patch_

Why is that? Consider the patch again:

_new_patch.patch_

When applying the patch with the -C1 option, Git is looking for the lines:

Like this one
And that one

in order to add the line !!!This is the new line!!! between these two lines. These lines exist (and, importantly, they appear one right after the other). As a result, Git can successfully add the line between them, even though the line numbers changed.

Similarly, Git would look for the lines:

How wonderful
So we are writing an example
Git is awesoome!

As Git can find these lines, Git can erase the middle one.

If we changed one of these lines, say, changed "How wonderful" to "How very wondeful", then Git would not be able to find the string above, and thus the patch would not apply.

Recap - Git Diff and Patch

In this chapter, you learned what a diff is, and the difference between a diff and a patch. You learned how to generate various patches using different switches for git diff. You also learned what the output of git diff looks like, and how it is constructed. Ultimately, you learned how patches are applied, and specifically the importance of context.

Understanding diffs is a major milestone for understanding many other processes within Git - for example, merging or rebasing, that we will explore in the next chapters.

Chapter 7 - Understanding Git Merge

By reading this chapter, you are going to really understand git merge, one of the most common operations you'll perform in your Git repositories.

What is a Merge in Git?

Merging is the process of combining the recent changes from several branches into a single new commit. This commit points back to these branches.

In a way, merging is the complement of branching in version control: a branch allows you to work simultaneously with others on a particular set of files, whereas a merge allows you to later combine separate work on branches that diverged from a common ancestor commit.

OK, let's take this bit by bit.

Remember that in Git, a branch is just a name pointing to a single commit. When we think about commits as being "on" a specific branch, they are actually reachable through the parent chain from the commit that the branch is pointing to.

That is, if you consider this commit graph:

_Commit graph with feature_1_

You see the branch feature_1, which points to a commit with the SHA-1 value of ba0d2. As in previous chapters, I only write the first 5 digits of the SHA-1 value for brevity.

Notice that commit 54a9d is also "on" this branch, as it is the parent commit of ba0d2. So if you start from the pointer of feature_1, you get to ba0d2, which then points to 54a9d. You can go on the chain of parents, and all these reachable commits are considered to be "on" feature_1.

When you merge with Git, you merge commits. Almost always, we merge two commits by referring to them with the branch names that point to them. Thus we say we "merge branches" - though under the hood, we actually merge commits.

Time to Get Hands-on

For this chapter, I will use the following repository:

https://github.com/Omerr/gitting_things_merge.git

As in previous chapters, I encourage you to clone it locally and have the same starting point I am using for this chapter.

OK, so let's say I have this simple repository here, with a branch called main, and a few commits with the commit messages of "Commit 1", "Commit 2", and "Commit 3":

A simple repository with three commits

Next, create a feature branch by typing git branch new_feature:

Creating a new branch with git branch

And switch HEAD to point to this new branch, by using git checkout new_feature (or git switch new_feature). You can look at the outcome by using git log:

_The output of git log after using git checkout new_feature_

As a reminder, you could also write git checkout -b new_feature, which would both create a new branch and change HEAD to point to this new branch.

If you need a reminder about branches and how they're implemented under the hood, please check out chapter 2. Yes, check out. Pun intended 😇

Now, on the new_feature branch, implement a new feature. In this example, I will edit an existing file that looks like this before the edit:

code.py before editing it

And I will now edit it to include a new function:

_Implementing new_feature_

And luckily, this is not a programming book, so this function is legit 😇

Next, stage and commit this change:

git add code.py

git commit -m "Commit 4"

Looking at the history, you have the branch new_feature, now pointing to "Commit 4", which points to its parent, "Commit 3". The branch main is also pointing to "Commit 3".

The history after committing "Commit 4"

Time to merge the new feature! That is, merge these two branches, main and new_feature. Or, in Git's lingo, merge new_feature into main. This means merging "Commit 4" and "Commit 3". This is pretty trivial, as after all, "Commit 3" is an ancestor of "Commit 4".

Check out the main branch (with git checkout main), and perform the merge by using git merge new_feature:

_Merging new_feature into main_

Since new_feature never really diverged from main, Git could just perform a fast-forward merge. So what happened here? Consider the history:

The result of a fast-forward merge

Even though you used git merge, there was no actual merging here. Actually, Git did something very simple - it reset the main branch to point to the same commit as the branch new_feature.

In case you don't want that to happen, but rather you want Git to really perform a merge, you could either change Git's configuration, or run the merge command with the --no-ff flag.

First, undo the last commit:

git reset --hard HEAD~1

Reminder: if this way of using reset is not clear to you, don't worry - we will cover it in detail in Part 3. It is not crucial for this introduction of merge, though. For now, it's important to understand that it basically undoes the merge operation.

Just to clarify, now if you checked out new_feature again:

git checkout new_feature

The history would look just like before the merge:

The history after using git reset --hard HEAD~1

Next, perform the merge with the --no-fast-forward flag (--no-ff for short):

git checkout main
git merge new_feature --no-ff

Now, if we look at the history using git lol:

History after merging with the --no-ff flag

(Reminder: git lol is an alias I added to Git to visibly see the history in a graphical manner. You can find it, along with the other components of my setup, at the My Setup part of the Introduction chapter.)

Considering this history, you can see Git created a new commit, a merge commit.

If you consider this commit a bit closer:

git log -n1

The merge commit has two parents

You will see that this commit actually has two parents - "Commit 4", which was the commit that new_feature pointed to when you ran git merge, and "Commit 3", which was the commit that main pointed to.

A merge commit has two parents: the two commits it merged.

The merge commit shows us the concept of merge quite well. Git takes two commits, usually referenced by two different branches, and merges them together.

After the merge, as you started the process from main, you are still on main, and the history from new_feature has been merged into this branch. Since you started with main, then "Commit 3", which main pointed to, is the first parent of the merge commit, whereas "Commit 4", which you merged into main, is the second parent of the merge commit.

Notice that you started on main when it pointed to "Commit 3", and Git went quite a long way for you. It changed the working tree, the index, and also HEAD and created a new commit object. At least when you use git merge without the --no-commit flag and when it's not a fast-forward merge, Git does all of that.

This was a super simple case, where the branches you merged didn't diverge at all. We will soon consider more interesting cases.

By the way, you can use git merge to merge more than two commits - actually, any number of commits. This is rarely done, and to adhere to the practicality principle of this book, I won't delve into it.

Another way to think of git merge is by joining two or more development histories together. That is, when you merge, you incorporate changes from the named commits, since the time their histories diverged from the current branch, into the current branch. I used the term "branch" here, but I am stressing this again - we are actually merging commits.

Time For a More Advanced Case

Time to consider a more advanced case, which is probably the most common case where we use git merge explicitly - where you need to merge branches that did diverge from one another.

Assume we have two people working on this repo now, John and Paul.

John created a branch:

git checkout -b john_branch

_A new branch, john_branch_

And John has written a new song in a new file, lucy_in_the_sky_with_diamonds.md. Well, I believe John Lennon didn't really write in Markdown format, or use Git for that matter, but let's pretend he did for this explanation.

git add lucy_in_the_sky_with_diamonds.md
git commit -m "Commit 5"

While John was working on this song, Paul was also writing, on another branch. Paul had started from main:

git checkout main

And created his own branch:

git checkout -b paul_branch

And Paul wrote his song into a file called penny_lane.md. Paul staged and committed this file:

git add penny_lane.md
git commit -m "Commit 6"

So now our history looks like this - where we have two different branches, branching out from main, with different histories:

The history after John and Paul committed

John is happy with his branch (that is, his song), so he decides to merge it into the main branch:

git checkout main
git merge john_branch

Actually, this is a fast-forward merge, as we have learned before. You can validate that by looking at the history (using git lol, for example):

_Merging john_branch into main results in a fast-forward merge_

At this point, Paul also wants to merge his branch into main, but now a fast-forward merge is no longer relevant - there are two different histories here: the history of main's and that of paul_branch's. It's not that paul_branch only adds commits on top of main branch or vice versa.

Now things get interesting. 😎😎

First, let Git do the hard work for you. After that, we will understand what's actually happening under the hood.

git merge paul_branch

Consider the history now:

_When you merge paul_branch, you get a new merge commit_

What you have is a new commit, with two parents - "Commit 5" and "Commit 6".

In the working dir, you can see that both John's song as well as Paul's song are there (if you use ls, you will see both files in the working dir).

Nice, Git really did merge the changes for you. But how does that happen?

Undo this last commit:

git reset --hard HEAD~

How to Perform a Three-way Merge in Git

It's time to understand what's really happening under the hood. 😎

What Git has done here is it called a 3-way merge. In outlining the process of a 3-way merge, I will use the term "branch" for simplicity, but you should remember you could also merge two (or more) commits that are not referenced by a branch.

The 3-way merge process includes these stages:

First, Git locates the common ancestor of the two branches. That is, the common commit from which the merging branches most recently diverged. Technically, this is actually the first commit that is reachable from both branches. This commit is then called the merge base.

Second, Git calculates two diffs - one diff from the merge base to the first branch, and another diff from the merge base to the second branch. Git generates patches based on those diffs.

Third, Git applies both patches to the merge base using a 3-way merge algorithm. The result is the state of the new merge commit.

The three steps of the 3-way merge algorithm: (1) locate the common ancestor (2) calculate diffs from the merge base to the first branch, and from the merge base to the second branch (3) apply both patches together

So, back to our example.

In the first step, Git looks from both branches - main and paul_branch - and traverses the history to find the first commit that is reachable from both. In this case, this would be… which commit?

Correct, the merge commit (the one with "Commit 3" and "Commit 4" as its parents).

If you are not sure, you can always ask Git directly:

git merge-base main paul_branch

The merge base is the merge commit with "Commit 3" and "Commit 4" as its parents. Note: the previous commit merge is blurred as it is not reachable via the current history following the reset command

By the way, this is the most common and simple case, where we have a single obvious choice for the merge base. In more complicated cases, there may be multiple possibilities for a merge base, but this is not within our focus.

In the second step, Git calculates the diffs. So it first calculates the diff between the merge commit and "Commit 5":

git diff 4f90a62 4683aef

(The SHA-1 values will be different on your machine.)

The diff between the merge commit and "Commit 5"

If you don't feel comfortable with the output of git diff, you can read the previous chapter where I described it in detail.

You can store that diff to a file:

git diff 4f90a62 4683aef > john_branch_diff.patch

Next, Git calculates the diff between the merge commit and "Commit 6":

git diff 4f90a62 c5e4951

The diff between the merge commit and "Commit 6"

Write this one to a file as well:

git diff 4f90a62 c5e4951 > paul_branch_diff.patch

Now Git applies those patches on the merge base.

First, try that out directly - just apply the patches (I will walk you through it in a moment). This is not what Git really does under the hood, but it will help you gain a better understanding of why Git needs to do something different.

Checkout the merge base first, that is, the merge commit:

git checkout 4f90a62

And apply John's patch first (as a reminder, this is the patch shown in the image with the caption "The diff between the merge commit and "Commit 5""):

git apply --index john_branch_diff.patch

Notice that for now there is no merge commit. git apply updates the working dir as well as the index, as we used the --index switch.

You can observe the status using git status:

Applying John's patch on the merge commit

So now John's new song is incorporated into the index. Apply the other patch:

git apply --index paul_branch_diff.patch

As a result, the index contains changes from both branches.

Now it's time to commit your merge. Since the porcelain command git commit always generates a commit with a single parent, you would need the underlying plumbing command - git commit-tree.

If you need a reminder about porcelain vs plumbing commands, check out chapter 4 where I explained these terms, and created an entire repo from scratch.

Remember that every Git commit object points to a single tree. So you need to record the contents of the index in a tree:

git write-tree

Now you get the SHA-1 value of the created tree, and you can create a commit object using git commit-tree:

git commit-tree  -p  -p  -m "Merge commit!"

Creating a merge commit

Great, so you have created a commit object!

Recall that git merge also changes HEAD to point to the new merge commit object. So you can simply do the same:

git reset --hard db315a

If you look at the history now:

The history after creating a merge commit and resetting HEAD

(Note: in this state, HEAD is "detached" - that is, it directly points to a commit object rather than a named reference. gg does not show HEAD when it is "detached", so don't be confused if you can't see HEAD in the output of gg.)

This is almost what we wanted. Remember that when you ran git merge, the result was HEAD pointing to main which pointed to the newly created commit (as shown in the image with the caption "When you merge paul_branch, you get a new merge commit". What should you do then?

Well, what you want is to modify main, so you can just point it to the new commit:

git checkout main
git reset --hard db315a

And now you have the same result as when you ran git merge: main points to the new commit, which has "Commit 5" and "Commit 6" as its parents. You can use git lol to verify that.

So this is exactly the same result as the merge done by Git, with the exception of the timestamp and thus the SHA-1 value, of course.

Overall, you got to merge both the contents of the two commits - that is, the state of the files, and also the history of those commits - by creating a merge commit that points to both histories.

In this simple case, you could actually just apply the patches using git apply, and everything works quite well.

Quick Recap of a Three-way Merge

So to quickly recap, on a three-way merge, Git:

First, locates the merge base - the common ancestor of the two branches. That is, the first commit that is reachable from both branches.
Second, Git calculates two diffs - one diff from the merge base to the first branch, and another diff from the merge base to the second branch.
Third, Git applies both patches to the merge base, using a 3-way merge algorithm. I haven't explained the 3-way merge yet, but I will elaborate on that later. The result is the state of the new merge commit.

You can also understand why it's called a "3-way merge": Git merges three different states - that of the first branch, that of the second branch, and their common ancestor. In our previous example, main, paul_branch, and the merge commit (with "Commit 3" and "Commit 4" as parents), respectively.

This is unlike, say, the fast-forward examples we saw before. The fast-forward examples are actually a case of a two-way merge, as Git only compares two states - for example, where main pointed to, and where john_branch pointed to.

Moving on

Still, this was a simple case of a 3-way merge. John and Paul created different songs, so each of them touched a different file. It was pretty straightforward to execute the merge.

What about more interesting cases?

Let's assume that now John and Paul are co-authoring a new song.

So, John checked out main branch and started writing the song:

git checkout main

John's new song

He staged and committed it ("Commit 7"):

git add a_day_in_the_life.md
git commit -m "Commit 7"

John's new song is committed

Now, Paul branches:

git checkout -b paul_branch_2

And edits the song, adding another verse:

Paul added a new verse

Of course, the original song does not include the title "Paul's Verse", but I added it here for clarity.

Paul stages and commits the changes:

git add a_day_in_the_life.md
git commit -m "Commit 8"

The history after introducing "Commit 8"

John also branches out from main and adds an additional two lines at the end:

git checkout main
git checkout -b john_branch_2

John added the two last lines

John stages and commits his changes too ("Commit 9"):

git add a_day_in_the_life.md
git commit -m "Commit 9"

This is the resulting history:

The history after John's last commit

So, both Paul and John modified the same file on different branches. Will Git be successful in merging them?

Say now we don't go through main, but John will try to merge Paul's new branch into his branch:

git merge paul_branch_2

Wait! Don't run this command! Why would you let Git do all the hard work? You are trying to understand the process here.

So, first, Git needs to find the merge base. Can you see which commit that would be?

Correct, it would be the last commit on the main branch, where the two diverged - that is, "Commit 7".

You can verify that by using:

git merge-base john_branch_2 paul_branch_2

"Commit 7" is the merge base

Checkout the merge base so you can later apply the patches you will create:

git checkout main

Great, now Git should compute the diffs and generate the patches. You can observe the diffs directly:

git diff main paul_branch_2

_The output of git diff main paul_branch_2_

Will applying this patch succeed? Well, no problem, Git has all the context lines in place.

Switch to the merge-base (which is "Commit 7", also referenced by main), and ask Git to apply this patch:

git checkout main
git diff main paul_branch_2 > paul_branch_2.patch
git apply --index paul_branch_2.patch

And this worked, no problem at all.

Now, compute the diff between John's new branch and the merge base. Notice that you haven't committed the applied changes, so john_branch_2 still points at the same commit as before, "Commit 9":

git diff main john_branch_2

_The output of git diff main john_branch_2_

Will applying this diff work?

Well, indeed, yes. Notice that even though the line numbers have changed on the current version of the file, thanks to the context lines Git is able to locate where it needs to add these lines…

Git can rely on the context lines

Save this patch and apply it then:

git diff main john_branch_2 > john_branch_2.patch
git apply --index john_branch_2.patch

Observe the result file:

The result after applying Paul's patch

Cool, exactly what we wanted.

You can now create the tree and relevant commit:

git write-tree

Don't forget to specify both parents:

git commit-tree  -p paul_branch_2 -p john_branch_2 -m "Merging new changes"

See how I used the branch names here? After all, they are just pointers to the commits we want.

Cool, look at the log from the new commit:

_git lol <SHA_OF_THE_MERGE_COMMIT> after creating the merge commit_

The history after creating the merge commit

Exactly what we wanted.

You can also let Git perform the job for you. You can checkout john_branch_2, which you haven't moved - so it still points to the same commit as it did before the merge. So all you need to do is run:

git checkout john_branch_2
git merge paul_branch_2

Observe the resulting history:

git lol after letting Git perform the merge

A visualization of the history after letting Git perform the merge

Just as before, you have a merge commit pointing to "Commit 8" and "Commit 9" as its parents. "Commit 9" is the first parent since you merged into it.

But this was still quite simple… John and Paul worked on the same file, but on very different parts. You could also directly apply Paul's changes to John's branch. If you go back to John's branch before the merge:

git reset --hard HEAD~

And now apply Paul's changes:

git apply --index paul_branch_2.patch

You will get the same result.

But what happens when the two branches include changes on the same files, in the same locations?

More Advanced Git Merge Cases

What would happen if John and Paul were to coordinate a new song, and work on it together?

In this case, John creates the first version of this song in the main branch:

git checkout main
nano everyone.md

The contents of everyone.md prior to the first commit

By the way, this text is indeed taken from the version that John Lennon recorded for a demo in 1968. But this isn't a book about the Beatles. If you're curious about the process the Beatles underwent while writing this song, you can follow the links in the end of this chapter.

git add everyone.md
git commit -m "Commit 10"

Introducing "Commit 10"

Now John and Paul split. Paul creates a new verse in the beginning:

git checkout -b paul_branch_3
nano everyone.md

Paul added a new verse in the beginning

Also, while talking to John, they decided to change the word "feet" to "foot", so Paul adds this change as well.

And Paul adds and commits his changes to the repo:

git add everyone.md
git commit -m "Commit 11"

The history after introducing "Commit 11"

You can observe Paul's changes, by comparing this branch's state to the state of branch main:

git diff main

The output of git diff main from Paul's branch

Store this diff in a patch file:

git diff main > paul_3.patch

Now back to main…

git checkout main

John decides to make another change, in his own new branch:

git checkout -b john_branch_3

And he replaces the line "Everyone had the boot in" with the line "Everyone had a wet dream". In addition, John changed the word "feet" to "foot", following his talk with Paul.

Observe the diff:

git diff main

The output of git diff main from John's branch

Store this output as well:

git diff main > john_3.patch

Now, stage and commit:

git add everyone.md
git commit -m "Commit 12"

This should be your current history:

The history after introducing "Commit 12"

Note that I deleted john_branch_2 and paul_branch_2 for simplicity. Of course, you can erase them from Git by using git branch -D . As a result, these branch names will not appear in the output of git log or other similar commands.

This also applies to commits that are no longer reachable from any named reference, such as "Commit 8" or "Commit 9". Since they are not reachable from any named reference via the parents' chain, they will not be included in the output of commands such as git log.

Back to our story - Paul told John he had added a new verse, so John would like to merge Paul's changes.

Can John simply apply Paul's patch?

Consider the patch again:

git diff main paul_branch_3

_The output of git diff main paul_branch_3_

As you can see, this diff relies on the line "Everyone had the boot in", but this line no longer exists on John's branch. As a result, you could expect applying the patch to fail. Go on, give it a try:

git apply paul_3.patch

Applying the patch failed

Indeed, you can see that it failed.

But should it really fail?

As explained earlier, git merge uses a 3-way merge algorithm, and this can come in handy here. What would be the first step of this algorithm?

Well, first, Git would find the merge base - that is, the common ancestor of Paul's branch and John's branch. Consider the history:

The history after introducing "Commit 12"

So the common ancestor of "Commit 11" and "Commit 12" is "Commit 10". You can verify this by running the command:

git merge-base john_branch_3 paul_branch_3

Now we can take the patches we generated from the diffs on both branches, and apply them to main. Would that work?

First, try to apply John's patch, and then Paul's patch.

Consider the diff:

git diff main john_branch_3

_The output of git diff main john_branch_3_

We can store it in a file:

git diff main john_branch_3 > john_3.patch

And apply this patch on main:

git checkout main
git apply john_3.patch

Let's consider the result:

nano everyone.md

The contents of everyone.md after applying John's patch

The line changed as expected. Nice 😎

Now, can Git apply Paul's patch? To remind you, this is the patch:

The contents of Paul's patch

Well, Git cannot apply this patch, because this patch assumes that the line "Everyone had the boot in" exists. Trying to apply it is liable to fail:

git apply -v paul_3.branch

Applying Paul's patch failed

What you tried to do now, applying Paul's patch on the main branch after applying John's patch, is the same as being on john_branch_3, and attempting to apply the patch. That is, running:

git apply paul_3.patch

What would happen if we tried the other way around?

First, clean up the state:

git reset --hard

And start from Paul's branch:

git checkout paul_branch_3

Can we apply John's patch? As a reminder, this is the status of everyone.md on this branch:

_The contents of everyone.md on paul_branch_3_

And this is John's patch:

The contents of John's patch

Would applying John's patch work?

Try to answer yourself before reading on.

You can try:

git apply john_3.patch

Git fails to apply John's patch

Well, no! Again, if you are not sure what happened, you can always ask git apply to be a bit more verbose:

git apply -v john_3.patch

You can get more information by using the -v flag

Git is looking for "Everyone put the feet down", but Paul has already changed this line so it now consists of the word "foot" instead of "feet". As a result, applying this patch fails.

Notice that changing the number of context lines here (that is, using git apply with the -C flag, as discussed in the previous chapter) is irrelevant - Git is unable to locate the actual line that the patch is trying to erase.

But actually, Git can make this work, if you just add a flag to apply, telling it to perform a 3-way merge under the hood:

git apply -3 john_3.patch

Applying with -3 flag succeeds

And consider the result:

The contents of everyone.md after the merge

Exactly what we wanted! You have Paul's verse, and both of John's changes!

So, how was Git able to accomplish that?

Well, as I mentioned, Git really did a 3-way merge, and with this example, it will be a good time to dive into what this actually means.

How Git's 3-way Merge Algorithm Works

Get back to the state before applying this patch:

git reset --hard

You have now three versions: the merge base, which is "Commit 10", Paul's branch, and John's branch. In general terms, we can say these are the merge base, commit A and commit B. Notice that the merge base is by definition an ancestor of both commit A and commit B.

To perform the merge, Git looks at the diff between the three different versions of the file in question on these three revisions. In your case, it's the file everyone.md, and the revisions are "Commit 10", Paul's branch - that is, "Commit 11", and John's branch, that is, "Commit 12".

Git makes the merging decision based on the status of each line in each of these versions.

The three versions considered for the 3-way merge

In case not all three versions match, that is a conflict. Git can resolve many of these conflicts automatically, as we will now see.

Let's consider specific lines.

The first lines here exist only on Paul's branch:

Lines that appear on Paul's branch only

This means that the state of John's branch is equal to the state of the merge base. So the 3-way merge goes with Paul's version.

In general, if the state of the merge base is the same as A, the algorithm goes with B. The reason is that since the merge base is the ancestor of both A and B, Git assumes that this line hasn't changed in A, and it has changed in B, which is the most recent version for that line, and should thus be taken into account.

If the state of the merge base is the same as A, and this state is different from B, the algorithm goes with B

Next, you can see lines where all three versions agree - they exist on the merge base, A and B, with equal data.

Lines where all three versions agree

In this case the algorithm has a trivial choice - just take that version.

In case all three versions agree, the algorithm goes with that single version

In a previous example, we saw that if the merge base and A agree, and B's version is different, the algorithm picks B. This works in the other direction too - for example, here you have a line that exists on John's branch, different than that on the merge base and Paul's branch.

A line where Paul's version matches the merge base's version, and John has a different version

Hence, John's version is chosen.

If the state of the merge base is the same as B, and this state is different from A, the algorithm goes with A

Now consider another case, where both A and B agree on a line, but the value they agree upon is different from the merge base: both John and Paul agreed to change the line "Everyone put their feet down" to "Everyone put their foot down":

A line where Paul's version matches John's version, yet the merge base has a different version

In this case, the algorithm picks the version on both A and B.

In case A and B agree on a version which is different from the merge base's version, the algorithm picks the version on both A and B

Notice this is not a democratic vote. In the previous case, the algorithm picked the minority version, as it resembled the newest version of this line. In this case, it happens to pick the majority - but only because A and B are the revisions that agree on the new version.

The same would happen if we used git merge:

git merge john_branch_3

Without specifying any flags, git merge will default to using a 3-way merge.

By default, git merge uses a 3-way merge algorithm

The status of everyone.md after running git merge john_branch would be the same as the result you achieved by applying the patches with git apply -3.

If you consider the history:

Git's history after performing the merge

You will see that the merge commit indeed has two parents: the first is "Commit 11", that is, where paul_branch_3 pointed to before the merge. The second is "Commit 12", where john_branch_3 pointed to, and still points to now.

What will happen if you now merge from main? That is, switch to the main branch, which is pointing to "Commit 10":

git checkout main

And then merge Paul's branch?

git merge paul_branch_3

Indeed, we get a fast-forward merge - as before running this command, main was an ancestor of paul_branch_3.

A fast-forward merge

So, this is a 3-way merge. In general, if all versions agree on a line, then this line is used. If A and the merge base match, and B has another version, B is taken. In the opposite case, where the merge base and B match, the A version is selected. If A and B match, this version is taken, whether the merge base agrees or not.

This description leaves one open question though: What happens in cases where all three versions disagree?

Well, that's a conflict that Git does not resolve automatically. In these cases, Git calls for a human's help.

How to Resolve Merge Conflicts

By following so far, you should understand the basics of the command git merge, and how Git can automatically resolve some conflicts. You also understand what cases are automatically resolved.

Next, let's consider a more advanced case.

Say Paul and John keep working on this song.

Paul creates a new branch:

git checkout -b paul_branch_4

And he decides to add some "Yeah"s to the song, so he changes this verse as follows:

Paul's additions

So Paul stages and commits these changes:

git add everyone.md
git commit -m "Commit 13"

Paul also creates another song, let_it_be.md and adds it to the repo:

git add let_it_be.md
git commit -m "Commit 14"

This is the history:

The history after Paul introduced "Commit 14"

Going back to main:

git checkout main

John also branches out:

git checkout -b john_branch_4

And John also works on the song "Everyone had a hard year", later to be called "I've got a feeling" (again, this is not a book about the Beatles, so I won't elaborate on it here. See the additional links if you are curious).

John decides to change all occurrences of "Everyone" to "Everybody":

John changes all occurrences of "Everyone" to "Everybody"

He stages and commits this song to the repo:

git add everyone.md
git commit -m "Commit 15"

Nice. Now John also creates another song, across_the_universe.md. He adds it to the repo as well:

git add across_the_universe.md
git commit -m "Commit 16"

Observe the history again:

The history after John introduced "Commit 16"

You can see that the history diverges from main, to two different branches - paul_branch_4, and john_branch_4.

At this point, John would like to merge the changes introduced by Paul.

What is going to happen here?

Remember the changes introduced by Paul:

git diff main paul_branch_4

_The output of git diff main paul_branch_4_

What do you think? Will merge work?

Try it out:

git merge paul_branch_4

A merge conflict

We have a conflict!

Git cannot merge these branches on its own. You can get an overview of the merge state, using git status:

The output of git status right after the merge operation

The changes that Git had no problem resolving are staged for commit. And there is a separate section for "unmerged paths" - these are files with conflicts that Git could not resolve on its own.

It's time to understand why and when these conflicts happen, how to resolve them, and also how Git handles them under the hood.

Alright then! I hope you are at least as excited as I am. 😇

Let's recall what we know about 3-way merges:

First, Git will look for the merge base - the common ancestor of john_branch_4 and paul_branch_4. Which commit would that be?

It would be the tip of the main branch, the commit in which we merged john_branch_3 into paul_branch_3.

Again, if you are not sure, you can verify that by running:

git merge-base john_branch_4 paul_branch_4

And at the current state, git status knows which files are staged and which aren't.

Consider the process for each file, which is the same as the 3-way merge algorithm we considered per line, but on a file's level:

across_the_universe.md exists on John's branch, but doesn't exist on the merge base or on Paul's branch. So Git chooses to include this file. Since you are already on John's branch and this file is included in the tip of this branch, it is not mentioned by git status.

let_it_be.md exists on Paul's branch, but doesn't exist on the merge base or John's branch. So git merge "chooses" to include it.

What about everyone.md? Well, here we have three different states of this file: its state on the merge base, its state on John's branch, and its state on Paul's branch. While performing a merge, Git stores all of these versions on the index.

Let's observe that by looking directly at the index with the command git ls-files:

git ls-files -s --abbrev

The output of git ls-files -s --abbrev after the merge operation

You can see that everyone.md has three different entries. Git assigns each version a number that represents the "stage" of the file, and this is a distinct property of an index entry, alongside the file's name and the mode bits.

When there is no merge conflict regarding a file, its "stage" is 0. This is indeed the state for across_the_universe.md, and for let_it_be.md.

On a conflict's state, we have:

Stage 1 - which is the merge base.
Stage 2 - which is "your" version. That is, the version of the file on the branch you are merging into. In our example, this would be john_branch_4.
Stage 3 - which is "their" version, also called the MERGE_HEAD. That is, the version on the branch you are merging (into the current branch). In our example, that is paul_branch_4.

To observe the file's contents in a specific stage, you can use a command I introduced in a previous post, git cat-file, and provide the blob's SHA:

git cat-file -p

Using git cat-file to present the content of the file on John's branch, right from its state in the index

And indeed, this is the content we expected - from John's branch, where the lines start with "Everybody" rather than "Everyone".

A nice trick that allows you to see the content quickly without providing the blob's SHA-1 value, is by using git show, like so:

git show ::everyone.md

For example, to get the content of the same version as with git cat-file -p , you can write git show :2:everyone.md.

Git records the three states of the three commits into the index in this way at the start of the merge. It then follows the three-way merge algorithm to quickly resolve the simple cases:

In case all three stages match, then the selection is trivial.

If one side made a change while the other did nothing - that is, stage 1 matches stage 2- then we choose stage 3, or vice versa. That's exactly what happened with let_it_be.md and across_the_universe.md.

In case of a deletion on the incoming branch, for example, and given there were no changes on the current branch, then we would see that stage 1 matches stage 2, but there is no stage 3. In this case, git merge removes the file for the merged version.

What's really cool here is that for matching, Git doesn't need the actual files. Rather, it can rely on the SHA-1 values of the corresponding blobs. This way, Git can easily detect the state a file is in.

Git performs the same 3-way merge algorithm on a files level

For everyone.md you have this special case - where stage 1, stage 2 and stage 3 are all different from one another. That is, they have different blob SHAs. It's time to go deeper and understand the merge conflict. 😊

One way to do that would be to simply use git diff. In a previous chapter, we examined git diff in detail, and saw that it shows the differences between various combinations of the working tree, index or commits.

But git diff also has a special mode for helping with merge conflicts:

git diff

The output of git diff during a merge conflict

This output may be confusing at first, but once you get used to it, it's pretty clear. Let's start by understanding it, and then see how you can resolve conflicts with other, more visual tools.

The conflicted section is separated by the "equal" marks (====), and marked with the corresponding branches. In this context, "ours" is the current branch. In this example, that would be john_branch_4, the branch that HEAD was pointing to when we initiated the git merge command. "Theirs" is the MERGE_HEAD, the branch that we are merging in - in this case, paul_branch_4.

So git diff without any special flags shows changes between the working tree and the index - which in this case are the conflicts yet to be resolved. The output doesn't include staged changes, which is very convenient for resolving the conflict.

Time to resolve this manually. Fun!

So, why is this a conflict?

For Git, Paul and John made different changes to the same line, for a few lines. John changed it to one thing, and Paul changed it to another thing. Git cannot decide which one is correct.

This is not the case for the last lines, like the line that used to be "Everyone had a hard year" on the merge base. Paul hasn't changed this line, or the lines surrounding it, so its version on paul_branch_4, or "theirs" in our case, agrees with the merge_base. Yet John's version, "ours", is different. Thus git merge can easily decide to take this version.

But what about the conflicted lines?

In this case, I know what I want, and that is actually a combination of these lines. I want the lines to start with "Everybody", following John's change, but also to include Paul's "yeah"s. So go ahead and create the desired version by editing everyone.md:

nano everyone.md

Editing the file manually to achieve the desired state

To compare the result file to what you had in the branch prior to the merge, you can run:

git diff --ours

Similarly, if you wish to see how the result of the merge differs from the branch you merged into our branch, you can run:

git diff --theirs

You can even see how the result is different from both sides using:

git diff --base

Now you can stage the fixed version:

git add everyone.md

After staging, if you look at git status, you will see no conflicts:

After staging the fixed version everyone.md, there are no conflicts

You can now simply use git commit, and Git will present you with a commit message containing details about the merge. You can modify it if you like, or leave it as is. Regardless of the commit message, Git will create a "merge commit" - that is, a commit with more than one parent.

To validate that, consider the history:

The history after completing the merge operation

john_branch_4 now points to the new merge commit. The incoming branch, "theirs", in this case, paul_branch_4, stays where it was.

How to Use VS Code to Resolve Conflicts

You will now see how to resolve the same conflict using a graphical tool. For this example, I use VS Code, which is a free and popular code editor. There are many other tools, but the process is similar, so I will just show VS Code as an example.

First, get back to the state before the merge:

git reset --hard HEAD~

And try to merge again:

git merge paul_branch_4

You should be back at the same status:

Back at the conflicting status

Let's see how this appears on VS Code:

Conflict resolution with VS Code

VS Code marks the different versions with "Current Change" - which is the "ours" version, the current HEAD, and "Incoming Change" for the branch we are merging into the active branch. You can accept one of the changes (or both) by clicking on one of the options.

If you clicked on Resolve in Merge editor, you'll get a more visual view of the state. VS Code shows the status of each line:

VS Code's Merge Editor

If you look closely, you will see that VS Code shows changes within words - for example, showing that "Everyone" was changed to "Everybody", marking the changed parts.

You can accept either version, or you can accept a combination. In this case, if you click on "Accept Combination", you get this result:

VS Code's Merge Editor after clicking on "Accept Combination"

VS Code did a really good job! The same three way merge algorithm was implemented here and used on the word level rather than the line level. So VS Code was able to actually resolve this conflict in a rather impressive way. Of course, you can modify VS Code's suggestion, but it provided a very good start.

One More Powerful Tool

Well, this was the first time in this book that I've used a tool with a graphical user interface. Indeed, graphical interfaces can be convenient to understand what's going on when you are resolving merge conflicts.

However, like in many other cases, when we need to really understand what's going on, the command line becomes handy. So, let's get back to the command line and learn a tool that can come in handy in more complicated cases.

Again, go back to the state before the merge:

git reset --hard HEAD~

And merge:

git merge paul_branch_4

And say, you are not exactly sure what happened. Why is there a conflict? One very useful command would be:

git log -p --merge

As a reminder, git log shows the history of commits that are reachable from HEAD. Adding -p tells git log to show the commits along with the diffs they introduced. The --merge switch makes the command show all commits containing changes relevant to any unmerged files, on either branch, together with their diffs.

This can help you identify the changes in history that led to the conflicts. So in this example, you'd see:

The output of git log -p --merge

The first commit we see is "Commit 15", as in this commit John modified everyone.md, a file that still has conflicts. Next, Git shows "Commit 13", where Paul changed everyone.md:

The output of git log -p --merge - continued

Notice that git log --merge did not mention previous commits that changed everyone.md before "Commit 13", as they didn't affect the current conflict.

This way, git log tells you all you need to know to understand the process that got you into the current conflicting state. Cool! 😎

Using the command line, you can also ask Git to take only one side of the changes - either "ours" or "theirs", even for a specific file.

You can also instruct Git to take some parts of the diffs of one file and another from another file. I will provide links that describe how to do that in the additional resources of this chapter in the appendix.

For the most part, you can accomplish that pretty easily, either manually or from the UI of your favorite IDE.

For now, it's time for a recap.

Recap - Understanding Git Merge

In this chapter, you got an extensive overview of merging with Git. You learned that merging is the process of combining the recent changes from several branches into a single new commit. The new commit has two parents - those commits which had been the tips of the branches that were merged.

We considered a simple, fast-forward merge, which is possible when one branch diverged from the base branch, and then just added commits on top of the base branch.

We then considered three-way merges, and explained the three-stage process:

First, Git locates the merge base. As a reminder, this is the first commit that is reachable from both branches.
Second, Git calculates two diffs - one diff from the merge base to the first branch, and another diff from the merge base to the second branch. Git generates patches based on those diffs.
Third and last, Git applies both patches to the merge base using a 3-way merge algorithm. The result is the state of the new merge commit.

We dove deeper into the process of a 3-way merge, whether at a file level or a hunk level. We considered when Git is able to rely on a 3-way merge to automatically resolve conflicts, and when it just can't.

You saw the output of git diff when we are in a conflicting state, and how to resolve conflicts either manually or with VS Code.

There is much more to be said about merges - different merge strategies, recursive merges, and so on. Yet, I believe this chapter covered everything needed so you have a robust understanding of what merge is, and what happens under the hood in the vast majority of cases.

Chapter 8 - Understanding Git Rebase

One of the most powerful tools a developer can have in their toolbox is git rebase. Yet it is notorious for being complex and misunderstood.

The truth is, if you understand what it actually does, git rebase is a very elegant, and straightforward tool to achieve so many different things in Git.

In the previous chapters in this part, you learned what Git diffs are, what a merge is, and how Git resolves merge conflicts. In this chapter, you will understand what Git rebase is, why it's different from merge, and how to rebase with confidence.

Short Recap - What is Git Merge?

Under the hood, git rebase and git merge are very, very different things. Then why do people compare them all the time?

The reason is their usage. When working with Git, we usually work in different branches and introduce changes to those branches.

In the previous chapter, we considered the example where John and Paul (of the Beatles) were co-authoring a new song. They started from the main branch, and then each diverged, modified the lyrics, and committed their changes.

Then, the two wanted to integrate their changes, which is something that happens very frequently when working with Git.

_A diverging history - paul_branch and john_branch diverged from main_

There are two main ways to integrate changes introduced in different branches in Git, or in other words, different commits and commit histories. These are merge and rebase.

In the previous chapter, we got to know git merge pretty well. We saw that when performing a merge, we create a merge commit - where the contents of this commit are a combination of the two branches, and it also has two parents, one in each branch.

So, say you are on the branch john_branch (assuming the history depicted in the drawing above), and you run git merge paul_branch. You will get to this state - where on john_branch, there is a new commit with two parents. The first one will be the commit on the john_branch branch where HEAD was pointing to a state before performing the merge - in this case, "Commit 6". The second will be the commit pointed to by paul_branch, "Commit 9".

_The result of running git merge paul_branch: a new Merge Commit with two parents_

Look again at the history graph: you created a diverged history. You can actually see where it branched and where it merged again.

So when using git merge, you do not rewrite history - but rather, you add a commit to the existing history. And specifically, a commit that creates a diverged history.

How is `git rebase` Different than `git merge`?

When using git rebase, something different happens.

Let's start with the big picture: if you are on paul_branch, and use git rebase john_branch, Git goes to the common ancestor of John's branch and Paul's branch. Then it takes the patches introduced in the commits on Paul's branch, and applies those changes to John's branch.

So here, you use rebase to take the changes that were committed on one branch - Paul's branch - and replay them on a different branch, john_branch.

_The result of running git rebase john_branch: the commits on paul_branch were "replayed" on top of john_branch_

Wait, what does that mean?

We will now take this bit by bit to make sure you fully understand what's happening under the hood 😎

`cherry-pick` as a Basis for Rebase

It is useful to think of rebase as performing git cherry-pick - a command that takes a commit, computes the patch this commit introduces by computing the difference between the parent's commit and the commit itself, and then cherry-pick "replays" this difference.

Let's do this manually.

If we look at the difference introduced by "Commit 5" by performing git diff main :

Running git diff to observe the patch introduced by "Commit 5"

As always, you are encouraged to run the commands yourself while reading this chapter. Unless noted otherwise, I will use the following repository:

https://github.com/Omerr/rebase_playground.git

I recommend you clone it locally and have the same starting point I am using for this chapter.

You can see that in this commit, John started working on a song called "Lucy in the Sky with Diamonds":

The output of git diff - the patch introduced by "Commit 5"

As a reminder, you can also use the command git show to get the same output:

git show

Now, if you cherry-pick this commit, you will introduce this change specifically, on the active branch. Switch to main first:

git checkout main (or git switch main)

And create another branch:

git checkout -b my_branch (or git switch -c my_branch)

_Creating my_branch that branches from main_

Next, cherry-pick "Commit 5":

git cherry-pick

Using cherry-pick to apply the changes introduced in "Commit 5" onto main

Consider the log (output of git lol):

The output of git lol

It seems like you copy-pasted "Commit 5". Remember that even though it has the same commit message, and introduces the same changes, and even points to the same tree object as the original "Commit 5" in this case - it is still a different commit object, as it was created with a different timestamp.

Looking at the changes, using git show HEAD:

The output of git show HEAD

They are the same as "Commit 5"'s.

And of course, if you look at the file (say, by using nano lucy_in_the_sky_with_diamonds.md), it will be in the same state as it has been after the original "Commit 5".

Cool! 😎

You can now remove the new branch so it doesn't appear on your history every time:

git checkout main
git branch -D my_branch

Beyond `cherry-pick` - How to Use `git rebase`

You can view git rebase as a way to perform multiple cherry-picks one after the other - that is, to "replay" multiple commits. This is not the only thing you can do with rebase, but it's a good starting point for our explanation.

It's time to play with git rebase!

Before, you merged paul_branch into john_branch. What would happen if you rebased paul_branch on top of john_branch? You would get a very different history.

In essence, it would seem as if we took the changes introduced in the commits on paul_branch, and replayed them on john_branch. The result would be a linear history.

To understand the process, I will provide the high level view, and then dive deeper into each step. The process of rebasing one branch on top of another branch is as follows:

Find the common ancestor.
Identify the commits to be "replayed".
For every commit X, compute diff(parent(X), X), and store it as a patch(X).
Move HEAD to the new base.
Apply the generated patches in order on the target branch. Each time, create a new commit object with the new state.

The process of making new commits with the same change sets as existing ones is also called "replaying" those commits, a term we have already used.

Time to Get Hands-On with Rebase

Before running the following command command, make sure you have john_branch locally, so run:

git checkout john_branch

Start from Paul's branch:

git checkout paul_branch

This is the history:

Commit history before performing git rebase

And now, to the exciting part:

git rebase john_branch

And observe the history:

The history after rebasing

With git merge you added to the history, while with git rebase you rewrite history. You create new commit objects. In addition, the result is a linear history graph - rather than a diverging graph.

The history after rebasing

In essence, you "copied" the commits that were on paul_branch and that were introduced after "Commit 4", and "pasted" them on top of john_branch.

The command is called "rebase", because it changes the base commit of the branch it's run from. That is, in your case, before running git rebase, the base of paul_branch was "Commit 4" - as this is where the branch was "born" (from main). With rebase, you asked Git to give it another base - that is, pretend as if it had been born from "Commit 6".

To do that, Git took what used to be "Commit 7", and "replayed" the changes introduced in this commit onto "Commit 6". Then it created a new commit object. This object differs from the original "Commit 7" in three aspects:

It has a different timestamp.
It has a different parent commit - "Commit 6", rather than "Commit 4".
The tree object it is pointing to is different - as the changes were introduced to the tree pointed to by "Commit 6", and not the tree pointed to by "Commit 4".

Notice the last commit here, "Commit 9'". The snapshot it represents (that is, the tree that it points to) is exactly the same tree you would get by merging the two branches. The state of the files in your Git repository would be the same as if you used git merge. It's only the history that is different, and the commit objects of course.

Now, you can simply use:

git checkout main
git merge paul_branch

Hm.... What would happen if you ran this last command? Consider the commit history again, after checking out main:

The history after rebasing and checking out main

What would it mean to merge main and paul_branch?

Indeed, Git can simply perform a fast-forward merge, as the history is completely linear (if you need a reminder about fast-forward merges, check out the previous chapter). As a result, main and paul_branch now point to the same commit:

The result of a fast-forward merge

Advanced Rebasing in Git

Now that you understand the basics of rebase, it is time to consider more advanced cases, where additional switches and arguments to the rebase command will come in handy.

In the previous example, when you only used rebase (without additional switches), Git replayed all the commits from the common ancestor to the tip of the current branch.

But rebase is a super-power. It's an almighty command capable of…well, rewriting history. And it can come in handy if you want to modify history to make it your own.

Undo the last merge by making main point to "Commit 4" again:

git reset --hard

"Undoing" the last merge operation

And undo the rebasing by using:

git checkout paul_branch
git reset --hard

"Undoing" the rebase operation

Notice that you got to exactly the same history you used to have:

Visualizing the history after "undoing" the rebase operation

To be clear, "Commit 9" doesn't just disappear when it's not reachable from the current HEAD. Rather, it's still stored in the object database. And as you used git reset now to change HEAD to point to this commit, you were able to retrieve it, and also its parent commits since they are also stored in the database. Pretty cool, huh? 😎

You will learn more about git reset in the next part, where we discuss undoing changes in Git.

View the changes that Paul introduced:

git show HEAD

git show HEAD shows the patch introduced by "Commit 9"

Keep going backwards in the commit graph:

git show HEAD~

git show HEAD~ (same as git show HEAD~1) shows the patch introduced by "Commit 8"

And one commit further:

git show HEAD~2

git show HEAD~2 shows the patch introduced by "Commit 7"

Perhaps Paul doesn't want this kind of history. Rather, he wants it to seem as if he introduced the changes in "Commit 7" and "Commit 8" as a single commit.

For that, you can use an interactive rebase. To do that, we add the -i (or --interactive) switch to the rebase command:

git rebase -i

Or, since main is pointing to "Commit 4", we can run:

git rebase -i main

By running this command, you tell Git to use a new base, "Commit 4". So you are asking Git to go back to all commits that were introduced after "Commit 4" and that are reachable from the current HEAD, and replay those commits.

For every commit that is replayed, Git asks us what we'd like to do with it:

git rebase -i main prompts you to select what to do with each commit

In this context it's useful to think of a commit as a patch. That is, "Commit 7", as in "the patch that "Commit 7" introduced on top of its parent".

One option is to use pick. This is the default behavior, which tells Git to replay the changes introduced in this commit. In this case, if you just leave it as is - and pick all commits - you will get the same history, and Git won't even create new commit objects.

Another option is squash. A squashed commit will have its contents "folded" into the contents of the commit preceding it. So in our case, Paul would like to squash "Commit 8" into "Commit 7":

Squashing "Commit 8" into "Commit 7"

As you can see, git rebase -i provides additional options, but we won't go into all of them in this chapter. If you allow the rebase to run, you will get prompted to select a commit message for the newly created commit (that is, the one that introduced the changes of both "Commit 7" and "Commit 8"):

Providing the commit message: Commits 7+8

And look at the history:

The history after the interactive rebase

Exactly as we wanted! On paul_branch, we have "Commit 9" (of course, it's a different object than the original "Commit 9"). This object points to "Commits 7+8", which is a single commit introducing the changes of both the original "Commit 7" and the original "Commit 8". This commit's parent is "Commit 4", where main is pointing to.

Oh wow, isn't that cool? 😎

git rebase grants you unlimited control over the shape of any branch. You can use it to reorder commits, or to remove incorrect changes, or modify a change in retrospect. Alternatively, you could perhaps move the base of your branch onto another commit, any commit that you wish.

How to Use the `--onto` Switch of `git rebase`

Let's consider one more example. Get to main again:

git checkout main

And delete the pointers to paul_branch and john_branch so you don't see them in the commit graph anymore:

git branch -D paul_branch
git branch -D john_branch

Next, branch from main to a new branch:

git checkout -b new_branch

_Creating new_branch that diverges from main_

This is the clean history you should have:

_A clean history with new_branch that diverges from main_

Now, change the file code.py (for example, add a new function) and commit your changes:

nano code.py

_Adding the function new_branch to code.py_

git add code.py
git commit -m "Commit 10"

Get back to main:

git checkout main

And introduce another change - adding a docstring at the beginning of the file:

Added a docstring at the beginning of the file

Time to stage and commit these changes:

git add code.py
git commit -m "Commit 11"

And yet another change, perhaps add @Author to the docstring:

Added @Author to the docstring

Commit this change as well:

git add code.py
git commit -m "Commit 12"

Oh wait, now I realize that I wanted you to make the changes introduced in "Commit 11" as a part of the new_branch. Ugh. What can you do?

Consider the history:

The history after introducing "Commit 12"

Instead of having "Commit 11" reside only on the main branch, I want it to be on both the main branch as well as new_branch. Visually, I would want to move it down the graph here:

Visually, I want you to "push down" "Commit 10"

Can you see where I am going? 😇

Well, rebase allows you to basically replay the changes introduced in new_branch, those introduced in "Commit 10", as if they had been originally conducted on "Commit 11", rather than "Commit 4".

To do that, you can use other arguments of git rebase. Specifically, you can use git rebase --onto, which optionally takes three parameters:

git rebase --onto

That is, you take all commits between old_parent and until, and you "cut" and "paste" them onto new_parent.

In this case, you'd tell Git that you want to take all the history introduced between the common ancestor of main and new_branch, which is "Commit 4", and have the new base for that history be "Commit 11". To do that, use:

git rebase --onto  main new_branch

The history before and after the rebase, "Commit 10" has been "pushed"

And look at our beautiful history! 😍

The history before and after the rebase, "Commit 10" has been "pushed"

Let's consider another case.

Say I started working on a new feature, and by mistake I started working from feature_branch_1, rather than from main.

So to emulate this, create feature_branch_1:

git checkout main
git checkout -b feature_branch_1

And erase new_branch so you don't see it in the graph anymore:

git branch -D new_branch

Create a simple Python file called 1.py:

A new file, 1.py, with print('Hello world!')

Stage and commit this file:

git add 1.py
git commit -m  "Commit 13"

Now branch out from feature_branch_1 (this is the mistake you will later fix):

git checkout -b feature_branch_2

And create another file, 2.py:

Creating 2.py

Stage and commit this file as well:

git add 2.py
git commit -m  "Commit 14"

And introduce some more code to 2.py:

Modifying 2.py

Stage and commit these changes too:

git add 2.py
git commit -m  "Commit 15"

So far you should have this history:

The history after introducing "Commit 15"

Get back to feature_branch_1 and edit 1.py:

git checkout feature_branch_1

Modifying 1.py

Now stage and commit:

git add 1.py
git commit -m  "Commit 16"

Your history should look like this:

The history after introducing "Commit 16"

Say now you realize that you've made a mistake. You actually wanted feature_branch_2 to be born from the main branch, rather than from feature_branch_1.

How can you achieve that?

Try to think about it given the history graph and what you've learned about the --onto flag for the rebase command.

Well, you want to "replace" the parent of your first commit on feature_branch_2, which is "Commit 14", so that it's on top of main branch - in this case, "Commit 12" - rather than the beginning of feature_branch_1 - in this case, "Commit 13". So again, you will be creating a new base, this time for the first commit on feature_branch_2.

You want to move around "Commit 14" and "Commit 15"

How would you do that?

First, switch to feature_branch_2:

git checkout feature_branch_2

And now you can use:

git rebase --onto main

This tells Git to take the history with "Commit 13" as a base, and change that base to be "Commit 12" (pointed to by main) instead.

As a result, you have feature_branch_2 based on main rather than feature_branch_1:

The commit history after performing rebase

The syntax of the command is:

git rebase --onto

How to Rebase on a Single Branch

You can also use git rebase while looking at the history of a single branch.

Let's see if you can help me here.

Say I worked from feature_branch_2, and specifically edited the file code.py. I started by changing all strings to be wrapped by double quotes rather than single quotes:

Changing ' into " in code.py

Then, I staged and committed:

git add code.py
git commit -m "Commit 17"

I then decided to add a new function at the beginning of the file:

_Adding the function another_feature_

Again, I staged and committed:

git add code.py
git commit -m "Commit 18"

And now I realized that I actually forgot to change the single quotes to double quotes wrapping __main__ (as you might have noticed), so I did that too:

Changing '__main__' into "__main__"

Of course, I staged and committed this change:

git add code.py
git commit -m "Commit 19"

Now, consider the history:

The commit history after introducing "Commit 19"

It isn't really nice, is it? I mean, I have two commits that are related to one another, "Commit 17" and "Commit 19" (turning 's into "s), but they are split by the unrelated "Commit 18" (where I added a new function). What can we do? Can you help me?

Intuitively, I want to edit the history here:

These are the commits I want to edit

So, what would you do?

You are right!

I can rebase the history from "Commit 17" to "Commit 19", on top of "Commit 15". To do that:

git rebase --interactive --onto

Notice I specified "Commit 15" as the beginning of the range of commits, excluding this commit. And I didn't need to explicitly specify HEAD as the last parameter.

Using rebase --onto on a single branch

(Note: If you follow the steps above with my repository and get a merge conflict, you may have a different configuration than on my machine with regards to whitespace characters at line endings. In that case, you can add the --ignore-whitespace switch to the rebase command, resulting in the following command: git rebase --ignore-whitespace --interactive --onto . If you are curious to find out more about this issue, search for autocrlf.)

After following your advice and running the rebase command (thanks! 😇) I get the following screen:

Interactive rebase

So what would I do? I want to put "Commit 19" before "Commit 18", so it comes right after "Commit 17". I can go further and squash them together, like so:

Interactive rebase - changing the order of commit and squashing

Now when I get prompted for a commit message, I can provide the message "Commit 17+19":

Providing a commit message

And now, see our beautiful history:

The resulting history

Thanks again!

More Rebase Use Cases + More Practice

By now I hope you feel comfortable with the syntax of rebase. The best way to actually understand it is to consider various cases and figure out how to solve them yourself.

With the upcoming use cases, I strongly suggest you stop reading after I've introduced each use case, and then try to solve it on your own.

How to Exclude Commits

Say you have this history on another repo:

Another commit history

Before playing around with it, store a tag to "Commit F" so you can get back to it later:

git tag original_commit_f

(A tag is a named reference to a commit, just like a branch - but it doesn't change when you add additional commits. It is like a constant named reference.)

Now, you actually don't want the changes in "Commit C" and "Commit D" to be included. You could use an interactive rebase like before and remove their changes. Or, you could use git rebase --onto again. How would you use --onto in order to "remove" these two commits?

You can rebase HEAD on top of "Commit B", where the old parent was actually "Commit D", and now it should be "Commit B". Consider the history again:

The history again

Rebasing so that "Commit B" is the base of "Commit E" means "moving" both "Commit E" and "Commit F", and giving them another base - "Commit B". Can you come up with the command yourself?

git rebase --onto   HEAD

Notice that using the syntax above (exactly as provided) would not move main to point to the new commit, so the result is a "detached" HEAD. If you use gg or another tool that displays the history reachable from branches, it might confuse you:

Rebasing with --onto results in a detached HEAD

But if you simply use git log (or my alias git lol), you will see the desired history:

The resulting history

I don't know about you, but these kinds of things make me really happy. 😊😇

By the way, you could omit HEAD from the previous command as this is the default value for the third parameter. So just using:

git rebase --onto

Would have the same effect. The last parameter actually tells Git where the end of the current sequence of commits to rebase is. So the syntax of git rebase --onto with three arguments is:

git rebase --onto

How to Move Commits Across Branches

So let's say we get to the same history as before:

git checkout original_commit_f

And now I want only "Commit E" to be on a branch based on "Commit B". That is, I want to have a new branch, branching from "Commit B", with only "Commit E".

The current history, considering "Commit E"

So, what does this mean in terms of rebase? Consider the image above. What commit (or commits) should I rebase, and which commit would be the new base?

I know I can count on you here 😉

What I want is to take "Commit E", and this commit only, and change its base to be "Commit B". In other words, to replay the changes introduced in "Commit E" onto "Commit B".

Can you apply that logic to the syntax of git rebase?

Here it is (this time I'm writing instead of , for brevity):

git rebase --onto

Now the history looks like so:

The history after rebase

Notice that rebase moved HEAD, but not any other reference named (such as a branch or a tag). In other words, you are in a detached HEAD state. So here too, using gg or another tool that displays the history reachable from branches and tags might confuse you. You can use git log (or my alias git lol) to display the reachable history from HEAD.

Awesome!

A Note About Conflicts

Note that when performing a rebase, you may run into conflicts just as when merging. You may have conflicts because, when rebasing, you are trying to apply patches on a different base, perhaps where the patches do not apply.

For example, consider the previous repository again, and specifically, consider the change introduced in "Commit 12", pointed to by main:

git show main

The patch introduced in "Commit 12"

I already covered the format of git diff in detail in chapter 6, but as a quick reminder, this commit instructs Git to add a line after the two lines of context:

This is a sample file


And before these three lines of context:

```patch

def new_feature(): print('new feature')


Say you are trying to rebase "Commit 12" onto another commit. If, for some reason, these context lines don't exist as they do in the patch on the commit you are rebasing onto, then you will have a conflict.

### Zooming Out for the Big Picture

![Comparing rebase and merge](https://www.freecodecamp.org/news/content/images/2023/12/compare_rebase_merge.png)
_Comparing rebase and merge_

In the beginning of this chapter, I started by mentioning the similarity between `git merge` and `git rebase`: both are used to integrate changes introduced in different histories.

But, as you now know, they are very different in how they operate. While merging results in a _diverged_ history, rebasing results in a _linear_ history. Conflicts are possible in both cases. And there is one more column described in the table above that requires some close attention.

Now that you know what "Git rebase" is, and how to use interactive rebase or rebase `--onto`, as I hope you agree, `git rebase` is a super powerful tool. Yet, it has one huge drawback when compared with merging.

**Git rebase changes the history.**

This means that you should **not** rebase commits that exist outside your local copy of the repository, and that other people may have based their commits on.

In other words, if the only commits in question are those you created locally - go ahead, use rebase, go wild.

But if the commits have been pushed, this can lead to a huge problem - as someone else may rely on these commits that you later overwrite, and then you and they will have different versions of the repository.

This is unlike `merge` which, as we have seen, does not modify history.

For example, consider the last case where we rebased and resulted in this history:

![The history after rebase](https://www.freecodecamp.org/news/content/images/2023/12/history_after_rebase_3-1.png)
_The history after rebase_

Now, assume that I have already pushed this branch to the remote. And after I had pushed the branch, another developer pulled it and branched out from "Commit C". The other developer didn't know that meanwhile, I was locally rebasing my branch, and would later push it again.

This results in an inconsistency: the other developer works from a commit that is no longer available on my copy of the repository.

I will not elaborate on what exactly this causes in this book, as my main message is that you should definitely avoid such cases. If you're interested in what would actually happen, I'll leave a link to a useful resource in the [additional references](#heading-additional-references-by-part). For now, let's summarize what we have covered.

### Recap - Understanding Git Rebase

In this chapter, you learned about `git rebase`, a super-powerful tool to rewrite history in Git. You considered a few use cases where git rebase can be helpful, and how to use it with one, two, or three parameters, with and without the `--onto` switch.

I hope I was able to convince you that `git rebase` is powerful - but also that it is quite simple once you get the gist. It is a tool you can use to "copy-paste" commits (or, more accurately, patches). And it's a useful tool to have under your belt. In essence, `git rebase` takes the patches introduced by commits, and replays them on another commit. As described in this chapter, this is useful in many different scenarios.

## Part 2 - Summary

In this part you learned about branching and integrating changes in Git.

You learned what a **diff** is, and the difference between a diff and a **patch**. You also learned how the output of `git diff` is constructed.

Understanding diffs is a major milestone for understanding many other processes within Git such as merging or rebasing.

Then, you got an extensive overview of merging with Git. You learned that **merging** is the process of **combining the recent changes from several branches into a single new commit**. The new commit has multiple parents - those commits which had been the tips of the branches that were merged. In most cases, merging combines the changes from two branches, and the resulting merge commit then has two parents - one from each branch.

We considered a simple, fast-forward merge, which is possible when one branch diverged from the base branch, and then just added commits on top of the base branch.

We then considered three-way merges, and explained the three-stage process:

* First, Git locates the merge base. As a reminder, this is the first commit that is reachable from both branches.
* Second, Git calculates two diffs - one diff from the merge base to the _first_ branch, and another diff from the merge base to the _second_ branch. Git generates patches based on those diffs.
* Third and last, Git applies both patches to the merge base using a 3-way merge algorithm. The result is the state of the new merge commit.

You saw the output of `git diff` when we are in a conflicting state, and how to resolve conflicts either manually or with VS Code.

Ultimately, you got to know Git rebase. You saw that `git rebase` is powerful - but also that it is quite simple once you understand what it does. It is a tool to "copy-paste" commits (or, more accurately, patches).

![Comparing rebase and merge](https://www.freecodecamp.org/news/content/images/2023/12/compare_rebase_merge-1.png)
_Comparing rebase and merge_

Both `git merge` and `git rebase` are used to integrate changes introduced in different histories.

Yet, they differ in how they operate. While merging results in a _diverged_ history, rebasing results in a _linear_ history. `git rebase` _changes_ the history, whereas `git merge` adds to the existing history.

With this deep understanding of diffs, patches, merge and rebase, you should feel confident introducing changes to a git repository.

The next part will focus on what happens when things go wrong - how you can change history (with or without `git rebase`), or find "lost" commits.

# Part 3 - Undoing Changes

Did you ever get to a point where you said: "Uh-oh, what did I just do?" I guess you have, just like about anyone who uses Git.

Perhaps you committed to the wrong branch. Perhaps you lost some code that you had written. Perhaps you committed something that you didn't mean to.

This part will give you the tools to rewrite history with confidence, thereby "undoing" all kinds of changes in Git. 

Just like the other parts of the book, this part will be practical yet in-depth - so instead of providing you with a list of things to do when things go wrong, we will understand the underlying mechanisms, so that you will feel confident whenever you get to the "uh-oh" moment. Actually, you will find these moments as opportunities for an interesting challenge, rather than a dreadful scenario.

## Chapter 9 - Git Reset

Our journey starts with a powerful command that can be used to undo many different actions with Git - `git reset`.

### A Short Reminder - Recording Changes

In [chapter 3](#heading-chapter-3-how-to-record-changes-in-git), you learned how to record changes in Git. If you remember everything from this part, feel free to jump to the next section.

It is very useful to think about Git as a system for recording snapshots of a filesystem in time. Considering a Git repository, it has three "states" or "trees":

1. The **working directory**, a directory that has a repository associated with it.
2. The **staging area (index)** which holds the tree for the next commit.
3. The **repository**, which is a collection of commits and references.

![The three "trees" of a Git repo](https://www.freecodecamp.org/news/content/images/2023/12/3_trees.png)
_The three "trees" of a Git repo_

Note regarding the drawing conventions I use: I include `.git` within the working directory, to remind you that it is a folder within the project's folder on the filesystem. The `.git` folder actually contains the objects and references of the repository, as explained in [chapter 4](#heading-chapter-4-how-to-create-a-repo-from-scratch).

#### Hands-on Demonstration

Use `git init` to initialize a new repository. Write some text into a file called `1.txt`:

```bash
mkdir my_repo
cd my_repo
git init
echo Hello world > 1.txt

Out of the three tree states described above, where is 1.txt now?

In the working tree, as it hasn't yet been introduced to the index.

The file 1.txt is now a part of the working dir only

In order to stage it, to add it to the index, use:

git add 1.txt

Using git add stages the file so it is now in the index as well

Notice that once you stage 1.txt, Git creates a blob object with the content of this file, and adds it to the internal object database (within .git folder), as covered in chapter 3 and chapter 4. I do not draw it as part of the "repository" as in this representation, the "repository" refers to a tree of commits and their references, and this blob has not been a part of any commit.

Now, use git commit to commit your changes to the repository:

git commit -m "Commit 1"

Using git commit creates a commit object in the repository

You created a new commit object, which includes a pointer to a tree describing the entire working tree. In this case, this tree consists only of 1.txt within the root folder. In addition to a pointer to the tree, the commit object includes metadata, such as timestamps and author information.

When considering the diagrams, notice that we only have a single copy of the file 1.txt on disk, and a corresponding blob object in Git's object database. The "repository" tree now shows this file as it is part of the active commit - that is, the commit object "Commit 1" points to a tree that points to the blob with the contents of 1.txt, the same blob that the index is pointing to.

For more information about the objects in Git (such as commits and trees), refer to chapter 1.

Next, create a new file, and add it to the index, as before:

echo second file > 2.txt
git add 2.txt

The file 2.txt is in the working dir and the index after staging it with git add

Next, commit:

git commit -m "Commit 2"

Importantly, git commit does two things:

First, it creates a commit object, so there is an object within Git's internal object database with a corresponding SHA-1 value. This new commit object also points to the parent commit. That is the commit that HEAD was pointing to when you wrote the git commit command.

A new commit object has been created, at first - main still points to the previous commit

Second, git commit moves the pointer of the active branch — in our case, that would be main, to point to the newly created commit object.

git commit also updates the active branch to point to the newly created commit object

Introducing `git reset`

You will now learn how to reverse the process of introducing a commit. For that, you will get to know the command git reset.

`git reset --soft`

The very last step you did before was to git commit, which actually means two things — Git created a commit object and moved main, the active branch. To undo this step, use the following command:

git reset --soft HEAD~1

The syntax HEAD~1 refers to the first parent of HEAD. Consider a case where I had more than one commit in the commit-graph, say "Commit 3" pointing to "Commit 2", which is, in turn, pointing to "Commit 1. And consider HEAD was pointing to "Commit 3". You could use HEAD~1 to refer to "Commit 2", and HEAD~2 would refer to "Commit 1".

So, back to the command: git reset --soft HEAD~1

This command asks Git to change whatever HEAD is pointing to. (Note: In the diagrams below, I use *HEAD for "whatever HEAD is pointing to".) In our example, HEAD is pointing to main. So Git will only change the pointer of main to point to HEAD~1. That is, main will point to "Commit 1".

However, this command did not affect the state of the index or the working tree. So if you use git status you will see that 2.txt is staged, just like before you ran git commit:

git status shows that 2.txt is in the index, but not in the active commit

The state is now:

Resetting main to "Commit 1"

(Note: I removed 2.txt from the "repository" in the diagram as it is not part of the active commit - that is, the tree pointed to by "Commit 1" does not reference this file. However, it has not been removed from the file system - as it still exists in the working tree and the index.)

What about git log? It will start from HEAD , go to main, and then to "Commit 1":

The output of git log

Notice that this means that "Commit 2" is no longer reachable from our history.

Does that mean the commit object of "Commit 2" is deleted?

No, it's not deleted. It still resides within Git's internal object database of objects.

If you push the current history now, by using git push, Git will not push "Commit 2" to the remote server (as it is not reachable from the current HEAD), but the commit object still exists on your local copy of the repository.

Now, commit again - and use the commit message of "Commit 2.1" to differentiate this new object from the original "Commit 2":

git commit -m "Commit 2.1"

This is the resulting state:

Creating a new commit

I omitted "Commit 2" as it is not reachable from HEAD, even though its object exists in Git's internal object database.

Why are "Commit 2" and "Commit 2.1" different? Even if we used the same commit message, and even though they point to the same tree object (of the root folder consisting of 1.txt and 2.txt), they still have different timestamps, as they were created at different times. Both "Commit 2" and "Commit 2.1" now point to "Commit 1", but only "Commit 2.1" is reachable from HEAD.

`git reset --mixed`

It's time to undo even further. This time, use:

git reset --mixed HEAD~1

(Note: --mixed is the default switch for git reset.)

This command starts the same as git reset --soft HEAD~1. That is, the command takes the pointer of whatever HEAD is pointing to now, which is the main branch, and sets it to HEAD~1, in our example - "Commit 1".

The first step of git reset --mixed is the same as git reset --soft

Next, Git goes further, effectively undoing the changes we made to the index. That is, changing the index so that it matches with the current HEAD, the new HEAD after setting it in the first step.

If we ran git reset --mixed HEAD~1, then HEAD (main) would be set to HEAD~1 ("Commit 1"), and then Git would match the index to the state of "Commit 1" - in this case, it means that 2.txt would no longer be part of the index.

The second step of git reset --mixed is to match the index with the new HEAD

It's time to create a new commit with the state of the original "Commit 2". This time you need to stage 2.txt again before creating it:

git add 2.txt
git commit -m "Commit 2.2"

Creating "Commit 2.2"

Similarly to "Commit 2.1", I "name" this commit "Commit 2.2" to differentiate it from the original "Commit 2" or "Commit 2.1" - these commits result in the same state as the original "Commit 2", but they are different commit objects.

`git reset --hard`

Go on, undo even more!

This time, use the --hard switch, and run:

git reset --hard HEAD~1

Again, Git starts with the --soft stage, setting whatever HEAD is pointing to (main), to HEAD~1 ("Commit 1").

The first step of git reset --hard is the same as git reset --soft

Next, moving on to the --mixed stage, matching the index with HEAD. That is, Git undoes the staging of 2.txt.

The second step of git reset --hard is the same as git reset --mixed

Next comes the --hard step, where Git goes even further and matches the working dir with the stage of the index. In this case, it means removing 2.txt also from the working dir.

The third step of git reset --hard matches the state of the working dir with that of the index

So to introduce a change to Git, you have three steps: you change the working dir, the index, or the staging area, and then you commit a new snapshot with those changes. To undo these changes:

If we use git reset --soft, we undo the commit step.
If we use git reset --mixed, we also undo the staging step.
If we use git reset --hard, we undo the changes to the working dir.

The three main switches of git reset

Real-Life Scenarios

Scenario #1

So in a real-life scenario, write "I love Git" into a file (love.txt), as we all love Git 😍. Go ahead, stage and commit this as well:

echo I love Git > love.txt
git add love.txt
git commit -m "Commit 2.3"

Creating "Commit 2.3"

Also, save a tag so that you can get back to this commit later if needed:

git tag scenario-1

Oh, oops!

Actually, I didn't want you to commit it.

What I actually wanted you to do is write some more love words in this file before committing it.

What can you do?

Well, one way to overcome this would be to use git reset --mixed HEAD~1, effectively undoing both the committing and the staging actions you took:

git reset --mixed HEAD~1

Undoing the staging and committing steps

So main points to "Commit 1" again, and love.txt is no longer a part of the index. However, the file remains in the working dir. You can now add more content to it:

echo and Gitting Things Done >> love.txt

Adding more love lyrics

Stage and commit your file:

git add love.txt
git commit -m "Commit 2.4"

Introducing "Commit 2.4"

Well done!

You got this clear, nice history of "Commit 2.4" pointing to "Commit 1".

You now have a new tool in your toolbox, git reset.

This tool is super, super useful, and you can accomplish almost anything with it. It's not always the most convenient tool to use, but it's capable of solving almost any rewriting-history scenario if you use it carefully.

For beginners, I recommend using only git reset for almost any time you want to undo in Git. Once you feel comfortable with it, move on to other tools.

Scenario #2

Let us consider another case.

Create a new file called new.txt; stage and commit:

echo this is a new file > new.txt
git add new.txt
git commit -m "Commit 3"

Creating new.txt and "Commit 3"

(Note: In the drawing I omitted the files from the repository to avoid clutter. Commit 3 includes 1.txt, love.txt and new.txt at this stage).

Oops. Actually, that's a mistake. You were on main, and I wanted you to create this commit on a feature branch. My bad 😇

There are two most important tools I want you to take from this chapter. The second is git reset. The first and by far more important one is to whiteboard the current state versus the state you want to be in.

For this scenario, the current state and the desired state look like so:

Scenario #2: current-vs-desired states

(Note: In following diagrams, I will refer to the current state as the "original" state - before starting the process of rewriting history.)

You will notice three changes:

main points to "Commit 3" (the blue one) in the current state, but to "Commit 2.4" in the desired state.
feature_branch doesn't exist in the current state, yet it exists and points to "Commit 3" in the desired state.
HEAD points to main in the current state, and to feature_branch in the desired state.

If you can draw this and you know how to use git reset, you can definitely get yourself out of this situation.

So again, the most important thing is to take a breath and draw this out.

Observing the drawing above, how do you get from the current state to the desired one?

There are a few different ways of course, but I will present one option only for each scenario. Feel free to play around with other options as well.

You can start by using git reset --soft HEAD~1. This would set main to point to the previous commit, "Commit 2.4":

git reset --soft HEAD~1

Changing main: "Commit 3" is still there, just not reachable from HEAD

Peeking at the current-vs-desired diagram again, you can see that you need a new branch, right? You can use git switch -c feature_branch for it, or git checkout -b feature_branch (which does the same thing):

git switch -c feature_branch

_Creating feature_branch branch_

This command also updates HEAD to point to the new branch.

Since you used git reset --soft, you didn't change the index, so it currently has exactly the state you want to commit - how convenient! You can simply commit to feature_branch:

git commit -m "Commit 3.1"

_Committing to feature_branch branch_

And you got to the desired state.

Scenario #3

Ready to apply your knowledge to additional cases?

Still on feature_branch, add some changes to love.txt, and create a new file called cool.txt. Stage them and commit:

echo Some changes >> love.txt
echo Git is cool > cool.txt
git add love.txt
git add cool.txt
git commit -m "Commit 4"

The history, as well as the state of the index and the working dir after creating "Commit 4"

Oh, oops, actually I wanted you to create two separate commits, one with each change...

Want to try this one yourself (before reading on)?

You can undo the committing and staging steps:

git reset --mixed HEAD~1

Following this command, the index no longer includes those two changes, but they're both still in your file system:

Resulting state after using git reset --mixed HEAD~1

So now, if you only stage love.txt, you can commit it separately:

git add love.txt
git commit -m "Love"

Resulting state after committing the changes to love.txt

Then, do the same for cool.txt:

git add cool.txt
git commit -m "Cool"

Committing separately

Nice!

Scenario #4

To clear up the state, switch to main and use reset --hard to make it point to "Commit 3.1", while setting the index and the working dir to the state of "Commit 3.1":

git checkout main
git reset --hard

Resetting main to "Commit 3.1"

Create another file (another.txt) with some text, and add some text to love.txt. Stage both changes, and commit them:

echo Another file > another.txt
echo More love >> love.txt
git add another.txt
git add love.txt
git commit -m "Commit 4.1"

This should be the result:

A new commit

Oops...

So this time, I wanted it to be on another branch, but not a new branch, rather - an already-existing branch.

So what can you do?

I'll give you a hint. The answer is really short and really easy. What do we do first?

No, not reset. We draw. That's the first thing to do, as it would make everything else so much easier. So this is the current state:

The new commit on main appears blue

And the desired state?

We want the "blue" commit to be on another, existing, branch

How do you get from the current state to the desired state, what would be easiest?

One way would be to use git reset as you did before, but there is another way that I would like you to try.

Note that the following commands indeed assume the branch existing exists on your repository, yet you haven't created it earlier. To match a state where this branch actually exists, you can use the following commands:

git checkout 
git checkout -b existing
echo "Hello" > x.txt
git add x.txt
git commit -m "Commit X"
git checkout  -- love.txt
git commit -m "Commit Y"
git checkout main

(The command git checkout -- love.txt copies the contents of love.txt from "Commit 3.1" to the index and the working dir, so that you can commit it on the existing branch. We need the state of love.txt on "Commit Y" to be the same as of "Commit 3.1" to avoid conflicts.)

Now your history should match the one shown in the picture with the caption "We want the "blue" commit to be on another, existing, branch".

First, move HEAD to point to existing branch:

git switch existing

Switch to the existing branch

Intuitively, what you want to do is take the changes introduced in "Commit 4.1", and apply these changes ("copy-paste") on top of existing branch. And Git has a tool just for that.

To ask Git to take the changes introduced between a commit and its parent commit and just apply these changes on the active branch, you can use git cherry-pick, a command we introduced in chapter 8. This command takes the changes introduced in the specified revision and applies them to the state of the active commit. Run:

git cherry-pick

You can specify the SHA-1 identifier of the desired commit, but you can also use git cherry-pick main, as the commit whose changes you are applying is the one main is pointing to.

git cherry-pick also creates a new commit object, and updates the active branch to point to this new object, so the resulting state would be:

The result after using git cherry-pick

I mark the commit as "Commit 4.2" since it has a different timestamp, parent and SHA-1 value than "Commit 4.1", though the changes it introduces are the same.

You made good progress - the desired commit is now on the existing branch! But we don't want these changes to exist on main branch. git cherry-pick only applied the changes to the existing branch. How can you remove them from main?

One way would be to switch back to main, and then reset it:

git switch main
git reset --hard HEAD~1

And the result:

The resulting state after resetting main

You did it!

Note that git cherry-pick actually computes the difference between the specified commit and its parent, and then applies the difference to the active commit. This means that sometimes, Git won't be able to apply those changes due to a conflict.

Also, note that you can ask Git to cherry-pick the changes introduced in any commit, not only commits referenced by a branch.

Recap - Git Reset

In this chapter, we learned how git reset operates, and clarified its three main modes of operation:

git reset --soft , which changes whatever HEAD is pointing to - to .
git reset --mixed , which goes through the --soft stage, and also sets the state of the index to match that of HEAD.
git reset --hard , which goes through the --soft and --mixed stages, and then sets the state of the working dir to match that of the index.

You then applied your knowledge about git reset to solve some real-life issues that arise when using Git.

By understanding the way Git operates, and by whiteboarding the current state versus the desired state, you can confidently tackle all kinds of scenarios.

In the future chapters, we will cover additional Git commands and how they can help us solve all kinds of undesired situations.

Chapter 10 - Additional Tools for Undoing Changes

In the previous chapter, you met git reset. Indeed, git reset is a super powerful tool, and I highly recommend to use it until you feel completely comfortable with it.

Yet, git reset is not the only tool at our disposal. Some of the times, it is not the most convenient tool to use. In others, it's just not enough. This short chapter touches a few tools that are helpful for undoing changes in Git.

`git commit --amend`

Consider Scenario #1 from the previous chapter again. As a reminder, you wrote "I love Git" into a file (love.txt), staged and committed this file:

The state after creating "Commit 2.3"

And then I realized I didn't want you to commit it at that state, but rather - write some more love words in this file before committing it.

To match this state, simply checkout the tag you created, which points to "Commit 2.3":

git checkout scenario-1

In the previous chapter, when we introduced git reset, you solved this issue by using git reset --mixed HEAD~1, effectively undoing both the committing and the staging actions you took.

Now I would like you to consider another approach. Keep working at the state of the last introduced commit ("Commit 2.3", referenced by the tag "scenario-1"), and make the changes you want:

echo And I love this book >> love.txt

Add this change to the index:

git add love.txt

Now, you can use git commit with the --amend switch, which tells it to override the commit HEAD is pointing to. Actually, it will create another, new commit, pointing to HEAD~1 ("Commit 1" in our example), and make HEAD point to this newly created commit. By providing the -m argument you can specify a new commit message as well:

git commit --amend -m "Commit 2.4"

After running this command, HEAD points to main, which points to "Commit 2.4", which in turn points to "Commit 1". The previous "Commit 2.3" is no longer reachable from the history.

The state after using git commit --amend (Commit "2.3" is unreachable and thus not included in the drawing)

This tool is useful when you want to quickly override the last commit you created. Indeed, you could use git reset to accomplish the same thing, but you can view git commit --amend as a more convenient shortcut.

`git revert`

Okay, so another day, another problem.

Add the following text to love.txt, stage and commit as follows:

echo This is more tezt >> love.txt
git add love.txt
git commit -m "Commit 3"

The state after committing "Commit 3"

And push it to the remote server:

git push origin HEAD

Um, oops 😓…

I just noticed something. I had a typo there. I wrote "This is more tezt" instead of "This is more text". Whoops. So what's the big problem now? I pushed, which means that someone else might have already pulled those changes.

If I override those changes by using git reset, we will have different histories, and all hell might break loose. You can rewrite your own copy of the repo as much as you like until you push it.

Once you push the change, you need to be certain no one else has fetched those changes if you are going to rewrite history.

Alternatively, you can use another tool called git revert. This command takes the commit you're providing it with and computes the diff from its parent commit, just like git cherry-pick, but this time, it computes the reverse changes. That is, if in the specified commit you added a line, the reverse would delete the line, and vice versa.

In our case we are reverting "Commit 3", so the reverse would be to delete the line "This is more tezt" from love.txt. Since "Commit 3" is referenced by main and HEAD, we can use any of these named references in this command:

Using git revert to undo the changes

git revert created a new commit object, which means it's an addition to the history. By using git revert, you didn't rewrite history. You admitted your past mistake, and this commit is an acknowledgment that you made a mistake and now you fixed it.

Some would say it's the more mature way. Some would say it's not as clean a history as you would get if you used git reset to rewrite the previous commit. But this is a way to avoid rewriting history.

You can now fix the typo and commit again:

echo This is more text >> love.txt
git add love.txt
git commit -m "Commit 3.1"

The resulting state after redoing the changes

You can use git revert to revert a commit other than HEAD. Say that you want to reverse the parent of HEAD, you can use:

git revert HEAD~1

Or you could provide the SHA-1 of the commit to revert.

Notice that since Git will apply the reverse patch of the previous patch - this operation might fail, as the patch may no longer apply and you might get a conflict.

Git Rebase as a Tool for Undoing Things

In chapter 8, you learned about Git rebase. We then considered it mainly as a tool to combine changes introduced in different branches. Yet, as long as you haven't pushed your changes, using rebase on your own branch can be a very convenient way to rearrange your commit history.

For that, you would usually rebase on a single branch, and use interactive rebase. Consider again this example covered in chapter 8, where I worked from feature_branch_2, and specifically edited the file code.py. I started by changing all strings to be wrapped by double quotes rather than single quotes:

Changing ' into " in code.py

Then, I staged and committed:

git add code.py
git commit -m "Commit 17"

I then decided to add a new function at the beginning of the file:

_Adding the function another_feature_

Again, I staged and committed:

git add code.py
git commit -m "Commit 18"

And now I realized I actually forgot to change the single quotes to double quotes wrapping the __main__ (as you might have noticed), so I did that too:

Changing '__main__' into "__main__"

Of course, I staged and committed this change:

git add code.py
git commit -m "Commit 19"

Now, consider the history:

The commit history after introducing "Commit 19"

As explained in chapter 8, I got to a state with two commits that are related to one another, "Commit 17" and "Commit 19" (turning 's into "s), but they are split by the unrelated "Commit 18" (where I added a new function).

This is a classic case where git rebase would come in handy, to undo the local changes before pushing a clean history.

Intuitively, I want to edit the history here:

These are the commits I want to edit

I can rebase the history from "Commit 17" to "Commit 19", on top of "Commit 15". To do that:

git rebase --interactive --onto

Using rebase --onto on a single branch

This results in the following screen:

Interactive rebase

So what would I do? I want to put "Commit 19" before "Commit 18", so it comes right after "Commit 17". I can go further and squash them together, like so:

Interactive rebase - changing the order of commit and squashing

Now when I get prompted for a commit message, I can provide the message "Commit 17+19":

Providing a commit message

And now, see our beautiful history:

The resulting history

The syntax used above, git rebase --interactive --onto would be the most commonly used syntax by those who use rebase regularly. The state of mind these developers usually have is to create atomic commits while working, all the time, without being scared to change them later. Then, before pushing their changes, they would rebase the entire set of changes since the last push, and rearrange it so the history becomes coherent.

`git reflog`

Time to consider a more startling case.

Go back to "Commit 2.4":

git reset --hard

Get some work done, write some code, and add it to love.txt. Stage this change, and commit it:

echo lots of work >> love.txt
git add love.txt
git commit -m "Commit 3.2"

(I'm using "Commit 3.2" to indicate that this is not the same commit as "Commit 3" we used when explaining git revert.)

Another commit - "Commit 3.2"

I did the same on my machine, and I used the Up arrow key on my keyboard to scroll back to previous commands, and then I hit Enter, and… Wow.

Whoops.

Did I just git reset -- hard?

Did I just use git reset --hard? 😨

What actually happened? As you learned in the previous chapter, Git moved the pointer to HEAD~1, so the last commit, with all of my precious work, is not reachable from the current history. Git also removed all the changes from the staging area, and then matched the working dir to the state of the staging area.

That is, everything matches this state where my work is… gone.

Freak out time. Freaking out.

But, really, is there a reason to freak out? Not really… We're relaxed people. What do we do? Well, intuitively, is the commit really, really gone?

No. Why not? It still exists inside the internal database of Git.

If I only knew where that is, I would know the SHA-1 value that identifies this commit, and we could restore it. I could even undo the undoing, and reset back to this commit.

Actually, the only thing I really need here is the SHA-1 of the "deleted" commit.

Now the question is, how do I find it? Would git log be useful?

Well, not really. git log would go to HEAD, which points to main, which points to the parent commit of the commit we are looking for. Then, git log would trace back through the parent chain, which does not include the commit with my precious work.

git log doesn't help in this case

Thankfully, the very smart people who created Git also created a backup plan for us, and that is called the reflog.

While you work with Git, whenever you change HEAD, which you can do by using git reset, but also other commands like git switch or git checkout, Git adds an entry to the reflog.

git reflog shows us where HEAD was

We found our commit! It's the one starting with 0fb929e.

We can also relate to it by its "nickname" - HEAD@{1}. Similar to the way Git uses HEAD~1 to get to the first parent of HEAD, and HEAD~2 to refer to the second parent of HEAD and so on, Git uses HEAD@{1} to refer to the first reflog parent of HEAD, that is, where HEAD pointed to in the previous step.

We can also ask git rev-parse to show us its value:

Using git rev-parse HEAD@{1}

Note: In case you are using Windows, you may need to wrap it with quotation marks - like so:

git rev-parse "HEAD@{1}"

Another way to view the reflog is by using git log -g, which asks git log to actually consider the reflog:

The output of git log -g

You can see in the output of git log -g that the reflog's entry HEAD@{0}, just like HEAD, points to main, which points to "Commit 2". But the parent of that entry in the reflog points to "Commit 3".

So to get back to "Commit 3", you can just use git reset --hard HEAD@{1} (or the SHA-1 value of "Commit 3"):

git reset --hard HEAD@{1}

And now, if you git log:

Our history is back!!!

We saved the day!

What would happen if I used this command again? And ran git reset --hard HEAD@{1}?

Git would set HEAD to where HEAD was pointing before the last reset, meaning to "Commit 2". We can keep going all day:

git reset --hard again

Recap - Additional Tools for Undoing Changes

In the previous chapter, you learned how to use git reset to undo changes.

In this chapter, you extended your toolbox for undoing changes in Git with a few new commands:

git commit --amend - which "overrides" the last commit with the stage of the index. Mostly useful when you just committed something and want to modify that last commit.
git revert - which creates a new commit, that reverts a past commit by adding a new commit to the history with the reversed changes. Useful especially when the "faulty" commit has already been pushed to the remote.
git rebase - which you already know from chapter 8, and is useful for rewriting the history of multiple commits, especially before pushing them.
git reflog (and git log -g) - which tracks all changes to HEAD, so you might find the SHA-1 value of a commit you need to get back to.

The most important tool, even more important than the tools I just listed, is to whiteboard the current situation vs the desired one. Trust me on this, it will make every situation seem less daunting and the solution more clear.

There are additional tools that allow you to reverse changes in Git (I will provide links in the appendix), but the collection of tools covered here should prepare you to tackle any challenge with confidence.

Chapter 11 - Exercises

This chapter includes a few exercises to deepen your understanding of the tools you learned in Part 3. The full version of this book also includes detailed solutions for each.

The exercises are found on this repository:

https://github.com/Omerr/undo-exercises.git

Each exercise exists on a branch with the name exercise_XX, so Exercise 1 is found on branch exercise_01, Exercise 2 is found on branch exercise_02 and so on.

Note: As explained in previous chapters, if you work with commits that can be found on a remote server (which you are in this case, as you are using my repository "undo-exercises"), you should probably use git revert instead of git reset. Similar to git rebase, the command git reset also rewrites history - and thus you should refrain from using it on commits that others may have relied on.

For the purposes of these exercises, you can assume no one else has cloned or pulled code from the remote repository. Just remember - in real life, you should probably use git revert instead of commands that rewrite history in such cases.

Exercise 1

On branch exercise_01, consider the file hello.txt:

The file hello.txt

This file includes a typo (in the last character). Find the commit that introduced this typo.

Exercise (1a)

Remove this commit from the reachable history using git reset (with the right arguments), fix the typo, and commit again. Consider your history.

Revert to the previous state.

Exercise (1b)

Remove the faulty commit using git commit --amend, and get to the same state of the history as in the end of exercise (1a).

Revert to the previous state.

Exercise (1c)

revert the faulty commit using git revert and fix the typo. Consider your history.

Revert to the previous state.

Exercise (1d)

Using git rebase, get to the same state as in the end of exercise (1a).

Exercise 2

Switch to exercise_02, and consider the contents of exercise_02.txt:

_The contents of exercise_02.txt_

A simple file, with one character at each line.

Consider the history (using git lol):

git lol

Oh my. Each character was introduced in a separate commit. That doesn't make any sense!

Use the tools you've acquired to create a history where the creation of exercise_02.txt is all done in a single commit.

Exercise 3

Consider the history on branch exercise_03:

_The history on exercise_03_

This seems like a mess. You will notice that:

The order is skewed. We need "Commit 1" to be the earliest commit on this branch, and have "Initial Commit" as its parent, followed by "Commit 2" and so on.
We shouldn't have "Commit 2a" and "Commit 2b", or "Commit 4a" and "Commit 4b" - these two pairs need to be combined into a single commit each - "Commit 2" and "Commit 4".
There is a typo on the commit message of "Commit 1", it should not have 3 ms.

Fix these issues, but rely on the changes of each original commit. The resulting history should look like so:

The desired history

Exercise 4

This exercise actually consists of three branches: exercise_04, exercise_04_a, and exercise_04_b.

To see the history of these branches without others, use the following syntax:

git lol --branches="exercise_04*"

The result is:

_The output of git lol --branches="exercise_04*"_

Your goal is to make exercise_04_b independent of exercise_04_a. That is, get to this history:

The desired history

Good luck!

Part 4 - Amazing and Useful Git Tools

Git has lots of commands, and these commands have so many options and arguments. I could try to cover them all (though they do change over time), but I don't see a point in that. You should probably know a subset of these commands really well, those that you use regularly. Then, you can always search for a specific command to perform a task at hand.

This part relies on the basics you acquired in the previous parts, and covers specific commands and options that you may find useful. Given your understanding of how Git works, having these small tools can make you a real pro in Gitting things done.

Chapter 12 - Git Log

You used git log many times across different chapters, and you had probably used it many times before reading this book.

Most developers use git log, few use it effectively. In this chapter you will learn useful tweaks for making the most of git log. Once you feel comfortable with the different switches of this command, it will be a game changer in your day to day work with Git.

Thinking about it, git log encompasses the essence of every version control system - that is, to record changes in versions. You record versions so that you can consider the history of your project - perhaps revert or apply specific changes, prefer to switch to a different point in time and test things there. Perhaps you would like to know who contributed a certain piece of code or when they did that.

While git does preserve this information by using commit objects, that also point to their parent commits, and references to commit objects (such as branches or HEAD), this storing of versions is not enough. Without being able to find the relevant commit you would like to consider, or gather the relevant information about it, having this data stored is pretty useless.

You can think of your commit objects as different books that pile up in a huge stack, or in a library, filling long shelves. The information you might need is in these books, but if you don't have an index - a way to know in which book the information you seek lies, or where this book is located within the library - you wouldn't be able to make much use of it. git log is this indexing of your library - it's a way to find the relevant commits and the information about them.

The useful arguments for git log that you will learn in this chapter either format how commits are displayed in the log, or filter specific commits.

git lol, an alias which I have used throughout the book, uses some of these switches, as I will demonstrate. Feel free to tweak this alias (or create another from scratch) after reading this chapter.

As in other chapters, the goal is not to provide a complete reference, therefore I will not provide all different switches of git log. I will focus on the switches I believe you will find useful.

Filtering Commits

Consider the default output of git log:

The output of git log without additional switches

The log starts from HEAD, and follows the parent chain.

Commits (Not) Reachable From...

When you write git log , git log will include all entries reachable from . By "reachable", I refer to reachable by following the parent chain. So running git log without any arguments is equivalent to running git log HEAD.

You can specify multiple revisions for git log - if you write git log branch_1 branch_2, you ask git log to include every commit that is reachable from branch_1 or branch_2 (or both).

git log will exclude any commits that are reachable from revisions preceded by a ^.

For example, the following command:

git log branch_1 ^branch_2

asks git log to include every commit that is reachable from branch_1, but not those reachable from branch_2.

Consider the history when I use git log feature_branch_1 on this repo:

_git log feature_branch_1_

The history includes all commits reachable by feature_branch_1. Since this branch "branched off" main (that is, "Commit 12", which main points to, is reachable from the parent chain) - the log also includes the commits reachable from main.

What would happen if I ran this command?

git log feature_branch_1 ^main

_git log feature_branch_1 ^main_

Indeed, git log outputs only "Commit 13" and "Commit 16", which are reachable from feature_branch_1 but not from main.

`git log --all`

To follow commits that are reachable from any named reference or (any refs in refs/) or HEAD.

By Author

If you know you are looking for a commit that a specific person has authored, you can filter these commits by using that user's name or email, like so:

git log --author="Name"

You can use regular expressions to look for author names that match a specific pattern, for example:

git log --author="John\|Jane"

will filter commits authored by either John or Jane.

By Date

When you know that the change you are looking for has been committed within a specific timeframe, you can use --before or --after to filter commits from that timeframe.

For example, to get all commits introduced after April 12th, 2023 (inclusive), use:

git log --after="2023-04-12"

By Paths

You can ask git log to only show commits where changes to files in specific paths have been introduced. Notice that this does not mean any commit that points to a tree that includes the files in question, but rather that if we compute the difference between the commit in question and its parent, we would see that at least one of the paths has been modified.

For example, you can use:

git log --all -- 1.py

to find all commits that are reachable from any named pointer, or HEAD, and introduce a change to 1.py. You can specify multiple paths:

git log --all -- 1.py 2.py

The previous command will make git log include reachable commits that introduced a change to 1.py or 2.py (or both).

You can also use a glob pattern, for example:

git log -- *.py

will include commits reachable from HEAD that include a change to any file in the root directory whose name ends with a .py. To look for any file whose name ends with .py, you can use:

git log -- **/*.py

By Commit Message

If you know the commit message (or parts of it) of the commit you are searching, you can use the --grep switch for "git log", for example:

git log --grep="Commit 12"

yields back the commit with the message "Commit 12".

By Diff Content

This one is super useful, and it saved me countless times. By using git log -S, you can search for commits that introduce or remove a particular line of source code.

This comes in handy, for example, when you know you have created something in the repo, but you don't know where it is now. You can't find it anywhere on your filesystem (it's not in HEAD), and you know it must be there - lurking somewhere in this library (bunch of commits) that you have.

Say I remember I wrote a line with the text Git is awesome, but I can't find it now. I could run:

git log --all -S"Git is awesome"

Notice I used --all to avoid restraining myself to commits reachable from HEAD.

You can also search for a regular expression, using -G:

git log --all -G"Git .* awesome"

Formatting Log

Consider the default output of git log again:

The output of git log without additional switches

The log starts from HEAD, and follows the parent chain.

Each log entry begins with a line starting with commit and then the SHA-1 of the commit, perhaps followed by additional pointers that point to this commit.
It is then followed by the author, date, and commit message.

`--oneline`

The main difficulty with the default output of git log is that it is hard to understand a history with more than a few commits, as you simply don't see them all.

In the output of git log shown before, only four commit objects appeared on my screen. Using git log --oneline provides a more concise view, showing the SHA-1 of the commit, next to its message, and named references if relevant:

The output of git log --oneline

If you wish to omit the named references, you can add the --no-decorate switch:

The output of git log --oneline --no-decorate

To explicitly ask for git log to show decorations, you can use git log --decorate.

`--graph`

git log --oneline shows a compact representation. That is great when we have a linear history, perhaps on a single branch. But what happens when we have multiple branches, that may diverge from one another?

Consider the output of the following command on my repository:

git log --oneline feature_branch_1 feature_branch_2

_The output of git log --oneline feature_branch_1 feature_branch_2_

git log outputs any commit reachable by feature_branch_1, feature_branch_2, or both. But what does the history look like? Did feature_branch_2 diverge from feature_branch_1? Or did it diverge from main? It is impossible to tell from this view.

This is where --graph comes in handy, drawing an ASCII graph representing the branch structure of the commit history. If we add this option to the previous command:

_The output of git log --oneline --graph feature_branch_1 feature_branch_2_

You can actually see that feature_branch_1 branched from main (as "Commit 12", main, is the parent of "Commit 13"), and also that feature_branch_2 branched from main (as the parent of "Commit 14" is also "Commit 12").

The * symbol tells us which branch a certain commit is "on", so you can know for sure that "Commit 13" is on feature_branch_1, and not feature_branch_2.

`--pretty=format`

The above result is already very useful! Yet, it lacks a few things. We don't know the author or the time of the commit. These two information details were included in the default output of git log which was very long. Perhaps we can add them in a more compact way?

By using --pretty=format:, you can display the information of each commit in various ways using printf-style placeholders.

In the following command, the %s, %an and %cd placeholders are replaced by the commit's subject (message), author name, and the commit's date, respectively.

git log --oneline --graph feature_branch_1 feature_branch_2 --pretty=format:"%s (%an) [%cd]"

The output looks like this:

_git log --oneline --graph feature_branch_1 feature_branch_2 --pretty=format:"%s (%an) [%cd]_

That's useful, but not really great to look at. We can then use other formatting tricks, specifically %C(color) that will switch the color to color, until reaching a %Creset that resets the color. To make the author name's yellow, you can use:

git log --oneline --graph feature_branch_1 feature_branch_2 --pretty=format:"%s %C(yellow)(%an)%Creset [%cd]"

_git log --oneline --graph feature_branch_1 feature_branch_2 --pretty=format:"%s %C(yellow)(%an)%Creset [%cd]"_

For some colors, like red or green, it is unnecessary to include the parenthesis, so Cred is enough.

How is `git lol` Structured?

When I run git lol, it actually executes the following:

git log --graph --pretty=format:'%Cred%h%Creset -%C(yellow)%d%Creset %s %Cgreen(%cr) %C(bold blue)<%an>%Creset' --abbrev-commit

Can you take this bit by bit?

You already know --graph, which makes the output include an ASCII graph.

--abbrev-commit uses a short prefix from the full SHA-1 of the commit (in my configuration, the first seven characters).

The rest is just coloring of various details about the commit:

git lol --all

git lol --all

I like this output because I find it clear. It gives me the information I need, with enough coloring so that every detail stands out without hurting my eyes. But if you prefer other information, other colors, a different order, or anything else - go ahead and tweak it to your liking.

Setting an alias

As you know, I set git lol as an alias - that is, when I run git lol, it executes the long command I provided previously.

How can you create an alias in Git?

The easiest way is to use git alias, like so:

git config --global alias.co checkout

This command sets co to be an alias for the command checkout, so you can use git co main instead of git checkout main.

To define git lol as an alias, you can use:

git config --global alias.lol 'log --graph --pretty=format:'%Cred%h%Creset -%C(yellow)%d%Creset %s %Cgreen(%cr) %C(bold blue)<%an>%Creset' --abbrev-commit'

Chapter 13 - Git Bisect

Oops.

I have a bug.

Yes, that happens some times, to all of us. Something in my system is broken, and I can't tell why. I have been debugging for a while, but the solution is not clear.

I can tell that two weeks ago, this didn't happen. Luckily for me, I have been using Git (obviously, I know...), so I can go back in time and test a past version of my code. Indeed, in this version - everything worked fine.

But... I have made many changes in these two weeks. Alas, not just me - my entire team has contributed commits that add, delete, or modify parts of the code base. Where do I begin? Should I go over every change introduced in those two weeks?

Enter - git bisect.

The goal of git bisect is help you find the commit where a bug was introduced, in an effective manner.

How Does `git bisect` Work?

git bisect first asks you to mark one commit as "bad" (where the bug occurs), and another commit as "good" (one without the bug). Then, it checks out a commit halfway between these two commits, and then asks you to identify the commit as either "good" or "bad". This process is repeated until you find the first "bad" commit.

The key here is using binary search - by looking at the halfway point and deciding if it is the new top or bottom of the list of commits, you can find the right commit efficiently. Even if you have 10,000 commits to hunt through, it only takes a maximum of 13 steps to find the first commit that introduced the bug.

`git bisect` Example

For this example, I will use the repository on https://github.com/Omerr/bisect-exercise.git. To create it, I adapted the open source repository https://github.com/bast/git-bisect-exercise (according to its license).

In this repository, we have a single python file that is used to compute the value of pi (which is approximately 3.14). If you run python3 get_pi.py on main, however, you will get a wrong result:

A wrong result, we have a bug

This branch consists of more than 500 commits.

Find the first commit on this branch by using:

git log --oneline | tail -n 1

git log --oneline | tail -n 1

If you checkout to this commit and run python3 get_pi.py again, the result is correct:

From the first commit, the result is valid

So somewhere between HEAD and commit f0ea950, a change was introduced that resulted in this wrong output.

To find it using git bisect, start the bisect process, and mark this commit as "good":

git bisect start
git bisect good

By default, git bisect good would take HEAD as the "good" commit. To mark main as "bad", you can use git bisect bad main:

git bisect bad main

git bisect checked out commit number 251, the "middle point" of main branch. Does the state in this commit produce the right or wrong output?

Trying again...

We still get the wrong output, which means we can discard commits 252 through 500 (and additional commits after that), and narrow our search to commits 2 through 251. Mark this as bad:

Mark as bad

git bisect checked out the "middle" commit (number 126), and running the code again results in the right answer! This means that this commit is "good", and that the first "bad" commit is somewhere between 127 and 251. Mark it as "good":

Mark as good

Nice, git bisect takes us to commit 188, as this is the "middle" commit between 127 and 251. By running the code again, you can see that the result is wrong, so this is actually a "bad" commit, which means the first faulty commit is somewhere between 127 and 188. As you can see, git bisect narrows down the search space by half on each iteration.

Come on, now it's your turn - keep going from here! Test the result of python3 get_pi.py and use git bisect good or git bisect bad to mark the commit accordingly. What is the faulty commit?

When you are done, use git bisect reset to stop the bisect process.

Automatic `git bisect`

In the previous example, you could simply run python3 get_pi.py and check the result. Other times, the process of validating whether a certain commit is "good" or "bad" can be tricky, error prone, or just time consuming.

It is possible to automate the process of git bisect by creating code that would be executed on each iteration, returning 0 when the current commit is "good", and a value between 1-127 (inclusive), except 125, if it should be considered "bad".

The syntax is:

git bisect run my_script arguments

As this book is not about programming and doesn't assume you know a specific programming language, I will not show an example of implementing my_script. The README.md file in the repository used in this chapter (https://github.com/Omerr/bisect-exercise.git) includes an example for a script that you can run with git bisect run to automatically find the faulty commit for the previous example.

Chapter 14 - Other Useful Commands

This chapter highlights a few commands that had have already been mentioned in previous chapters. I am putting them here together so that you can come back to them as a reference when needed.

`git cherry-pick`

Introduced in chapter 8, this command takes a given commit, computes the patch this commit introduces by computing the difference between the parent's commit and the commit itself, and then cherry-pick "replays" this difference. It is like "copy-pasting" a commit, that is, the diff this commit introduced.

In chapter 8 we considered the difference introduced by "Commit 5" (using git diff main ):

Running git diff to observe the patch introduced by "Commit 5"

You can see that in this commit, John started working on a song called "Lucy in the Sky with Diamonds":

The output of git diff - the patch introduced by "Commit 5"

As a reminder, you can also use the command git show to get the same output:

git show

Now, if you cherry-pick this commit, you will introduce this change specifically, on the active branch. You can switch to main branch:

git checkout main (or git switch main)

And create another branch:

git checkout -b my_branch (or git switch -c my_branch)

_Creating my_branch that branches from main_

Next, cherry-pick "Commit 5":

git cherry-pick

Using cherry-pick to apply the changes introduced in "Commit 5" onto main

Consider the log (output of git lol):

The output of git lol

Looking at the changes, using git show HEAD:

The output of git show HEAD

They are the same as "Commit 5"'s.

`git revert`

git revert is essentially the reverse of git cherry-pick, introduced in chapter 10. This command takes the commit you're providing it with and computes the diff from its parent commit, just like git cherry-pick, but this time, it computes the reverse changes. That is, if in the specified commit you added a line, the reverse would delete the line, and vice versa.

`git add -p`

Staging changes is an integral part of introducing changes to Git. Sometimes, you wish to stage all changes together (with git add .), or perhaps stage all changes of a specific file (using git add ). Yet there are times where it would be convenient to stage only certain parts of modified files.

In chapter 6, we introduced git add -p. This command allows you to stage certain parts of files, by splitting them into hunks (p stands for patch). For example, say you have this file, my_file.py:

_my_file.py_

You then modify this file - by changing text within function_1, and also adding a new function, function_5:

_my_file.py after the changes_

If you used git add my_file.py at this point, you would stage both of these changes together. In case you want to separate them into different commits, you could use git add -p, which splits these two changes and asks you about each one as a standalone hunk:

git add -p

By typing ?, you can see what the different options stand for:

Using a ? to get a description of the different options

In this case, say we only want to stage the change introducing function_5. We do not want to stage the change of function_1, so we select n:

_Not staging the change to function_1_

Next, we are prompted for the second change - the one introducing function_5. We want to stage this hunk indeed, to can do so we can type y.

Summary

Well, this was FUN!

Can you believe how much you have learned?

In Part 1 you learned about - blobs, trees, and commits.

You then learned about branches, seeing that they are nothing but a named reference to a commit.

You learned the process of recording changes in Git, and that it involves the working directory, the staging area (index), and the repository.

Then - you created a new repository from scratch, by using echo and low-level commands such as git hash-object. You created a blob, a tree, and a commit object pointing to that tree.

In Part 2 you learned about branching and integrating changes in Git.

You learned what a diff is, and the difference between a diff and a patch. You also learned how the output of git diff is constructed.

Then, you got an extensive overview of merging with Git, specifically understanding the three-way merge algorithm. You understood when merging conflicts occur, when Git can resolve them automatically, and how to resolve them manually when needed.

You saw that git rebase is powerful - but also that it is quite simple once you understand what it does. You understood the differences between merging and rebasing, and when you should use each.

In Part 3 you learned how to undo changes in Git - especially when things go wrong. You learned how to use a bunch of tools, like git reset, git commit --amend, git revert, git reflog (and git log -g).

In Part 4 you acquired additional powerful tools, like different switches of git log, git bisect, git cherry-pick, git revert and git add -p.

Wow, you should be proud of yourself!

A Message From Me to You

Indeed, this was fun, but all things must pass. You finished reading this book, but this doesn't mean your learning journey ends here.

What you have acquired, more than any specific tool, is intuition and understanding of how Git operates, and how to think about various operations in Git. Keep researching, reading, and using Git. I am sure you will be able to teach me something new, and by all means - please do.

If you liked this book, please share it with more people.

If you want to read more of my Git articles and handbooks, here they are:

Acknowledgements

Many people helped make this book the best it can be. Among them, I was lucky to have many beta readers that provided me with feedback so that I can improve the book. Specifically, I would like to thank Jason S. Shapiro, Anna Łapińska, C. Bruce Hilbert, and Jonathon McKitrick for their thorough reviews.

Abbey Rennemeyer has been a wonderful editor. After she has reviewed my posts for freeCodeCamp for over three years, it was clear that I would like to ask her to be the editor of this book as well. She helped me improve the book in many ways, and I am grateful for her help.

Quincy Larson founded the amazing community at freeCodeCamp, motivated me throughout emails and face to face discussions. I thank him for starting this incredible community, and for his friendship.

Estefania Cassingena Navone designed the cover of this book. I am grateful for her professional work and her patience with my perfectionism and requests.

Daphne Gray-Grant's website, "Publication Coach", has provided me with inspiring as well as technical advice that has greatly helped me with my writing process.

If You Wish to Support This Book

If you would like to support this book, you are welcome to buy the Paperback version, an E-Book version, or buy me a coffee. Thank you!

Contact Me

This book has been created to help you and people like you learn, understand Git, and apply their knowledge in real life.

Right from the beginning, I asked for feedback and was lucky to receive it from great people (mentioned in the Acknowledgements) to make sure the book achieves these goals. If you liked something about this book, felt that something was missing or needed improvement - I would love to hear from you. Please reach out at gitting.things@gmail.com.

Thank you for learning and allowing me to be a part of your journey.

Omer Rosenbaum

Appendixes

Additional References - By Part

(Note - this is a short list. You can find a longer list of references on the E-Book or printed version.)

Part 1

Git Internals YouTube playlist - by Brief:
https://www.youtube.com/playlist?list=PL9lx0DXCC4BNUby5H58y6s2TQVLadV8v7
Tim Berglund's lecture - "Git From the Bits Up":
https://www.youtube.com/watch?v=MYP56QJpDr4
as promised, docs: Git for the confused:
https://www.gelato.unsw.edu.au/archives/git/0512/13748.html

Part 2

Diffs and Patches

Git Diffs algorithms:

https://en.wikipedia.org/wiki/Diff

The most default diff algorithm in Git is Myers:

Git Merge

Git Rebase

Part 3

About the Author

Omer Rosenbaum is Swimm’s Chief Technology Officer. He's the author of the Brief YouTube Channel. He's also a cyber training expert and founder of Checkpoint Security Academy. He's the author of Computer Networks (in Hebrew). You can find him on Twitter.

The Git Rebase Handbook – A Definitive Guide to Rebasing

Omer Rosenbaum — Mon, 03 Jul 2023 13:56:34 +0000

One of the most powerful tools a developer can have in their toolbox is git rebase. Yet it is notorious for being complex and misunderstood.

The truth is, if you understand what it actually does, git rebase is a very elegant, and straightforward tool to achieve so many different things in Git.

In previous posts, you understood what Git diffs are, what a merge is, and how Git resolves merge conflicts. In this post, you will understand what Git rebase is, why it's different from merge, and how to rebase with confidence 💪🏻

Notes before we start

I also created a video covering the contents of this post. If you wish to watch alongside reading, you can find it here.
If you want to play around with the repository I used and try out the commands for yourself, you can get the repo here.
I am working on a book about Git! Are you interested in reading the initial versions and providing feedback? Send me an email: gitting.things@gmail.com

OK, are you ready?

Short Recap - What is Git Merge? 🤔

Under the hood, git rebase and git merge are very, very different things. Then why do people compare them all the time?

The reason is their usage. When working with Git, we usually work in different branches and introduce changes to those branches.

In a previous tutorial, I gave an example where John and Paul (of the Beatles) were co-authoring a new song. They started from the main branch, and then each diverged, modified the lyrics and committed their changes.

Then, the two wanted to integrate their changes, which is something that happens very frequently when working with Git.

_A diverging history - paul_branch and john_branch diverged from main (Source: Brief)_

There are two main ways to integrate changes introduced in different branches in Git, or in other words, different commits and commit histories. These are merge and rebase.

In a previous tutorial, we got to know git merge pretty well. We saw that when performing a merge, we create a merge commit – where the contents of this commit are a combination of the two branches, and it also has two parents, one in each branch.

So, say you are on the branch john_branch (assuming the history depicted in the drawing above), and you run git merge paul_branch. You will get to this state – where on john_branch, there is a new commit with two parents. The first one will be the commit on john_branch branch where HEAD was pointing to before performing the merge, in this case - "Commit 6". The second will be the commit pointed to by paul_branch, "Commit 9".

_The result of running git merge paul_branch: a new Merge Commit with two parents (Source: Brief)_

Look again at the history graph: you created a diverged history. You can actually see where it branched and where it merged again.

So when using git merge, you do not rewrite history – but rather, you add a commit to the existing history. And specifically, a commit that creates a diverged history.

How is `git rebase` Different than `git merge`? 🤔

When using git rebase, something different happens. 🥁

So here, you use rebase to take the changes that were committed on one branch – Paul's branch – and replay them on a different branch, john_branch.

_The result of running git rebase john_branch: the commits on paul_branch were "replayed" on top of john_branch (Source: Brief)_

Wait, what does that mean? 🤔

We will now take this bit by bit to make sure you fully understand what's happening under the hood 😎

`cherry-pick` as a Basis for Rebase

It is useful to think of rebase as performing git cherry-pick – a command takes a commit, computes the patch this commit introduces by computing the difference between the parent's commit and the commit itself, and then cherry-pick "replays" this difference.

Let's do this manually.

If we look at the difference introduced by "Commit 5" by performing git diff main :

Running git diff to observe the patch introduced by "Commit 5" (Source: Brief)

(If you want to play around with the repository I used and try out the commands for yourself, you can get the repo here).

You can see that in this commit, John started working on a song called "Lucy in the Sky with Diamonds":

The output of git diff - the patch introduced by "Commit 5" (Source: Brief)

As a reminder, you can also use the command git show to get the same output:

git show

Now, if you cherry-pick this commit, you will introduce this change specifically, on the active branch. Switch to main first:

git checkout main (or git switch main)

And create another branch, just to be clear:

git checkout -b my_branch (or git switch -c my_branch)

_Creating my_branch that branches from main (Source: Brief)_

And cherry-pick this commit:

git cherry-pick

Using cherry-pick to apply the changes introduced in "Commit 5" onto main (Source: Brief)

Consider the log (output of git lol):

The output of git lol (Source: Brief)

(git lol is an alias I added to Git to visibly see the history in a graphical manner. You can find it here).

It seems like you copy-pasted "Commit 5". Remember that even though it has the same commit message, and introduces the same changes, and even points to the same tree object as the original "Commit 5" in this case – it is still a different commit object, as it was created with a different timestamp.

Looking at the changes, using git show HEAD:

The output of git show HEAD (Source: Brief)

They are the same as "Commit 5"'s.

And of course, if you look at the file (say, by using nano lucy_in_the_sky_with_diamonds.md), it will be in the same state as it has been after the original "Commit 5".

Cool! 😎

OK, you can now remove the new branch so it doesn't appear on your history every time:

git checkout main
git branch -D my_branch

Beyond `cherry-pick` – How to Use `git rebase`

You can look at git rebase as a way to perform multiple cherry-picks one after the other – that is, to "replay" multiple commits. This is not the only thing you can do with rebase, but it's a good starting point for our explanation.

It's time to play with git rebase! 👏🏻👏🏻

Before, you merged paul_branch into john_branch. What would happen if you rebased paul_branch on top of john_branch? You would get a very different history.

In essence, it would seem as if we took the changes introduced in the commits on paul_branch, and replayed them on john_branch. The result would be a linear history.

To understand the process, I will provide the high level view, and then dive deeper into each step. The process of rebasing one branch on top of another branch is as follows:

Find the common ancestor.
Identify the commits to be "replayed".
For every commit X, compute diff(parent(X), X), and store it as a patch(X).
Move HEAD to the new base.
Apply the generated patches in order on the target branch. Each time, create a new commit object with the new state.

The process of making new commits with the same changesets as existing ones is also called "replaying" those commits, a term we have already used.

Time to Get Hands-On with Rebase🙌🏻

Start from Paul's branch:

git checkout paul_branch

This is the history:

Commit history before performing git rebase (Source: Brief)

And now, to the exciting part:

git rebase john_branch

And observe the history:

The history after rebasing (Source: Brief)

( gg is an alias for an external tool I introduced in the video).

So whereas with git merge you added to the history, with git rebase you rewrite history. You create new commit objects. In addition, the result is a linear history graph – rather than a diverging graph.

The history after rebasing (Source: Brief)

In essence, we "copied" the commits that were on paul_branch and introduced after "Commit 4", and "pasted" them on top of john_branch.

The command is called "rebase", because it changes the base commit of the branch it's run from. That is, in your case, before running git rebase, the base of paul_branch was "Commit 4" – as this is where the branch was "born" (from main). With rebase, you asked Git to give it another base – that is, pretend as if it had been born from "Commit 6".

To do that, Git took what used to be "Commit 7", and "replayed" the changes introduced in this commit onto "Commit 6", and then created a new commit object. This object differs from the original "Commit 7" in three aspects:

It has a different timestamp.
It has a different parent commit – "Commit 6" rather than "Commit 4".
The tree object it is pointing to is different - as the changes were introduced to the tree pointed to by "Commit 6", and not the tree pointed to by "Commit 4".

Notice the last commit here, "Commit 9'". The snapshot it represents (that is, the tree that it points to) is exactly the same tree you would get by merging the two branches. The state of the files in your Git repository would be the same as if you used git merge. It's only the history that is different, and the commit objects of course.

Now, you can simply use:

git checkout main
git merge paul_branch

Hm.... What would happen if you ran this last command? 🤔 Consider the commit history again, after checking out main:

The history after rebasing and checking out main (Source: Brief)

What would it mean to merge main and paul_branch?

Indeed, Git can simply perform a fast-forward merge, as the history is completely linear (if you need a reminder about fast forward merges, check out this post). As a result, main and paul_branch now point to the same commit:

The result of a fast-forward merge (Source: Brief)

Advanced Rebasing in Git💪🏻

Now that you understand the basics of rebase, it is time to consider more advanced cases, where additional switches and arguments to the rebase command will come in handy.

In the previous example, when you only said rebase (without additional switches), Git replayed all the commits from the common ancestor to the tip of the current branch.

But rebase is a super-power, it's an almighty command capable of…well, rewriting history. And it can come in handy if you want to modify history to make it your own.

Undo the last merge by making main point to "Commit 4" again:

git reset -–hard 4>

"Undoing" the last merge operation (Source: Brief)

And undo the rebasing by using:

git checkout paul_branch
git reset -–hard 9>

"Undoing" the rebase operation (Source: Brief)

Notice that you got to exactly the same history you used to have:

Visualizing the history after "undoing" the rebase operation (Source: Brief)

Again, to be clear, "Commit 9" doesn't just disappear when it's not reachable from the current HEAD. Rather, it's still stored in the object database. And as you used git reset now to change HEAD to point to this commit, you were able to retrieve it, and also its parent commits since they are also stored in the database. Pretty cool, huh? 😎

OK, quickly view the changes that Paul introduced:

git show HEAD

git show HEAD shows the patch introduced by "Commit 9" (Source: Brief)

Keep going backwards in the commit graph:

git show HEAD~

git show HEAD~ (same as git show HEAD~1) shows the patch introduced by "Commit 8" (Source: Brief)

And one commit further:

git show HEAD~2

git show HEAD~2 shows the patch introduced by "Commit 7" (Source: Brief)

So, these changes are nice, but perhaps Paul doesn't want this kind of history. Rather, he wants it to seem as if he introduced the changes in "Commit 7" and "Commit 8" as a single commit.

For that, you can use an interactive rebase. To do that, we add the -i (or --interactive) switch to the rebase command:

git rebase -i

Or, since main is pointing to "Commit 4", we can simply run:

git rebase -i main

For every commit that is replayed, Git asks us what we'd like to do with it:

git rebase -i main prompts you to select what to do with each commit (Source: Brief)

In this context it's useful to think of a commit as a patch. That is, "Commit 7" as in "the patch that "Commit 7" introduced on top of its parent".

One option is to use pick. This is the default behavior, which tells Git to replay the changes introduced in this commit. In this case, if you just leave it as is – and pick all commits – you will get the same history, and Git won't even create new commit objects.

Another option is squash. A squashed commit will have its contents "folded" into the contents of the commit preceding it. So in our case, Paul would like to squash "Commit 8" into "Commit 7":

Squashing "Commit 8" into "Commit 7" (Source: Brief)

As you can see, git rebase -i provides additional options, but we won't go into all of them in this post. If you allow the rebase to run, you will get prompted to select a commit message for the newly created commit (that is, the one that introduced the changes of both "Commit 7" and "Commit 8"):

Providing the commit message: Commits 7+8 (Source: Brief)

And look at the history:

The history after the interactive rebase (Source: Brief)

Exactly as we wanted! We have on paul_branch "Commit 9" (of course, it's a different object than the original "Commit 9"). This points to "Commits 7+8", which is a single commit introducing the changes of both the original "Commit 7" and the original "Commit 8". This commit's parent is "Commit 4", where main is pointing to. You have john_branch.

The history after the interactive rebase - visualized (Source: Brief)

Oh wow, isn't that cool? 😎

How to Use the `--onto` Switch of `git rebase`

Let's consider one more example. Get to main again:

git checkout main

And delete the pointers to paul_branch and john_branch so you don't see them in the commit graph anymore:

git branch -D paul_branch
git branch -D john_branch

And now branch from main to a new branch:

git checkout -b new_branch

_Creating new_branch that diverges from main (Source: Brief)_

_A clean history with new_branch that diverges from main (Source: Brief)_

Now, add a few changes here and commit them:

nano code.py

_Adding the function new_branch to code.py (Source: Brief)_

git add code.py
git commit -m "Commit 10"

Get back to main:

git checkout main

And introduce another change:

Added a docstring at the beginning of the file (Source: Brief)

Time to stage and commit these changes:

git add code.py
git commit -m "Commit 11"

And yet another change:

Added @Author to the docstring (Source: Brief)

Commit this change as well:

git add code.py
git commit -m "Commit 12"

Oh wait, now I realize that I wanted you to make the changes introduced in "Commit 11" as a part of the new_branch. Ugh. What can you do? 🤔

Consider the history:

The history after introducing "Commit 12" (Source: Brief)

What I want is, instead of having "Commit 10" reside only on the main branch, I want it to be on both the main branch as well as the new_branch. Visually, I would want to move it down the graph here:

Visually, I want you to "push" "Commit 10" (Source: Brief)

Can you see where I am going? 😇

Well, as we understand, rebase allows us to basically replay the changes introduced in new_branch, those introduced in "Commit 10", as if they had been originally conducted on "Commit 11", rather than "Commit 4".

To do that, you can use other arguments of git rebase. You'd tell Git that you want to take all the history introduced between the common ancestor of main and new_branch, which is "Commit 4", and have the new base for that history be "Commit 11". To do that, use:

git rebase -–onto  main new_branch

The history before and after the rebase, "Commit 10" has been "pushed" (Source: Brief)

And look at our beautiful history! 😍

The history before and after the rebase, "Commit 10" has been "pushed" (Source: Brief)

Let's consider another case.

Say I started working on a branch, and by mistake I started working from feature_branch_1, rather than from main.

So to emulate this, create feature_branch_1:

git checkout main
git checkout -b feature_branch_1

And erase new_branch so you don't see it in the graph anymore:

git branch -D new_branch

Create a simple Python file called 1.py:

A new file, 1.py, with print('Hello world!') (Source: Brief)

Stage and commit this file:

git add 1.py
git commit -m  "Commit 13"

Now branched out (by mistake) from feature_branch_1:

git checkout -b feature_branch_2

And create another file, 2.py:

Creating 2.py (Source: Brief)

Stage and commit this file as well:

git add 2.py
git commit -m  "Commit 14"

And introduce some more code to 2.py:

Modifying 2.py (Source: Brief)

Stage and commit these changes too:

git add 2.py
git commit -m  "Commit 15"

So far you should have this history:

The history after introducing "Commit 15" (Source: Brief)

Get back to feature_branch_1 and edit 1.py:

git checkout feature_branch_1

Modifying 1.py (Source: Brief)

Now stage and commit:

git add 1.py
git commit -m  "Commit 16"

Your history should look like this:

The history after introducing "Commit 16" (Source: Brief)

Say now you realize, you've made a mistake. You actually wanted feature_branch_2 to be born from the main branch, rather than from feature_branch_1.

How can you achieve that? 🤔

Try to think about it given the history graph and what you've learned about the --onto flag for the rebase command.

Well, you want to "replace" the parent of your first commit on feature_branch_2, which is "Commit 14", to be on top of main branch, in this case, "Commit 12", rather than the beginning of feature_branch_1, in this case, "Commit 13". So again, you will be creating a new base, this time for the first commit on feature_branch_2.

You want to move around "Commit 14" and "Commit 15" (Source: Brief)

How would you do that?

First, switch to feature_branch_2:

git checkout feature_branch_2

And now you can use:

git rebase -–onto main

As a result, you have feature_branch_2 based on main rather than feature_branch_1:

The commit history after performing rebase (Source: Brief)

The syntax is of the command is:

git rebase --onto  <old_parent>

How to rebase on a single branch

You can also use git rebase while looking at a history of a single branch.

Let's see if you can help me here.

Say I worked from feature_branch_2, and specifically edited the file code.py. I started by changing all strings to be wrapped by double quotes rather than single quotes:

Changing ' into " in code.py (Source: Brief)

Then, I staged and committed:

git add code.py
git commit -m "Commit 17"

I then decided to add a new function at the beginning of the file:

_Adding the function another_feature (Source: Brief)_

Again, I staged and committed:

git add code.py
git commit -m "Commit 18"

And now I realized I actually forgot to change the single quotes to double quotes wrapping the __main__ (as you might have noticed), so I did that too:

Changing '__main__' into "__main__" (Source: Brief)

Of course, I staged and committed this change:

git add code.py
git commit -m "Commit 19"

Now, consider the history:

The commit history after introducing "Commit 19" (Source: Brief)

Intuitively, I want to edit the history here:

These are the commits I want to edit (Source: Brief)

So, what would you do?

You are right! 👏🏻

I can rebase the history from "Commit 17" to "Commit 19", on top of "Commit 15". To do that:

git rebase --interactive --onto  <SHA_OF_COMMIT_15>

Notice I specified "Commit 15" as the beginning of the range of commits, excluding this commit. And I didn't need to explicitly specify HEAD as the last parameter.

Using rebase --onto on a single branch (Source: Brief)

After following your advice and running the rebase command (thanks! 😇) I get the following screen:

Interactive rebase (Source: Brief)

So what would I do? I want to put "Commit 19" before "Commit 18", so it comes right after "Commit 17". I can go further and squash them together, like so:

Interactive rebase - changing the order of commit and squashing (Source: Brief)

Now when I get prompted for a commit message, I can provide the message "Commit 17+19":

Providing a commit message (Source: Brief)

And now, see our beautiful history:

The resulting history (Source: Brief)

Thanks again! 🙌🏻

More Rebase Use Cases + More Practice

By now I hope you feel comfortable with the syntax of rebase. The best way to actually understand it is to consider various cases and figure out how to solve them yourself.

With the upcoming use cases, I strongly suggest you stop reading after I've introduced each use case, and then try to solve it on your own.

How to Exclude Commits

Say you have this history on another repo:

Another commit history (Source: Brief)

Before playing around with it, store a tag to "Commit F" so you can get back to it later:

git tag original_commit_f

Now, you actually don't want the changes in "Commit C" and "Commit D" to be included. You could use an interactive rebase like before and remove their changes. Or, could can use again git rebase -–onto. How would you use --onto in order to "remove" these two commits?

You can rebase HEAD on top of "Commit B", where the old parent was actually "Commit D", and now it should be "Commit B". Consider the history again:

The history again (Source: Brief)

Rebasing so that "Commit B" is the base of "Commit E", means "moving" both "Commit E" and "Commit F", and giving them another base – "Commit B". Can you come up with the command yourself?

git rebase --onto  <SHA_OF_COMMIT_D> HEAD

Notice that using the syntax above would not move main to point to the new commit, so the result is a "detached" HEAD. If you use gg or another tool that displays the history reachable from branches it might confuse you:

Rebasing with --onto results in a detached HEAD (Source: Brief)

But if you simply use git log (or my alias git lol), you will see the desired history:

The resulting history (Source: Brief)

I don't know about you, but these kinds of things make me really happy. 😊😇

By the way, you could omit HEAD from the previous command as this is the default value for the third parameter. So just using:

git rebase --onto  <SHA_OF_COMMIT_D>

Would have the same effect. The last parameter actually tells Git where the end of the current sequence of commits to rebase is. So the syntax of git rebase --onto with three arguments is:

git rebase --onto

How to move commits across branches

So let's say we get to the same history as before:

git checkout original_commit_f

And now I want only "Commit E", to be on a branch based on "Commit B". That is, I want to have a new branch, branching from "Commit B", with only "Commit E".

The current history, considering "Commit E" (Source: Brief)

So, what does this mean in terms of rebase? Consider the image above. What commit (or commits) should I rebase, and which commit would be the new base?

I know I can count on you here 😉

What I want is to take "Commit E", and this commit only, and change its base to be "Commit B". In other words, to replay the changes introduced in "Commit E" onto "Commit B".

Can you apply that logic to the syntax of git rebase?

Here it is (this time I'm writing instead of , for brevity):

git rebase –-onto

Now the history looks like so:

The history after rebase (Source: Brief)

Awesome!

A Note About Conflicts

Note that when performing a rebase, you may run into conflicts just as when merging. You may have conflicts because when rebasing, you are trying to apply patches on a different base, perhaps where the patches do not apply.

For example, consider the previous repository again, and specifically, consider the change introduced in "Commit 12", pointed to by main:

git show main

The patch introduced in "Commit 12" (Source: Brief)

I already covered the format of git diff in detail in a previous post, but as a quick reminder, this commit instructs Git to add a line after the two lines of context:

This is a sample file


And before these three lines of context:

def new_feature():
  print('new feature')

Say you are trying to rebase "Commit 12" onto another commit. If, for some reason, these context lines don't exist as they do in the patch on the commit you are rebasing onto, then you will have a conflict. To learn more about conflicts and how to resolve them, see this guide.

Zooming Out for the Big Picture

Comparing rebase and merge (Source: Brief)

In the beginning of this guide, I started by mentioning the similarity between git merge and git rebase: both are used to integrate changes introduced in different histories.

But, as you now know, they are very different in how they operate. While merging results in a diverged history, rebasing results in a linear history. Conflicts are possible in both cases. And there is one more column described in the table above that requires some close attention.

Now that you know what "Git rebase" is, and how to use interactive rebase or rebase --onto, as I hope you agree, git rebase is a super powerful tool. Yet, it has one huge drawback when compared with merging.

Git rebase changes the history.

This means that you should not rebase commits that exist outside your local copy of the repository, and that other people may have based their commits on.

In other words, if the only commits in question are those you created locally – go ahead, use rebase, go wild.

But if the commits have been pushed, this can lead to a huge problem – as someone else may rely on these commits, that you later overwrite, and then you and they will have different versions of the repository.

This is unlike merge which, as we have seen, does not modify history.

For example, consider the last case where we rebased and resulted in this history:

The history after rebase (Source: Brief)

Now, assume that I have already pushed this branch to the remote. And after I had pushed the branch, another developer pulled it and branched out from "Commit C". The other developer didn't know that meanwhile, I was locally rebasing my branch, and would later push it again.

This results in an inconsistency: the other developer works from a commit that is no longer available on my copy of the repository.

I will not elaborate on what exactly this causes in this guide, as my main message is that you should definitely avoid such cases. If you're interested in what would actually happen, I'll leave a link to a useful resource below. For now, let's summarize what we have covered.

Recap

In this tutorial, you learned about git rebase, a super-powerful tool to rewrite history in Git. You considered a few use cases where git rebase can be helpful, and how to use it with one, two, or three parameters, with and without the --onto switch.

I hope I was able to convince you that git rebase is powerful – but also that it is quite simple once you get the gist. It is a tool to "copy-paste" commits (or, more accurately, patches). And it's a useful tool to have under your belt.

Additional References

About the Author

Omer Rosenbaum is Swimm’s Chief Technology Officer. He's the author of the Brief YouTube Channel. He's also a cyber training expert and founder of Checkpoint Security Academy. He's the author of Computer Networks (in Hebrew). You can find him on Twitter.

The Git Merge Handbook – Definitive Guide to Merging in Git

Omer Rosenbaum — Thu, 27 Apr 2023 17:07:19 +0000

By reading this post, you are going to really understand git merge, one of the most common operations you'll perform in your Git repositories.

Notes before we start

I also created two videos covering the contents of this post. If you wish to watch alongside reading, you can find them here (Part 1, Part 2).
I am working on a book about Git! Are you interested in reading the initial versions and providing feedback? Send me an email: gitting.things@gmail.com

OK, are you ready?

What is a Merge in Git?
Time to Get Hands-on 🙌🏻
Time For a More Advanced Case
Quick recap on a three-way merge
Moving on 👣
More Advanced Git Merge Cases
How Git's 3-way Merge Algorithm Works
How to Resolve Merge Conflicts
How to Use VS Code to Resolve Conflicts
One More Powerful Tool 🪛
Recap

What is a Merge in Git?

Merging is the process of combining the recent changes from several branches into a single new commit that will be on all those branches.

OK, let's take this bit by bit.

That is, if you consider this commit graph:

_Commit graph with two pointers (Source: Brief)_

You see the branch feature_1, which points to a commit with the SHA-1 value of ba0d2. Of course, as in other posts, I only write the first 5 digits of the SHA-1 value.

Notice that commit 54a9d is also on this branch, as it is the parent commit of ba0d2. So if you start from the pointer of feature_1, you get to ba0d2, which then points to 54a9d.

When you merge with Git, you merge commits. Almost always, we merge two commits by referring to them with the branch names that point to them. Thus we say we "merge branches" – though under the hood, we actually merge commits.

Time to Get Hands-on 🙌🏻

OK, so let's say I have this simple repository here, with a branch called main, and a few commits with the commit messages of "Commit 1", "Commit 2" and "Commit 3":

_A simple repository with three commits (Source: Brief)_

Next, create a feature branch by typing git branch new_feature:

_Creating a new branch with git branch (Source: Brief)_

And switch HEAD to point to this new branch, by using git checkout new_feature. You can look at the outcome by using git log:

_The output of git log after using git checkout new_feature (Source: Brief)_

As a reminder, you could also write git checkout -b new_feature, which would both create a new branch and change HEAD to point to this new branch.

If you need a reminder about branches and how they're implemented under the hood, please check out a previous post on the subject. Yes, check out. Pun intended 😇

Now, on the new_feature branch, implement a new feature. In this example I will edit an existing file that looks like this before the edit:

_code.py before editing it (Source: Brief)_

And I will now edit it to include a new function:

_Implementing new_feature (Source: Brief)_

And thankfully, this is not a programming tutorial, so this function is legit 😇
Next, stage and commit this change:

_Committing the changes to "Commit 4" (Source: Brief)_

Looking at the history, you have the branch new_feature, now pointing to "Commit 4", which points to its parent, "Commit 3". The branch main is also pointing to "Commit 3".

Check out the main branch (with git checkout main), and perform the merge by using git merge new_feature:

_Merging new_feature into main (Source: Brief)_

Since new_feature never really diverged from main, Git could just perform a fast-forward merge. So what happened here? Consider the history:

_The result of a fast-forward merge (Source: Brief)_

Even though you used git merge, there was no actual merging here. Actually, Git did something very simple – it reset the main branch to point to the same commit as the branch new_feature.

In case you don't want that to happen, but rather you want Git to really perform a merge, you could either change Git's configuration, or run the merge command with the --no-ff flag.

First, undo the last commit:

git reset --hard HEAD~1

If this way of using reset is not clear to you, feel free to check out a post where I covered git reset in depth. It is not crucial for this introduction of merge, though. For now, it's important to understand that it basically undoes the merge operation.

Just to clarify, now if you checked out new_feature again:

git checkout new_feature

The history would look just like before the merge:

_The history after using git reset --hard HEAD~1 (Source: Brief)_

Next, perform the merge with the --no-fast-forward flag (--no-ff for short):

git checkout main
git merge new_feature --no-ff

Now, if we look at the history using git lol:

_History after merging with the --no-ff flag (Source: Brief)_

(git lol is an alias I added to Git to visibly see the history in a graphical manner. You can find it here).

Considering this history, you can see Git created a new commit, a merge commit.

If you consider this commit a bit closer:

git log -n1

_The merge commit has two parents (Source: Brief)_

You will see that this commit actually has two parents – "Commit 4", which was the commit that new_feature pointed to when you ran git merge, and "Commit 3", which was the commit that main pointed to. So a merge commit has two parents: the two commits it merged.

The merge commit shows us the concept of merge quite well. Git takes two commits, usually referenced by two different branches, and merges them together.

After the merge, as you started the process from main, you are still on main, and the history from new_feature has been merged into this branch. Since you started with main, then "Commit 3", which main pointed to, is the first parent of the merge commit, whereas "Commit 4", which you merged into main, is the second parent of the merge commit.

This was a super simple case, where the branches you merged didn't diverge at all.

By the way, you can use git merge to merge more than two commits – actually, any number of commits. This is rarely done and I don't see a good reason to elaborate on it here.

Another way to think of git merge is by joining two or more development histories together. That is, when you merge, you incorporate changes from the named commits, since the time their histories diverged from the current branch, into the current branch. I used the term branch here, but I am stressing this again – we are actually merging commits.

Time For a More Advanced Case 💪🏻

Time to consider a more advanced case, which is probably the most common case where we use git merge explicitly – where you need to merge branches that did diverge from one another.

Assume we have two people working on this repo now, John and Paul.

John created a branch:

git checkout -b john_branch

_A new branch, john_branch (Source: Brief)_

git add lucy_in_the_sky_with_diamonds.md
git commit -m "Commit 5"

While John was working on this song, Paul was also writing, on another branch. Paul had started from main:

git checkout main

And created his own branch:

git checkout -b paul_branch

And Paul wrote his song into a file:

nano penny_lane.md

And committed it:

git add penny_lane.md
git commit -m "Commit 6"

So now our history looks like this – where we have two different branches, branching out from main, with different histories.

_The output of git lol shows the history after John and Paul committed (Source: Brief)_

John is happy with his branch (that is, his song), so he decides to merge it into the main branch:

git checkout main
git merge john_branch

Actually, this is a fast-forward merge, as we have learned before. You can validate that by looking at the history (using git lol, for example):

_Merging john_branch into main results in a fast-forwrad merge (Source: Brief)_

At this point, Paul also wants to merge his branch into main, but now a fast-forward merge is no longer relevant – there are two different histories here: the history of main's and that of paul_branch's. It's not that paul_branch only adds commits on top of main branch or vice versa.

Now things get interesting. 😎😎

First, let Git do the hard work for you. After that, we will understand what's actually happening under the hood.

git merge paul_branch

Consider the history now:

_When you merge paul_branch, you get a new merge commit (Source: Brief)_

What you have is a new commit, with two parents – "Commit 5" and "Commit 6". In the working dir, you can see that both John's song as well as Paul's song are there: ls

_The working dir after the merge (Source: Brief)_

Nice, Git really did merge the changes for us. But how does that happen?

Undo this last commit:

git reset --hard HEAD~

How to perform a three-way merge in Git

It's time to understand what's really happening under the hood. 😎

What Git has done here is it called a 3-way merge. In outlining the process of a 3-way merge, I will use the term "branch" for simplicity, but you should remember you could also merge two (or more) commits that are not referenced by a branch.

The 3-way merge process includes these stages:

Second, Git calculates two diffs – one diff from the merge base to the first branch, and another diff from the merge base to the second branch. Git generates patches based on those diffs.

Third, Git applies both patches to the merge base using a 3-way merge algorithm. The result is the state of the new, merge commit.

_The three steps of the 3-way merge algorithm: (1) locate the common ancestor; (2) calculate diffs from the merge base to the first branch, and from the merge base to the second branch; (3) apply both patches together (Source: Brief)_

So, back to our example.

In the first step, Git looks from both branches – main and paul_branch – and traverses the history to find the first commit that is reachable from both. In this case, this would be...which commit?

Correct, "Commit 4".

If you are not sure, you can always ask Git directly:

git merge-base main paul_branch

In the second step, Git calculates the diffs. So it first calculates the diff between "Commit 4" and "Commit 5":

git diff 4f90a62 4683aef

(The SHA-1 values will be different on your machine)

_The diff between "Commit 4" and "Commit 5" (Source: Brief)_

If you don't feel comfortable with the output of git diff, please read the previous post where I described it in detail.

You can store that diff to a file:

git diff 4f90a62 4683aef > john_branch_diff.patch

Next, Git calculates the diff between "Commit 4" and "Commit 6":

git diff 4f90a62 c5e4951

_The diff between "Commit 4" and "Commit 6" (Source: Brief)_

Write this one to a file as well:

git diff 4f90a62 c5e4951 > paul_branch_diff.patch

Now Git applies those patches on the merge base.

First, try that out directly – just apply the patches (I will walk you through it in a moment). This is not what Git really does under the hood, but it will help you gain a better understanding of why Git needs to do something different.

Checkout the merge base first, that is, "Commit 4":

git checkout 4f90a62

And apply John's patch first:

git apply -–index john_branch_diff.patch

Notice that for now there is no merge commit. git apply updates the working dir as well as the index, as we used the --index switch.

You can observe the status using git status:

_Applying John's patch on "Commit 4" (Source: Brief)_

So now John's new song is incorporated into the index. Apply the other patch:

git apply -–index paul_branch_diff.patch

As a result, the index contains changes from both branches.

Now it's time to commit your merge. Since the porcelain command git commit always generates a commit with a single parent, you would need the underlying plumbing command – git commit-tree.

If you need a reminder about porcelain vs plumbing commands, check out the post where I explained these terms, and created an entire repo from scratch.

Remember that every Git commit object points to a single tree. So you need to record the contents of the index in a tree:

git write-tree

Now you get the SHA-1 value of the created tree, and you can create a commit object using git commit-tree:

git commit-tree  -p  -p  -m "Merge commit!"

_Creating a merge commit (Source: Brief)_

Great, so you have created a commit object 💪🏻

Recall that git merge also changes HEAD to point to the new merge commit object. So you can simply do the same: git reset –-hard db315a

If you look at the history now:

_The history after creating a merge commit and resetting HEAD (Source: Brief)_

You can see that you've reached the same result as the merge done by Git, with the exception of the timestamp and thus the SHA-1 value, of course.

So you got to merge both the contents of the two commits – that is, the state of the files, and also the history of those commits – by creating a merge commit that points to both histories.

In this simple case, you could actually just apply the patches using git apply, and everything worked quite well.

Quick recap on a three-way merge

So to quickly recap, on a three-way merge, Git:

First, locates the merge base – the common ancestor of the two branches. That is, the first commit that is reachable from both branches.
Second, Git calculates two diffs – one diff from the merge base to the first branch, and another diff from the merge base to the second branch.
Third, Git applies both patches to the merge base, using a 3-way merge algorithm. I haven't explained the 3-way merge yet, but I will elaborate on that later. The result is the state of the new, merge commit.

You can also understand why it's called a "3-way merge": Git merges three different states – that of the first branch, that of the second branch, and their common ancestor. In our previous example, main, paul_branch, and Commit 4.

This is unlike, say, the fast-forward examples we saw before. The fast-forward examples are actually a case of a two-way merge, as Git only compares two states – for example, where main pointed to, and where john_branch pointed to.

Moving on 👣

Still, this was a simple case of a 3-way merge. John and Paul created different songs, so each of them touched a different file. It was pretty straightforward to execute the merge.

What about more interesting cases?

Let's assume that now John and Paul are co-authoring a new song.

So, John checkedout main branch and started writing the song:

git checkout main

_John's new song (Source: Brief)_

He staged and committed it ("Commit 7"):

_John's new song is committed (Source: Brief)_

Now, Paul branches:

git checkout -b paul_branch_2

And edits the song, adding another verse:

_Paul added a new verse (Source: Brief)_

Of course, in the original song, we don't have the title "Paul's Verse", but I'll add it here for simplicity.

Paul stages and commits the changes:

git add a_day_in_the_life.md
git commit -m "Commit 8"

John also branches out from main and adds a few last lines: ```git checkout main git checkout -b john_branch_2



![Image](https://www.freecodecamp.org/news/content/images/2023/04/image-256.png)
_Paul committed, and now it's John's turn again (Source: [Brief](https://www.youtube.com/watch?v=ZS4stBVdDII&ab_channel=Brief))_

![Image](https://www.freecodecamp.org/news/content/images/2023/04/image-257.png)
_John added a few lines (Source: [Brief](https://www.youtube.com/watch?v=ZS4stBVdDII&ab_channel=Brief))_

And he stages and commits his changes too ("Commit 9"):

![Image](https://www.freecodecamp.org/news/content/images/2023/04/image-258.png)
_John committed his changes (Source: [Brief](https://www.youtube.com/watch?v=ZS4stBVdDII&ab_channel=Brief))_

This is the resulting history:

![Image](https://www.freecodecamp.org/news/content/images/2023/04/image-350.png)
_The history after John's last commit (Source: [Brief](https://www.youtube.com/watch?v=ZS4stBVdDII&ab_channel=Brief))_

So, both Paul and John modified the same file on different branches. Will Git be successful in merging them? 🤔

Say now we don't go through `main,` but John will try to merge Paul's new branch into his branch:

git merge paul_branch_2


Wait!! 🤚🏻 Don't run this command! Why would you let Git do all the hard work? You are trying to understand the process here.

So, first, Git needs to find the merge base. Can you see which commit that would be?

Correct, it would be the last commit on `main` branch, where the two diverged.

You can verify that by using:

git merge-base john_branch_2 paul_branch_2



![Image](https://www.freecodecamp.org/news/content/images/2023/04/image-260.png)
_Finding the merge base (Source: [Brief](https://www.youtube.com/watch?v=ZS4stBVdDII&ab_channel=Brief))_

Great, now Git should compute the diffs and generate the patches. You can observe the diffs directly:

git diff main paul_branch_2



![Image](https://www.freecodecamp.org/news/content/images/2023/04/image-261.png)
_The output of `git diff main paul_branch_2` (Source: [Brief](https://www.youtube.com/watch?v=ZS4stBVdDII&ab_channel=Brief))_

Will applying this patch succeed? Well, no problem, Git has all the context lines in place.

Ask Git to apply this patch:

git diff main paul_branch_2 > paul_branch_2.patch git apply -–index paul_branch_2.patch


And this worked, no problem at all.

Now, compute the diff between John's new branch and the merge base. Notice that you haven't committed the applied changes, so `john_branch_2` still points at the same commit as before, "Commit 9":

git diff main john_branch_2


![Image](https://www.freecodecamp.org/news/content/images/2023/04/image-262.png)
_The output of `git diff main john_branch_2` (Source: [Brief](https://www.youtube.com/watch?v=ZS4stBVdDII&ab_channel=Brief))_

Will applying this diff work?

Well, indeed, yes. Notice that even though the line numbers have changed on the current version of the file, thanks to the context lines Git is able to locate where it needs to add these lines…

![Image](https://www.freecodecamp.org/news/content/images/2023/04/image-263.png)
_Git can rely on the context lines (Source: [Brief](https://www.youtube.com/watch?v=ZS4stBVdDII&ab_channel=Brief))_

Save this patch and apply it then:

git diff main john_branch_2 > john_branch_2.patch git apply –-index john_branch_2.patch


![Image](https://www.freecodecamp.org/news/content/images/2023/04/image-264.png)
_Apply Paul's patch (Source: [Brief](https://www.youtube.com/watch?v=ZS4stBVdDII&ab_channel=Brief))_

Observe the result file:

![Image](https://www.freecodecamp.org/news/content/images/2023/04/image-265.png)
_The result after applying Paul's patch (Source: [Brief](https://www.youtube.com/watch?v=ZS4stBVdDII&ab_channel=Brief))_

Cool, exactly what we wanted 👏🏻
You can now create the tree and relevant commit:

git write-tree


Don't forget to specify both parents:

git commit-tree -p paul_branch_2 -p john_branch_2 -m "Merging new changes"



![Image](https://www.freecodecamp.org/news/content/images/2023/04/image-266.png)
_Creating a merge commit (Source: [Brief](https://www.youtube.com/watch?v=ZS4stBVdDII&ab_channel=Brief))_

See how I used the branches names here? After all, they are just pointers to the commits we want.

Cool, look at the log from the new commit:

![Image](https://www.freecodecamp.org/news/content/images/2023/04/image-270.png)
_The history after creating the merge commit (Source: [Brief](https://www.youtube.com/watch?v=ZS4stBVdDII&ab_channel=Brief))_

Exactly what we wanted.

You can also let Git perform the job for you. You can simply checkout `john_branch_2`, which you haven't moved – so it still points to the same commit as it did before the merge. So all you need to do is run:

git merge paul_branch_2


Observe the resulting history:

![Image](https://www.freecodecamp.org/news/content/images/2023/04/image-271.png)
_The history after letting Git perform the merge (Source: [Brief](https://www.youtube.com/watch?v=BCNZ5Uxctuk&t=561s&ab_channel=Brief))_

Just as before, you have a merge commit pointing to "Commit 8" and "Commit 9" as its parents. "Commit 9" is the first parent since you merged into it.

But this was still quite simple… John and Paul worked on the same file, but on very different parts. You could also directly apply Paul's changes to John's branch. If you go back to John's branch before the merge:

git reset --hard HEAD~


And now apply Paul's changes:

git apply -–index paul_branch_2.patch


![Image](https://www.freecodecamp.org/news/content/images/2023/04/image-272.png)
_Applying Paul's changes directly to John's branch (Source: [Brief](https://www.youtube.com/watch?v=BCNZ5Uxctuk&t=561s&ab_channel=Brief))_

You will get the same result.

But what happens when the two branches include changes on the same files, in the same locations? 🤔

# More Advanced Git Merge Cases

What would happen if John and Paul were to coordinate a new song, and work on it together?

In this case, John creates the first version of this song in the `main` branch:

git checkout main nano everyone.md


![Image](https://www.freecodecamp.org/news/content/images/2023/04/image-273.png)
_The contents of `everyone.md` prior to the first commit (Source: [Brief](https://www.youtube.com/watch?v=BCNZ5Uxctuk&t=561s&ab_channel=Brief))_

By the way, this text is indeed taken from the version that John Lennon recorded for a demo in 1968. But this isn't an article about the Beatles, so if you're curious about the process the Beatles underwent while writing this song, you can follow the links in the appendix below.

git add everyone.md git commit -m "Commit 10"


![Image](https://www.freecodecamp.org/news/content/images/2023/04/image-275.png)
_Introducing "Commit 10" (Source: [Brief](https://www.youtube.com/watch?v=BCNZ5Uxctuk&t=561s&ab_channel=Brief))_

Now John and Paul split. Paul creates a new verse in the beginning:

git checkout -b paul_branch_3 nano everyone.md


![Image](https://www.freecodecamp.org/news/content/images/2023/04/image-276.png)
_Paul added a new verse in the beginning (Source: [Brief](https://www.youtube.com/watch?v=BCNZ5Uxctuk&t=561s&ab_channel=Brief))_

Also, while talking to John, they decided to change the word "feet" to "foot", so Paul adds this change as well.

And Paul adds and commits his changes to the repo:

git add everyone.md git commit -m "Commit 11"


You can observe Paul's changes, by comparing this branch's state to the state of branch `main`:

git diff main



![Image](https://www.freecodecamp.org/news/content/images/2023/04/image-277.png)
_The output of `git diff main` from Paul's branch (Source: [Brief](https://www.youtube.com/watch?v=BCNZ5Uxctuk&t=561s&ab_channel=Brief))_

Store this diff in a patch file:

git diff main > paul_3.patch


Now back to `main`...

git checkout main


John decides to make another change, in his own new branch:

git checkout -b john_branch_3


And he replaces the line "Everyone had the boot in" with the line "Everyone had a wet dream". In addition, John changed the word "feet" to "foot", following his talk with Paul.

Observe the diff:

git diff main


![Image](https://www.freecodecamp.org/news/content/images/2023/04/image-278.png)
_The output of `git diff main` from John's branch (Source: [Brief](https://www.youtube.com/watch?v=BCNZ5Uxctuk&t=561s&ab_channel=Brief))_

Store this output as well:

git diff main > john_3.patch


Now, stage and commit:

git add everyone.md git commit -m "Commit 12"


This is our current history:

![Image](https://www.freecodecamp.org/news/content/images/2023/04/image-351.png)
_The history after introducing "Commit 12" (Source: [Brief](https://www.youtube.com/watch?v=BCNZ5Uxctuk&t=561s&ab_channel=Brief))_

Paul told John he added a new verse, so John would like to merge Paul's changes.

Can John simply apply Paul's patch?

Consider the patch again:

git diff main paul_branch_3


![Image](https://www.freecodecamp.org/news/content/images/2023/04/image-277.png)
_The output of `git diff main paul_branch_3` (Source: [Brief](https://www.youtube.com/watch?v=BCNZ5Uxctuk&t=561s&ab_channel=Brief))_

As you can see, this diff relies on the line "Everyone had the boot in", but this line no longer exists on John's branch. As a result, you could expect applying the patch to fail. Go on, give it a try:

git apply paul_3.patch


![Image](https://www.freecodecamp.org/news/content/images/2023/04/image-280.png)
_Applying the patch failed (Source: [Brief](https://www.youtube.com/watch?v=BCNZ5Uxctuk&t=561s&ab_channel=Brief))_

Indeed, you can see that it failed.

But should it really fail? 🤔

As explained earlier, `git merge` uses a 3-way merge algorithm, and this can come in handy here. What would be the first step of this algorithm?

Well, first, Git would find the merge base – that is, the common ancestor of Paul's branch and John's branch. Consider the history:

![Image](https://www.freecodecamp.org/news/content/images/2023/04/image-351.png)
_The history after introducing "Commit 12" (Source: [Brief](https://www.youtube.com/watch?v=BCNZ5Uxctuk&t=561s&ab_channel=Brief))_

So the common ancestor of "Commit 11" and "Commit 12" is "Commit 10". We can verify this by running the command:

git merge-base john_branch_3 paul_branch_3


Now we can take the patches we generated from the diffs on both branches, and apply them to `main`. Would that work?

First, try to apply John's patch, and then Paul's patch.

Consider the diff:

git diff main john_branch_3



![Image](https://www.freecodecamp.org/news/content/images/2023/04/image-278.png)
_The output of `git diff main john_branch_3` (Source: [Brief](https://www.youtube.com/watch?v=BCNZ5Uxctuk&t=561s&ab_channel=Brief))_

We can store it in a file:

git diff main john_branch_3 > john_3.patch


And I want to apply this patch on `main`, so:

git checkout main git apply john_3.patch


Let's consider the result:

nano everyone.md


![Image](https://www.freecodecamp.org/news/content/images/2023/04/image-282.png)
_The contents of `everyone.md` after applying John's patch (Source: [Brief](https://www.youtube.com/watch?v=BCNZ5Uxctuk&t=561s&ab_channel=Brief))_

The line changed as expected. Nice 😎

Now, can Git apply Paul's patch? To remind you, this is the patch:

![Image](https://www.freecodecamp.org/news/content/images/2023/04/image-283.png)
_The contents of Paul's patch (Source: [Brief](https://www.youtube.com/watch?v=BCNZ5Uxctuk&t=561s&ab_channel=Brief))_

Well, Git **cannot** apply this patch, because this patch assumes that the line  "Everyone had the boot in" exists. Trying to apply is liable to fail:

git apply -v paul_3.branch


![Image](https://www.freecodecamp.org/news/content/images/2023/04/image-284.png)
_Applying Paul's patch failed._

What you tried to do now, applying Paul's patch on `main` branch after applying John's patch, is the same as being on `john_branch_3`, and attempting to apply the patch, that is:

```git checkout john_branch_3
git apply paul_3.patch

What would happen if we tried the other way around?

First, clean up the state:

git reset --hard

And start from Paul's branch:

git checkout paul_branch_3

Can we apply John's patch? As a reminder, this is the status of everyone.md on this branch:

_The contents of everyone.md on paul_branch_3 (Source: Brief)_

And this is John's patch:

_The contents of John's patch (Source: Brief)_

Would applying John's patch work? 🤔 Try to answer yourself before reading on.

You can try:

git apply john_3.patch

_Git fails to apply John's patch (Source: Brief)_

Well, no! Again, if you are not sure what happened, you can always ask git apply to be a bit more verbose:

git apply john_3.patch -v

_You can get more information by using the -v flag (Source: Brief)_

Git is looking for "Everyone put the feet down", but Paul has already changed this line so it now consists of the word "foot" instead of "feet". As a result, applying this patch fails.

Notice that changing the number of context lines here (that is, using git apply with the -C flag, as discussed in a previous post) is irrelevant – Git is unable to locate the actual line that the patch is trying to erase.

But actually, Git can make this work, if you just add a flag to apply, telling it to perform a 3-way merge under the hood:

git apply -3 john_3.patch

_Applying with -3 flag succeeds (Source: Brief)_

And let's consider the result:

_The contents of everyone.md after ther merge (Source: Brief)_

Exactly what we wanted! You have Paul's verse (marked in the image above), and both of John's changes!

So, how was Git able to accomplish that?

Well, as I mentioned, Git really did a 3-way merge, and with this example, it will be a good time to dive into what this actually means.

How Git's 3-way Merge Algorithm Works

Get back to the state before applying this patch:

git reset --hard

To perform the merge, Git looks at the diff between the three different versions of the file in question on these three revisions. In your case, it's the file everyone.md, and the revisions are "Commit 10", Paul's branch – that is, "Commit 11", and John's branch, that is, "Commit 12".

Git makes the merging decision based on the status of each line in each of these versions.

_The three versions considered for the 3-way merge (Source: Brief)_

In case not all three versions match, that is a conflict. Git can resolve many of these conflicts automatically, as we will now see.

Let's consider specific lines.

The first lines here exist only on Paul's branch:

_Lines that appear on Paul's branch only (Source: Brief)_

This means that the state of John's branch is equal to the state of the merge base. So the 3-way merge goes with Paul's version.

_If the state of the merge base is the same as A, and this state is different from B, the algorithm goes with B (Source: Brief)_

Next, you can see lines where all three versions agree – they exist on the merge base, A and B, with equal data.

_Lines where all three versions agree (Source: Brief)_

So the algorithm has a trivial choice – just take that version.

_In case all three versions agree, the algorithm goes with that single version (Source: Brief)_

In a previous example, we saw that if the merge base and A agree, and B's version is different, the algorithm picks B. This works in the other direction too – for example, here you have a line that exists on John's branch, different than that on the merge base and Paul's branch.

_A line where Paul's version matches the merge base's version, and John has a different version (Source: Brief)_

Hence, John's version is chosen.

_If the state of the merge base is the same as B, and this state is different from A, the algorithm goes with A (Source: Brief)_

Now consider another case, where both A and B agree on a line, but the value they agree upon is different from the merge base – both John and Paul agreed to change the line "Everyone put their feet down" to "Everyone put their foot down":

_A line where Paul's version matches the John's version; yet the merge base has a different version (Source: Brief)_

In this case, the algorithm picks the version on both A and B.

_In case A and B agree on a version which is different from the merge base's version, the algorithm picks the version on both A and B (Source: Brief)_

Notice this is not a democratic vote. In the previous case, the algorithm picked the minority version, as it resembled the newest version of this line. In this case, it happens to pick the majority – but only because A and B are the revisions that agree on the new version.

The same would happen if we used git merge:

git merge john_branch_3

Without specifying any flags, git merge will default to using a 3-way merge.

_By default, git merge uses a 3-way merge algorithm (Source: Brief)_

The status of everyone.md after running the command above would be the same as the result you achieved by applying the patches with git apply -3.

If you consider the history:

_Git's history after performing the merge (Source: Brief)_

What will happen if you now merge from main? That is, switch to the main branch, which is pointing to "Commit 10":

git checkout main

And then merge Paul's branch?

git merge paul_branch_3

Indeed, a fast forward, as before running this command, main was an ancestor of paul_branch_3.

_A fast-forward merge (Source: Brief)_

So, this is a 3-way merge. In general, if all versions agree on a line, then this line is used. If A and the merge base match, and B has another version, B is taken. In the opposite case, where the merge base and B match, the A version is selected. If A and B match, this version is taken, whether the merge base agrees or not.

This description leaves one open question though: What happens in cases where all three versions disagree?

Well, that's a conflict that Git does not resolve automatically. In these cases, Git calls for a human's help.

How to Resolve Merge Conflicts

By following so far, you should understand the basics of git merge, and how Git can automatically resolve some conflicts. You also understand what cases are automatically resolved.

Next, let's consider a more advanced case.

Say Paul and John keep working on this song.

Paul creates a new branch:

git checkout -b paul_branch_4

And he decides to add some "Yeah"s to the song, so he changes this verse as follows:

_Paul's additions (Source: Brief)_

So Paul stages and commits these changes:

```git add everyone.md git commit -m "Commit 13"


Paul also creates another song, `let_it_be.md` and adds it to the repo:

git add let_it_be.md git commit -m "Commit 14"


This is the history:


![Image](https://www.freecodecamp.org/news/content/images/2023/04/image-356.png)
_The history after Paul introduced "Commit 14" (Source: [Brief](https://www.youtube.com/watch?v=BCNZ5Uxctuk&t=561s&ab_channel=Brief))_

Back to `main`:

git checkout main


John also branches out:

git checkout -b john_branch_4


And John also works on the song "Everyone had a hard year", later to be called "I've got a feeling" (again, this is not an article about the Beatles, so I won't elaborate on it here. See the appendix if you are curious).

John decides to change all occurrences of "Everyone" to "Everybody":


![Image](https://www.freecodecamp.org/news/content/images/2023/04/image-307.png)
_John changes al occurrences of "Everyone" to "Everybody" (Source: [Brief](https://www.youtube.com/watch?v=BCNZ5Uxctuk&t=561s&ab_channel=Brief))_

He stages and commits this song to the repo:

git add everyone.md git commit -m "Commit 15"


Nice. Now John also creates another song, `across_the_universe.md`. He adds it to the repo as well:

git add across_the_universe.md git commit -m "Commit 16"


Observe the history again:


![Image](https://www.freecodecamp.org/news/content/images/2023/04/image-308.png)
_The history after John introduced "Commit 16" (Source: [Brief](https://www.youtube.com/watch?v=BCNZ5Uxctuk&t=561s&ab_channel=Brief))_

You can see that the history diverges from `main`, to two different branches – `paul_branch_4`, and `john_branch_4`.

At this point, John would like to merge the changes introduced by Paul.

What is going to happen here?

Remember the changes introduced by Paul:

git diff main paul_branch_4


![Image](https://www.freecodecamp.org/news/content/images/2023/04/image-309.png)
_The output of `git diff main paul_branch_4` (Source: [Brief](https://www.youtube.com/watch?v=BCNZ5Uxctuk&t=561s&ab_channel=Brief))_

What do you think? Will merge work? 🤔

Try it out:

git merge paul_branch_4



![Image](https://www.freecodecamp.org/news/content/images/2023/04/image-311.png)
_A merge conflict (Source: [Brief](https://www.youtube.com/watch?v=BCNZ5Uxctuk&t=561s&ab_channel=Brief))_

We have a conflict! 🥁

It seems that Git cannot merge these branches on its own. You can get an overview of the merge state, using `git status`:


![Image](https://www.freecodecamp.org/news/content/images/2023/04/image-310.png)
_The output of `git status` right after the `merge` operation (Source: [Brief](https://www.youtube.com/watch?v=BCNZ5Uxctuk&t=561s&ab_channel=Brief))_

The changes that Git had no problem resolving are staged for commit. And there is a separate section for "unmerged paths" – these are files with conflicts that Git could not resolve on its own.

It's time to understand why and when these conflicts happen, how to resolve them, and also how Git handles them under the hood.
Alright then! I hope you are at least as excited as I am. 😇

Let's recall what we know about 3-way merges:

First, Git will look for the merge base – the common ancestor of `john_branch_4` and `paul_branch_4`. Which commit would that be?

Correct, it would be the tip of `main` branch, the commit in which we merged `john_branch_3` into `paul_branch_3`.

Again, if you are not sure, you can verify that by running:

git merge-base john_branch_4 paul_branch_4


And at the current state, `git status` knows which files are staged and which aren't.

Consider the process for each file, which is the same as the 3-way merge algorithm we considered per line, but on a file's level:

`across_the_universe.md` exists on John's branch, but doesn't exist on the merge base or on Paul's branch. So Git chooses to include this file. Since you are already on John's branch and this file is included in the tip of this branch, it is not mentioned by `git status`.

`let_it_be.md` exists on Paul's branch, but doesn't exist on the merge-base or John's branch. So `git merge` "chooses" to include it.

What about `everyone.md`? Well, here we have three different states of this file: its state on the merge base, its state on John's branch, and its state on Paul's branch. While performing a `merge`, Git stores all of these versions on the **index**. 

Let's observe that by looking directly at the index with the command `git ls-files`:

git ls-files -s –-abbrev



![Image](https://www.freecodecamp.org/news/content/images/2023/04/image-312.png)
_The output of `git ls-files -s –-abbrev` after the merge operation (Source: [Brief](https://www.youtube.com/watch?v=BCNZ5Uxctuk&t=561s&ab_channel=Brief))_

You can see that `everyone.md` has three different entries. Git assigns each version a number that represents the "stage" of the file, and this is a distinct property of an index entry, alongside the file's name and the mode bits (I covered the index in [a previous post](https://medium.com/swimm/a-visualized-intro-to-git-internals-objects-and-branches-68df85864037)).

When there is no merge conflict regarding a file, its "stage" is `0`. This is indeed the state for `across_the_universe.md`, and for `let_it_be.md`.

On a conflict's state, we have:

* Stage `1` – which is the merge base.
* Stage `2` – which is "your" version. That is, the version of the file on the branch you are merging *into*. In our example, this would be `john_branch_4`.
* Stage `3` – which is "their" version, also called the `MERGE_HEAD`. That is, the version on the branch you are merging (into the current branch). In our example, that is `paul_branch_4`.

To observe the file's contents in a specific stage, you can use a command I introduced in [a previous post](https://medium.com/swimm/getting-hardcore-creating-a-repo-from-scratch-cc747edbb11c), `git cat-file`, and provide the blob's SHA:

git cat-file -p



![Image](https://www.freecodecamp.org/news/content/images/2023/04/image-313.png)
_Using `git cat-file` to present the content of the file on John's branch, right from its state in the index (Source: [Brief](https://www.youtube.com/watch?v=BCNZ5Uxctuk&t=561s&ab_channel=Brief))_

And indeed, this is the content we expected – from John's branch, where the lines start with "Everybody" rather than "Everyone".

A nice trick that allows you to see the content quickly without providing the blob's SHA-1 value, is by using `git show`, like so:

git show ::everyone.md


For example, to get the content of the same version as with `git cat-file -p `, you can write `git show :2:everyone.md`.

Git records the three states of the three commits into the index in this way at the start of the merge. It then follows the three-way merge algorithm to quickly resolve the simple cases:

In case all three stages match, then the selection is trivial.

If one side made a change while the other did nothing – that is, stage 1 matches stage 2, then we choose stage 3 – or vice versa. That's exactly what happened with `let_it_be.md` and `across_the_universe.md`.

In case of a deletion on the incoming branch, for example, and given there were no changes on the current branch, then we would see that stage 1 matches stage 2, but there is no stage 3. In this case, `git merge` removes the file for the merged version.

What's really cool here is that for matching, Git doesn't need the actual files. Rather, it can rely on the SHA-1 values of the corresponding blobs. This way, Git can easily detect the state a file is in.

![Image](https://www.freecodecamp.org/news/content/images/2023/04/image-352.png)
_Git performs the same 3-way merge algorithm on a files level (Source: [Brief](https://www.youtube.com/watch?v=BCNZ5Uxctuk&t=561s&ab_channel=Brief))_

Cool, so for `everyone.md` you have this special case – where stage 1, stage 2 and stage 3 are all different from one another. That is, they have different blob SHAs. It's time to go deeper and understand the merge conflict. 😊

One way to do that would be to simply use `git diff`. In [a previous post](https://www.freecodecamp.org/news/git-diff-and-patch/), we examined `git diff` in detail, and saw that it shows the differences between various combinations of the working tree, index or commits. 

But `git diff` also has a special mode for helping with merge conflicts:

`git diff`

![Image](https://www.freecodecamp.org/news/content/images/2023/04/image-314.png)
_The output of `git diff` during a conflict (Source: [Brief](https://www.youtube.com/watch?v=BCNZ5Uxctuk&t=561s&ab_channel=Brief))_

This output may be confusing at first, but once you get used to it, it's pretty clear. Let's start by understanding it, and then see how you can resolve conflicts with other, more visual tools.

The conflicted section is separated by the "equal" marks (====), and marked with the corresponding branches. In this context, "ours" is the current branch. In this example, that would be `john_branch_4`, the branch that `HEAD` was pointing to when we initiated the `git merge` command. "Theirs" is the `MERGE_HEAD`, the branch that we are merging in – in this case, `paul_branch_4`.

So `git diff` without any special flags shows changes between the working tree and the index, which in this case are the conflicts yet to be resolved. The output doesn't include staged changes, which is very convenient for resolving the conflict.

Time to resolve this manually. Fun!

So, why is this a conflict?

For Git, Paul and John made different changes to the same line, for a few lines. John changed it to one thing, and Paul changed it to another thing. Git cannot decide which one is correct.

This is not the case for the last lines, like the line that used to be "Everyone had a hard year" on the merge base. Paul hasn't changed this line, or the lines surrounding it, so its version on `paul_branch_4`, or "theirs" in our case, agrees with the merge_base. Yet John's version, "ours", is different. Thus `git merge` can easily decide to take this version.

But what about the conflicted lines?

In this case, I know what I want, and that is actually a combination of these lines. I want the lines to start with `Everybody`, following John's change, but also to include Paul's "yeah"s. So go ahead and create the desired version by editing `everyone.md`:
`nano everyone.md`

![Image](https://www.freecodecamp.org/news/content/images/2023/04/image-315.png)
_Editing the file manually to achieve the desired state (Source: [Brief](https://www.youtube.com/watch?v=BCNZ5Uxctuk&t=561s&ab_channel=Brief))_

To compare the result file to what you had in the branch prior to the merge, you can run:

git diff --ours


Similarly, if you wish to see how the result of the merge differs from the branch you merged into our branch, you can run:

git diff -–theirs


You can even see how the result is different from both sides using:

git diff -–base


Now you can stage the fixed version:

git add everyone.md


After staging, if you look at `git status`, you will see no conflicts:

![Image](https://www.freecodecamp.org/news/content/images/2023/04/image-316.png)
_After staging the fixed version `everyone.md`, there are no conflicts (Source: [Brief](https://www.youtube.com/watch?v=BCNZ5Uxctuk&t=561s&ab_channel=Brief))_

You can now simply use `git commit`, and Git will present you with a commit message containing details about the merge. You can modify it if you like, or leave it as is. Regardless of the commit message, Git will create a "merge commit" – that is, a commit with more than one parent. 

To validate that, consider the history:

![Image](https://www.freecodecamp.org/news/content/images/2023/04/image-317.png)
_The history after completing the merge operation (Source: [Brief](https://www.youtube.com/watch?v=BCNZ5Uxctuk&t=561s&ab_channel=Brief))_

`john_branch_4` now points to the new merge commit. The incoming branch, "theirs", in this case, `paul_branch_4`, stays where it was.

# How to Use VS Code to Resolve Conflicts

I will show you now how to resolve the same conflict using a graphical tool. For this example, I will use VS Code, which is free and very common. There are many other tools, yet the process is similar, so I will just show VS Code as an example. 

First, get back to the state before the merge:

git reset --hard HEAD~


And try to merge again:

git merge paul_branch_4


You should be back at the same status:

![Image](https://www.freecodecamp.org/news/content/images/2023/04/image-318.png)
_Back at the conflicting status (Source: [Brief](https://www.youtube.com/watch?v=BCNZ5Uxctuk&t=561s&ab_channel=Brief))_

Let's see how this appears on VS Code:

![Image](https://www.freecodecamp.org/news/content/images/2023/04/image-320.png)
_Conflict resolution with VS Code (Source: [Brief](https://www.youtube.com/watch?v=BCNZ5Uxctuk&t=561s&ab_channel=Brief))_

VS Code marks the different versions with "Current Change" – which is the "ours" version, the current `HEAD`, and "Incoming Change" for the branch we are merging into the active branch. You can accept one of the changes (or both) by clicking on one of the options.

If you clicked on `Resolve in Merge editor`, you would get a more visual view of the state. VS Code shows the status of each line:


![Image](https://www.freecodecamp.org/news/content/images/2023/04/image-321.png)
_VS Code's Merge Editor (Source: [Brief](https://www.youtube.com/watch?v=BCNZ5Uxctuk&t=561s&ab_channel=Brief))_

If you look closely, you will see that VS Code shows changes within words – for example, showing that "Every**one**" was changed to "Every**body**", marking the changed parts. 

You can accept either version, or you can accept a combination. In this case, if you click on "Accept Combination", you get this result:


![Image](https://www.freecodecamp.org/news/content/images/2023/04/image-322.png)
_VS Code's Merge Editor after clicking on "Accept Combination" (Source: [Brief](https://www.youtube.com/watch?v=BCNZ5Uxctuk&t=561s&ab_channel=Brief))_

VS Code did a really good job! The same three way merge algorithm was implemented here and used on the *word* level rather than the *line* level. So VS Code was able to actually resolve this conflict in a rather impressive way. Of course, you can modify VS Code's suggestion, but it provided a very good start.


# One More Powerful Tool 🪛
Well, this was the first time in this entire series of Git articles that I use a tool with a graphical user interface. Indeed, graphical interfaces can be very convenient to understand what's going on when you are resolving merge conflicts.

However, like in many other cases, when we need the big guns or *really* understand what's going on, the command line becomes handy. So let's get back to the command line and learn a tool that can come in handy in more complicated cases.

Again, go back to the state before the merge:

git reset --hard HEAD~


And merge:

git merge paul_branch_4


And say, you are not exactly sure what happened. Why is there a conflict? One very useful command would be:

git log -p -–merge ```

As a reminder, git log shows the history of commits that are reachable from HEAD. Adding -p tells git log to show the commits along the diffs they introduced. The --merge switch makes the command show all commits containing changes relevant to any unmerged files, on either branch, together with their diffs.

This can help you identify the changes in history that led to the conflicts. So in this example, you'd see:

_The output of git log -p -–merge (Source: Brief)_

The first commit we see is "Commit 15", as in this commit John modified everyone.md, a file that still has conflicts. Next, Git shows "Commit 13", where Paul changed everyone.md:

_The output of git log -p -–merge - continued (Source: Brief)_

Notice that git log --merge did not mention previous commits that had changed everyone.md before "Commit 13", as they had not affected the current conflict.

This way, git log tells you all you need to know to understand the process that got you into the current conflicting state. Cool! 😎

Using the command line, you can also ask Git to take only one side of the changes – either "ours" or "theirs", even for a specific file.

You can also instruct Git to take some parts of the diffs of one file and another from another file. I will provide links that describe how to do that in the additional resources section below.

For the most part, you can accomplish that pretty easily either manually or from the UI of your favorite IDE.

For now, it's time for a recap.

Recap

In this guide, you got an extensive overview of merging with Git. You learned that merging is the process of combining the recent changes from several branches into a single new commit. The new commit has two parents – those commits which had been the tips of the branches that were merged.

We considered a simple, fast-forward merge, which is possible when one branch diverged from the base branch, and then just added commits on top of the base branch.

We then considered three-way merges, and explained the three-stage process:

First, Git locates the merge base. As a reminder, this is the first commit that is reachable from both branches.
Second, Git calculates two diffs – one diff from the merge base to the first branch, and another diff from the merge base to the second branch. Git generates patches based on those diffs.
Third and last, Git applies both patches to the merge base using a 3-way merge algorithm. The result is the state of the new, merge commit.

You saw the output of git diff when we are in a conflicting state, and how to resolve conflicts either manually or with VS Code.

There is much more to be said about merges – different merge strategies, recursive merges, and so on. Yet, after this guide, you should have a robust understanding of what merge is, and what happens under the hood in the vast majority of cases.

About the Author

Omer Rosenbaum is Swimm’s Chief Technology Officer. He's the author of the Brief YouTube Channel. He's also a cyber training expert and founder of Checkpoint Security Academy. He's the author of Computer Networks (in Hebrew). You can find him on Twitter.

Additional References

Git Diff and Patch – Full Handbook for Developers

Omer Rosenbaum — Tue, 21 Feb 2023 21:42:37 +0000

Many of the interesting processes in Git like merging, rebasing, or even committing are based on diffs and patches.

Developers work with diffs all the time, whether using Git directly or relying on the IDE's diff view. In this post, you will learn what Git diffs and patches are, their structure, and how to apply patches.

In a previous post, you learned about Git’s objects. Specifically, we discussed that a commit is a snapshot of the working tree at a certain point in time, in addition to some meta-data.

So, what do I mean when I say diff? Let’s start with some history.

Git Diff's History 📖

Git’s diff is based on the diff utility on UNIX systems. diff was developed in the early 1970s on the Unix operating system. The first released version shipped with the 5th Edition of Unix in 1974.

git diff is a command that takes two inputs, and computes the difference between them. Inputs can be commits, but also files, and even files that have never been introduced to the repository.

Git diff takes two inputs, which can be commits or files (Source: Brief)

This is important – git diff computes the difference between two strings, which most of the time happen to consist of code, but not necessarily.

Time to Get Hands-On 🙌🏻

You are encouraged to run the commands yourself while reading this post.

Consider this very short text file, called file.txt on my machine, which consists of 6 lines:

file.txt consists of 6 lines (Source: Brief)

Now, modify this file a bit. Remove the second line, and insert a new line as the fourth line. Add an ! to the end of the last line, so you get this result:

After modifying file.txt, we get different 6 lines (Source: Brief)

Save this file with a new name, say new_file.txt.

Now we can run git diff to compute the difference between the files like so:

git diff -–no-index file.txt new_file.txt (I will explain the --no-index switch of this command later.)

The output of git diff (Source: Brief)

So the output of git diff shows quite a lot of things.

For now, focus on the part starting with This is a simple line. You can see that the added line (// new test) is preceded by a + sign. The deleted line is preceded by a - sign.

Addition lines are preceded by +, deletion lines by -, and modification lines are sequences of deletions and additions (Source: Brief)

The terms patch and diff are often used interchangeably, although there is a distinction, at least historically.

A diff shows the differences between two files, or snapshots, and can be quite minimal in doing so. A patch is an extension of a diff, augmented with further information such as context lines and filenames, which allow it to be applied more widely. It is a text document that describes how to alter an existing file or codebase.

These days, the Unix diff program, and git diff, can produce patches of various kinds.

A patch is a compact representation of the differences between two files. It describes how to turn one file into another.

That is, if you apply the “instructions” produced by git diff on file.txt – that is, remove the second line, insert // new text as the fourth line, and add another ! to the last line – you would get the content of new_file.txt.

Another important thing to note is that a patch is asymmetric: the patch from file.txt to new_file.txt is not the same as the patch for the other direction.

So, in this example, generating a patch between new_file.txt and file.txt, in this order, would mean exactly the opposite instructions than before - add the second line instead of removing it, and so on.

A patch consists of asymmetric instructions to get from one file to another (Source: Brief)

Try it out: git diff -–no-index new_file.txt file.txt

Running git diff in the reverse direction yields the reverse instructions - add a line instead of removing it, and so on (Source: Brief)

The patch format uses context, as well as line numbers, to locate differing file regions. This allows a patch to be applied to a somewhat earlier or later version of the first file than the one from which it was derived, as long as the applying program can still locate the context of the change.

The Structure of a Diff 🔍

So, it's time to dive deeper 😎.

Generate a diff from file.txt to new_file.txt again, and consider the output more carefully:

git diff -–no-index file.txt new_file.txt

The first line introduces the compared files. Git always gives one file the name a, and the other the name b. So in this case file.txt is called a, whereas new_file.txt is called b.

The first line in diff 's output introduces the files being compared (Source: Brief)

Then the second line, starting with index, includes the blob SHAs of these files. So even though in our case they are not even stored within a Git repo, Git shows their corresponding SHA-1 values.

If you need a reminder about blobs in particular and Git objects in general, check out this post.

The third value in this line, 100644, is the "mode bits", indicating that this is a "regular" file: not executable and not a symbolic link.

The use of two dots (..) here between the blob SHAs is just as a separator (unlike other cases where it is used within Git).

Other header lines might indicate the old and new mode bits if they changed, old and new filenames if the file were being renamed, and so on.

The second line in diff 's output includes the blob SHAs of the compared files, as well as the mode bits (Source: Brief)

The blob SHAs (also called "blob IDs") are helpful if this patch is later applied by Git to the same project and there are conflicts while applying it.

After the blob IDs, we have two lines: one starting with - signs, and the other starting with + signs. This is the traditional "unified diff" header, again showing the files being compared and the direction of the changes: - signs show lines in the A version but missing from the B version, and + signs, lines missing in A version but present in B.

If the patch were of this file being added or deleted in its entirety, then one of these would be /dev/null to signal that.

- signs show lines in the A version but missing from the B version; and + signs, lines missing in A version but present in B (Source: Brief)

Consider the case where we delete a file: rm file.txt

And then we use git diff:

diff's output for a deleted file (Source: Brief)

The A version, representing the state of the index, is currently file.txt, compared to the working dir where this file does not exist, so it is /dev/null. All lines are preceded by - signs as they exist only in the A version.

Going back to the previous diff:

diff's output includes changes sections called "hunks" or "chunks" (Source: Brief)

After this unified diff header, we get to the main part of the diff, consisting of "difference sections", also called "hunks" or "chunks" in Git.

Note that these terms are used interchangeably, and you may stumble upon either of them in Git's documentation and tutorials, as well as Git's source code.

Every hunk begins with a single line, starting with two @ signs. These signs are followed by at most four numbers, and then a header for the chunk - which is an educated guess by Git which sometimes works well.

Usually, it will include the beginning of a function or a class, when possible. In this example it doesn't include anything as this is a text file, so consider another example for a moment:

git diff -–no-index example.py example_changed.py

When possible, Git includes a header for each hunk, for example a function or class definition (Source: Brief)

In the image above the hunk's header includes the beginning of the function that includes the changed lines - def example_function(x).

Back to our previous example then:

Back to the previous diff (Source: Brief)

After the two @ signs, you can find four numbers.

The first numbers are preceded by a - sign as they refer to file A. The first number represents the line number corresponding to the first line in file A this hunk refers to. In the example above, it is 1, meaning that the line This is a simple file corresponds to line number 1 in version file A.

This number is followed by a comma (,), and then the number of lines this chunk consists of in file A. This number includes all context lines (the lines preceded with a space in the diff), or lines marked with a - sign, as they are part of file A, but not lines marked with a + sign, as they do not exist in file A.

In the example above this number is 6, counting the context line This is a simple file, the - line It has a nice poem:, then the three context lines, and lastly Are belong to you.

As you can see, the lines beginning with a space character are context lines, which means they appear as shown in both file A and file B.

Then, we have a + sign to mark the two numbers that refer to file B. First, the line number corresponding to the first line in file B, followed by the number of lines this chunk consists of - in file B.

This number includes all context lines, as well as lines marked with the + sign, as they are part of file B, but not lines marked with a - sign.

After the header of the chunk, we get the actual lines - either context, - or + lines.

Typically and by default, a hunk starts and ends with three context lines, in case there are of course three lines before and after the modified lines in the source file.

The patch format by git diff (Source: Brief)

How to Produce diffs ⌨️

The example above shows a diff between the two files exactly. A single patch file can contain the differences for any number of files, and git diff produces diffs for all altered files in the repository in a single patch.

Often, you will see the output of git diff showing two versions of the same file and the difference between them.

To demonstrate, consider this other repository:

cd ~/brief-example

At the current state, the active directory is a Git repository, with a clean status:

git status

In another repository with a clean status (Source: Brief)

Take an existing file, like this one:

_An example file - my_file.py (Source: Brief)_

And change one of its lines. For example, consider the second line:

_The contents of my_file.py after modifying the second line (Source: Brief)_

And run git diff:

_The output of git diff for my_file.py after changing it (Source: Brief)_

The output of git diff shows the difference between my_file.py's version in the staging area, which in this case is the same as the last commit (HEAD), and in the working directory.

I covered the terms "working directory", "staging area", and "commit" in a previous post, so check it out in case you missed it or would like to refresh your memory.

As a reminder, the terms "staging area" and "index" are interchangeable, and both are widely used.

At this state, the status of the working dir is the same as the status of the index and that of HEAD. (Source: Brief)

So to see the difference between the working dir and the staging area, use git diff, without any additional flags.

Without switches, git diff shows the difference between the staging area (Source: Brief)

As you can see, git diff lists here both file A and file B pointing to my_file.py. So file A here refers to the version of my_file.py in the staging area, whereas file B refers to its version in the working dir.

Note that if you modify my_file.py in a text editor, and don’t save the file, then git diff will not be aware of the changes you've made, as they haven’t been saved to the working dir.

There are a few switches we can provide to git diff to get the diff between the working dir and a specific commit, or between the staging area and the latest commit, or between two commits and so on.

First create a new file, new_file.txt, and save it. Currently the file is in the working dir, and it is actually untracked in Git.

_A simple new file saved as new_file.txt (Source: Brief)_

Now stage and commit this file: git add new_file.txt git commit -m "new file!"

Now, the state of HEAD is the same as the state of the staging area, as well as the working tree:

The state of HEAD is the same as the index and the working dir (Source: Brief)

Next, edit new_file.txt, by adding a new line at the beginning and another new line at the end:

_Modifying new_file.txt by adding a line in the beginning and another in the end (Source: Brief)_

As a result, the state is as follows:

After saving, the state in the working dir is different than that of the index or HEAD (Source: Brief)

A nice trick would be to use git add -p, which allows you you split the changes even within a file, and consider which ones you'd like to stage.

So in this case, add the first line to the index, but not the last line. To do that, you can split the hunk using s, then accept to stage the first hunk (using y), and not the second part (using n).

If you are not sure what each letter stands for, you can always use a ? and Git will tell you.

Using git add -p, you can stage only the first change (Source: Brief)

So now the state in HEAD is without either of those new lines. In the staging area we have the first line but not the last line, and in the working dir we have both new lines.

The state after staging only the first line (Source: Brief)

If you use git diff, what will happen?

git diff shows the difference between the index and the working dir (Source: Brief)

Well, as stated before, you get the diff between the staging area and the working tree.

What happens if you want to get the diff between HEAD and the staging area? For that, you can use git diff –cached:

git diff --cached shows the difference between HEAD and the index (Source: Brief)

And what if we want the difference between HEAD and the working tree? For that we can run git diff HEAD:

git diff HEAD shows the difference between HEAD and the working dir (Source: Brief)

To summarize the different switches for git diff, see this diagram that you can go back to as a reference when needed:

Different switches for git diff (Source: Brief)

As a reminder, at the beginning of this post you used git diff -–no-index. With the --no-index switch you can compare two files that are not part of the repository - or of any staging area.

Now, commit the changes you have in the staging area:

git commit -m "added a first line"

To observe the diff between this commit, and its parent commit, you can run the following command:

git diff HEAD~1 HEAD

The output of git diff HEAD~1 HEAD (Source: Brief)

By the way, we can omit the 1 above and write HEAD~, and get the same result. Using 1 is the explicit way to state you are referring to the first parent of the commit.

Note that writing the parent commit here, HEAD~1, first results in a diff showing how to get from the parent commit to the current commit. Of course, I could also generate the reverse diff by writing:

git diff HEAD HEAD~1

The output of git diff HEAD HEAD~1 generates the reverse patch (Source: Brief)

The different switches for git diff (Source: Brief)

A short way to view the diff between a commit and its parent, is by using git show, for example:

git show HEAD

This is the same as writing:

git diff HEAD~ HEAD

We can now update our diagram:

_The contents of new_file.txt after using git reset --hard HEAD~1 (Source: Brief)_

As a reminder, Git commits are snapshots - of the entire working directory of the repository, at a certain point in time. Yet, it is sometimes not very useful to regard a commit as a whole snapshot, but rather by the changes this specific commit introduced. In other words, by the diff between a parent commit to the next commit.

It is still important to remember that Git stores the entire snapshots, and the diff is dynamically generated from the snapshot data - by comparing the root trees of the commit and its parent.

Of course, Git can compare any two snapshots in time, not just adjacent commits, and also generate a diff of files not included in a repository.

How to Apply Patches 💪🏻

By using git diff you can see the patch, and you can then apply this patch using git apply.

Historical note 📔

The biggest problem with using patches is that it is hard to apply a patch when your working directory does not match the sender's previous commit.

Losing the commit history makes it difficult to resolve conflicts. You will better understand it as you dive deeper into the process of git apply.

A simple apply

What does it mean to apply a patch? It's time to try it out!

Take the output of git diff:

git diff HEAD~1 HEAD

And store it in a file:

git diff HEAD~1 HEAD > my_patch.patch

And reset to undo the last commit:

git reset –hard HEAD~1

If you are not completely comfortable with git reset, check a previous post that covered it in depth. In short, it allows us to "reset" the state of where HEAD is pointing to, as well as the state of the index and of the working dir.

In the example above, they are all set to the state of HEAD~1, or Commit 3 in the diagram.

So after running the reset command, the contents of the file are as follows:

nano new_file.txt

The patch you are about to apply, as generated by git diff (Source: Brief)

And we will apply this patch:

nano my_patch.patch

This patch tells git to find the lines:

This is a new file
With new content!

That used to be lines 1 and 2, and add a line START right above them.

Run this command to apply the patch:

git apply my_patch.patch

And as a result, you get this version of your file, just like the commit you have created before:

nano new_file.txt

_The contents of new_file.txt after applying the patch (Source: Brief)_

Understanding the Context Lines 🧑🏻‍🏫

To understand the importance of context lines, consider a more advanced scenario. What happens if line numbers have changed since you created the patch file? 🤔

To test, start by creating another file:

_Creating another file - another_file.txt (Source: Brief)_

Stage and commit this file:

git add another_file.txt

git commit -m "another file"

Now, change this file by adding a new line, and also erasing the line before the last one:

_Changes to another_file.txt (Source: Brief)_

Observe the difference between the original version of the file and the version including your changes:

git diff -- another_file.txt

_The output for git diff -- another_file.txt (Source: Brief)_

(Using -- another_file.txt tells Git to run the command diff, taking into consideration only another_file.txt, so you don't get the diff for other files.)

Store this diff into a patch file:

git diff -- another_file.txt > new_patch.patch

Now, reset your state to that before introducing the changes: git reset --hard

If you were to apply new_patch.patch now, it would simply work. Consider a more interesting case.

Modify another_file.txt again by adding a new line at the beginning:

_Adding a new line at the beginning of another_file.txt (Source: Brief)_

As a result, the line numbers are different from the original version where the patch has been created. Consider the patch you created before:

_new_patch.patch (Source: Brief)_

It assumes that the line So this is a file is the first line in another_file.txt, which is no longer the case. So...will git apply work?

git apply doesn't apply the patch (Source: Brief)

Well, no. The patch does not apply. But why? Is it really because of the change in line numbers?

To better understand the process Git is performing, you can add the --verbose flag to git apply, like so:

git apply --verbose new_patch.patch

git apply --verbose shows the process Git is taking to apply the patch (Source: Brief)

It seems that Git searched for the entire contents of the file, specifically, including the line So we are writing an example, that no longer exists in the file. As Git cannot find this line, it cannot apply the patch.

Why does Git look for the entire file? By default, Git looks for 3 lines of context before and after each change introduced in the patch. If you take three lines before and after the added line, and three lines before and after the deleted line (actually only one line after, as no other lines exist) - you get to the entire file.

You can ask Git to rely on fewer lines of context, using the -C argument. For example, to ask Git to look for 1 line of the surrounding context, run the following command:

git apply -C1 new_patch.patch

The patch applies cleanly! 🎉

Why is that? Consider the patch again:

_new_patch.patch (Source: Brief)_

When applying the patch with the -C1 option, Git is looking for the lines:

It has some really nice lines
Like this one

in order to add the line !!!This is the new line I am adding!!! between these two lines. These lines exist (and, importantly, they appear one right after the other). So Git can successfully add the line between them, even though the line numbers changed.

Similarly, Git would look for the lines:

And we are now learning about Git
So we are writing an example
Git is lovely!

As Git can find these lines, Git can erase the middle one.

If we changed one of these lines, say, changed And we are now learning about Git to And we are now learning about patches in Git, then Git would not be able to find the string above, and thus the patch would not apply.

Recap

In this post, you learned what a diff is, and the difference between a diff and a patch. You learned how to generate various patches using different switches for git diff.

You also learned what the output of git diff looks like, and how it is constructed. Ultimately, you learned how patches are applied, and specifically the importance of context.

Understanding diffs is a major milestone for understanding many other processes within Git - for example, merging or rebasing.

In future tutorials, you will use your knowledge from this post to dive into these other areas of Git.

About the Author

Omer Rosenbaum is Swimm’s Chief Technology Officer. He's the author of the Brief YouTube Channel. He's also a cyber training expert and founder of Checkpoint Security Academy. He's the author of Computer Networks (in Hebrew). You can find him on Twitter.

Additional References

Learn Wireshark – Computer Networking Tutorial

Omer Rosenbaum — Mon, 23 Jan 2023 23:35:33 +0000

In this post, you will learn about the single most important and useful tool in Computer Networks – Wireshark.

This post relies on basic knowledge of computer networks. Be sure to check my previous post about the five layers model if you need a refresher.

What is Wireshark?

Wireshark is a sniffer, as well as a packet analyzer.

What does that mean?

You can think of a sniffer as a measuring device. We use it to examine what’s going on inside a network cable, or in the air if we are dealing with a wireless network. A sniffer shows us the data that passes through our network card.

But Wireshark does more than that. A sniffer could just display a stream of bits - ones and zeroes, that the network card sees. Wireshark is also a packer analyzer that displays lots of meaningful data about the frames that it sees.

Wireshark is an open-source and free tool, and is widely used to analyze network traffic.

Wireshark can be helpful in many cases. It might be helpful for debugging problems in your network, for instance – if you can’t connect from one computer to another, and want to understand what’s going on.

It can also help programmers. For example, imagine that you were implementing a chat program between two clients, and something was not working. In order to understand what exactly is being sent, you may use Wireshark to see the data transmitted over the wire.

So, let’s get to know Wireshark.

How to Download and Install Wireshark

Start by downloading Wireshark from its official website:

https://www.wireshark.org/#download

Follow the instructions on the installer and you should be good to go.

How to Sniff Traffic with Wireshark

Launch Wireshark, and start by sniffing some data. For that, you can hit Ctrl+K (PC) or Cmd+K (Mac) to get the Capture Options window. Notice that you can reach this window in other ways. You can go to Capture->Options. Alternatively, you can click the Capture Options icon.

I encourage you to use keyboard shortcuts and get comfortable with them right from the start, as they'll allow you to save time and work more efficiently.

So, again, I’ve used Ctrl+K (or Cmd+K) and got this screen:

The Capture Options window in Wireshark (Source: Brief)

Here we can see a list of interfaces, and I happen to have quite a few. Which one is relevant? If you’re not sure at this point, you can look at the Traffic column, and see which interfaces currently have traffic.

Here we can see that Wi-Fi 3 has got traffic going through it, as the line is high. Select the relevant network interface, and then hit Enter, or click the button Start.

Let Wireshark sniff the network for a bit, and then stop the sniff using Ctrl+E / Cmd+E. Again, this can be achieved in other ways – such as going to Capture->Stop or clicking the Stop icon.

Consider the different sections:

Wireshark's sections (Source: Brief)

The section marked in red includes Wireshark’s menu, with all kinds of interesting options.

The main toolbar is marked in blue, providing quick access to some items from the menu.

Next, marked in green, is the display filter. We will get back to it shortly, as this is one of the most important features of Wireshark.

Then follows:

The Packet List Pane

The packet list pane is marked in orange. It displays a short summary of each packet captured.

(Note: the term Frame belongs to a sequence of bytes in the Data Link layer, while a Packet is a sequence of bytes from the Network layer. In this post I will use the terms interchangeably, though to be accurate, every packet is a frame, but not every frame is a packet, as there are frames that don't hold network layer data.)

As you can see in the image above, we have a few columns here:

NUMBER (No.) – The number of the packet in the capture file. This number won’t change, even if we use filters. This is just a sequential number – the first frame that you have sniffed gets the number 1, the second frame gets the number 2, and so on.

Time – The timestamp of the packet. It shows how much time has passed from the very first packet we have sniffed until we sniffed the packet in question. Therefore, the time for packet number 1 is always 0.

Source – The address where this packet is coming from. Don’t worry if you don’t understand the format of the addresses just yet, we will cover different addresses in future tutorials.

Destination – The address where this packet is going.

Protocol – The protocol name in a short version. This will be the top protocol – that is, the protocol of the highest layer.

Length – The length of each packet, in bytes.

Info – Additional information about the packet content. This changes according to the protocol.

By clicking on packets in this pane, you control what is displayed in the other two panes which I will now describe.

The Packet Details Pane

Click on one of the captured packets. In the example below I clicked on packet number 147:

Selecting a specific packet changes the packet details pane (Source: Brief)

Now, the packet details pane displays the packet selected in the packet list pane in more detail. You can see the layers here.

In the example above, we have Ethernet II as the second layer, IPv4 as the third layer, UDP as the fourth layer, and some data as a payload.

When we click on a specific layer, we actually see the header of that layer.

Notice that we don’t see the first layer on its own. As a reminder, the first layer is responsible for transmitting a single bit – 0 or 1 – over the network (if you need a refresher about the different layers, check out this post).

The packet bytes pane in Wireshark (Source: Brief)

Below the packet details pane, we have the packet bytes pane. It displays the data from the packet selected in the packet list pane. This is the actual data being sent over the wire. We can see the data in hexadecimal base, as well as ASCII form.

How to Use the Display Filter

Wireshark has many different functions, and today we will focus on one thing – the display filter.

As you can see, once you start sniffing data, you get a LOT of traffic. But you definitely don’t want to look at everything.

Recall the example from before – using Wireshark in order to debug a chat program that you’ve implemented. In that case, you would like to see the traffic related to the chat program only.

Let’s say I want to filter only messages sent by the source address of frame number 149 ( 192.168.1.3 ). I will cover IP addresses in future posts, but for now you can see that it consists four numbers, delimited by a dot:

The display filter in Wireshark (Source: Brief)

Now, even if you don’t know how to filter only packets sent from this IP address, you can use Wireshark to show you how it’s done.

For that, go to the right field we would like to filter – in this case, the source IP address. Then right click -> and choose filter -> Apply as Filter.

Applying a display filter (Source: Brief)

After applying the filter, you only see packets that have been sent from this address. Also, you can look at the display filter line and see the command used. In this way, you can learn about the display filter syntax (in this example, it is ip.src for the IP source address field):

Applying a display filter (Source: Brief)

Now, try to filter only packets that have been sent from this address, and to the address 172.217.16.142 (as in Frame 130 in the image above). How would you do that?

Well, you could go to the relevant field – in this case, the IP destination address. Now, right click -> Apply as Filter -> and select ...and Selected:

Applying a display filter (Source: Brief)

If you look at the display filter line after applying this filter:

Applying a display filter (Source: Brief)

You can also learn that you can use the && operand in order to perform and. You could also write the word and, instead, and get the same result.

Applying multiple conditions using && or and (Source: Brief)

How to Use Wireshark to Research the Ping Utility

Ping is a useful utility to check for remote servers’ connectivity.

This page explains how to use ping in Windows, and this page explains how to do that in OSX.

Now, we can try to ping

using the command line. By default, ping sends 4 requests and waits for a pong answer. If we want it to send a single request, we could use -n 1:

Using the command line to ping Google (Source: Brief)

You can see that Google has responded. The time it took for the message to return was 92 milliseconds. We will learn about the meaning of TTL in future posts.

Ping is useful to determine whether a remote service is available, and how fast it is to reach that service. If it takes a very long time to reach a reliable server such as google.com, we might have a connectivity problem.

Try it yourself

Now, try to use Wireshark to answer the following questions:

1) What protocol does the ping utility use?

2) Using only Wireshark, compute the RTT (Round Trip Time) – how long it took since your ping request was sent and until the ping reply was received?

Next, run the following command:

ping -n 1 -l 342 www.google.com

3) What is the main difference between the packet sent by this command, and the packet sent by the previous command? Where in Wireshark can you see this difference, inspecting the packets?

4) What is the content (data) provided in the ping request packet? What is the content provided in the ping response packet?

Let's solve it together

So the first question is:

What protocol does the ping utility use?

To answer that question, start sniffing in Wireshark, and simply run the ping command. Stop the sniff, and consider the packets pane:

Sniffing while running ping (source: Brief)

Wireshark marks the packets as Echo (ping) request and Echo (ping) reply.

Considering these packets, we can see they consist of Ethernet for the Data Link layer (though that may differ from one network to another), IPv4 as the Network layer, and then ICMP as the protocol for Ping itself. So the answer we found is: ICMP.

Next question:

Using only Wireshark, compute the Round Trip Time

Looking at the captured packets, we can see the Time column, and subtract the time of the Pong packet ( 7.888... ) from the time of the Ping packet ( 7.796...).

So in this case the RTT was: 92 ms. Of course, the value can be different when you run the ping utility.

What is the main difference between the packet sent by this command, and the packet sent by the previous command?

For question number 3, we are asked to run the following command:

ping -n 1 -l 342 www.google.com

Looking at the first run of ping, we can see the length of the packets are 74 bytes:

Sniffing while running ping (source: Brief)

Observing the packets sent after running ping with the -l 342 argument, we can see that the value is bigger:

Sniffing while running ping (source: Brief)

So the main difference is the amount of bytes sent as the data.

Question number four:

What is the content (data) provided in the ping request packet?

What is the content provided in the ping response packet?

Click on the request packet to observe the data sent:

Observing the data sent by the ping utility (source: Brief)

The answer for the ping request is a through w, over and over again.

Regarding the ping response – it is the same as the request.

Summary

Wireshark is a wonderful tool for anyone working with Computer Networks. It can help you understand how protocols work and also help you debug applications or network issues.

As you have seen, you can learn how things work by simply running Wireshark in the background while using them and then inspect the traffic. With this tool under your belt, the sky is the limit.

In future tutorials, we will also rely on our knowledge of Wireshark and use it to further understand various concepts in computer networks.

About the Author

Omer Rosenbaum is Swimm’s Chief Technology Officer. He's the author of the Brief YouTube Channel. He's also a cyber training expert and founder of Checkpoint Security Academy. He's the author of Computer Networks (in Hebrew). You can find him on Twitter.

Additional References

How to Handle Errors in Computer Networks

Omer Rosenbaum — Wed, 18 Jan 2023 16:05:43 +0000

There are some magical things about the Internet, and one thing in particular is that it works. In spite of so many obstacles, we can deliver our packets over the globe, and do so fast.

Even more specifically, one amazing thing about the Internet is its ability to handle errors.

What do I mean by errors? When a packet or a frame is received by a machine, we say it contains an error if the data that had been sent is not the data that was received. For instance, a single 1 was mistakenly received as a 0 after its transmission.

This can happen due to many different reasons. Perhaps there was some disturbance in the wire where the data was transmitted – say, a child rode her bicycle over the wire. Perhaps there was some collision in the air as many people transmitted at once. Maybe it was a device's error.

Regardless of the specific reason, you still get valid data on the Internet. Without handling errors, you may read the last sentence and instead of errors read errbbb. Weird, isn't it? So how does the Internet handle errors?

There are two main approaches for handling errors – detection, and correction. We shall start by describing detection, and then talk about correction.

What is Error Detection?

When dealing with error detection, we are looking for a boolean result – True, or False. Is the frame/packet valid, or not. That is all. We don’t want to know where the error occurred. If the frame is invalid, we will simply drop it.

So when the receiver receives a frame, they will determine whether an error has occurred. If the frame is valid, they will read it. If the frame contains errors - the receiver will drop it.

_Error Detection: we only want to know if the frame/packet is valid or not. (Source: Brief)_

One method for error detection is using a checksum. A common implementation of a checksum is called CRC – Cyclic Redundancy Check.

In this post we will not trouble ourselves with the mathematical implementation of CRCs in the real world (if you're interested, check out Wikipedia). Rather, we'll simply try to understand the concept. To do so, let’s implement a very simple checksum mechanism ourselves.

Consider a protocol for transmitting 10-digit phone numbers between endpoints. This protocol is extremely simple: each packet includes exactly 10 bytes, each one representing a digit. For example, a packet might include the following digits:

5551234567

_A packet with a payload of 10 digits (Source: Brief)_

For simplicity's sake, we will omit the headers of the packet and focus solely on the payload.

Now, we will add a checksum. Say that we add all the digits. So in this example, we would calculate 5 + 5 +5 +1+… all the way through 7. We would get 43. This would be our checksum value.

Now, the sender won’t only send the phone number, but also the checksum value right after it. In this example, the sender would send:

_The packet's data is followed by a checksum. (Source: Brief)_

Now, as the receiver, you can do the same thing. You will read the phone number, and calculate the checksum. You will add the digits, and get 43.

Since you've received the correct result (that is, your calculation based on the data matches the checksum value sent in the packet), you can assume that the frame is valid.

_The sender compares their calculated checksum value and the checksum in the packet. If the values match, the packet is assumed to be valid (Source: Brief)_

What happens in case of an error? 🤔

Let’s say, for instance, that the digit 2 was replaced by an 8. Now, even though the sender sent the same stream as before ( 555123456743 ), you, as the receiver, see something a bit different:

_A packet containing an error (Source: Brief)_

Now, you are calculating the checksum, adding all the digits. You get 49. Since this value is different from the checksum value specified in the original frame, 43, the frame is considered to be invalid and you drop it.

_The sender compares their calculated checksum value and the checksum in the packet. If the values don't match, the packet is assumed to be invalid (Source: Brief)_

Are there problems with this method? 🤔

Yes, there are. Consider, for example, what happens if there are two errors – and instead of the original stream ( 555123456743 ), you receive the following:

_A packet received with two errors, resulting in the stream 456123456743 (Source: Brief)_

What happens when you add the digits?

Even though the digits are not the same as the original packet, the checksum will remain correct, and the frame will be regarded as valid.

_Despite the errors, the checksum value happens to be correct, resulting in a false assumption that the packet is valid (Source: Brief)_

Real checksum functions, such as CRCs, are of course much better implemented than the one in our example – but in extremely rare cases, such problems may occur.

Notice that using this kind of method, error detection, we don’t know where the problem occurred, but only whether the frame is valid or not. If the checksum value is invalid, we assume that the frame is invalid and drop it.

What is Error Correction?

As mentioned earlier, detection is not the only way to handle errors. Another approach might be to find the error and correct it. How can we do that?

An extremely simple way would be to transmit the data many times – let’s say, three times. For example, the stream 5551234567 would be transmitted as follows:

_Sending the same data multiple times (Source: Brief)_

So we basically sent the data three times.

Now, in case of an error in one digit, the receiver can look at the other two digits, and choose the one that appears two times out of three.

So, for instance, if we had a problem and 2 was replaced with an 8, the receiver would get this stream:

_An error in one of the occurrences of the data (Source: Brief)_

Now, as a receiver, you can say: “I have 2, 8, 2… so it was probably 2 in the original message”.

Is this problematic? Well, in some rare cases, we might get the same error twice. So it is possible, even though unlikely, that two of the original twos have been received as eights.

So while the sender sent this stream:

_Sending the same data multiple times (Source: Brief)_

The first 2 was mistakenly read as an 8, and also the second 2 was received as an 8:

_Two identical errors; Rare, but possible (Source: Brief)_

Now, it looks as if the original message included an 8, and not a 2.

What can you do in order to lower the probability of such scenario?

The most simple solution would be to simply send the data even more times. Let’s say, five times. So now we duplicate all the data, and send it 5 times in total…

_Sending the data five(!) times (Source: Brief)_

Now, say that two errors occurred, and again two of the 2 digits were replaced with 8s.

_Two identical errors; Rare, but possible (Source: Brief)_

Clearly, it is very unlikely to get the same error twice, but even in this case, we still get 2 three times, so as the receiver you can tell, with a high probability, that the original message contained a 2, rather than an 8.

What's the Overhead?

Now would be a good time to introduce the term overhead. When we say overhead, we basically mean data or time needed to convey the actual message. Let’s first understand what this term means in general, and then consider it in the context of handling errors.

Let’s say that I have a lesson to teach in my university. My goal is to teach the lesson itself, which is also called the payload in that context – that is, the actual data or message I would like to convey.

In order to teach the lesson, or to convey the payload, I first have to physically get to the university – so I get out of my home, walk to the bus station, wait for the bus, take the bus, get off the bus, walk to the building, wait for the lesson to start – and only then do I actually get to teach the lesson.

This entire process is overhead that I have to pay in order to deliver the payload, in this case – to teach the lesson.

_Overhead and Payload are two extremely important terms in Computer Networks (Source: Brief)_

The same applies in computer networks. Our payload is the data, and there is always some overhead associated with sending it.

Back to Handling Errors

In the context here – sending the data three times, as suggested earlier, means that for every byte of payload we have two bytes of overhead. If we send the data five times, then for every byte of payload, we have four bytes of overhead. That’s a LOT!

Consider error detection, on the other hand. In our example protocol for sending phone numbers, how much overhead did we have?

Recall that for every ten-digit phone number, that is ten bytes, we included a two-digit checksum value. In other words, we had two bytes of overhead for ten bytes of payload. It is clear that in our example, error detection yields much smaller overhead in comparison to error correction.

_In the sample protocol, for every ten-digit phone number (ten bytes of payload), we included a two-digit checksum value (two bytes of overhead) (Source: Brief)_

There are better ways to achieve error correction with high accuracy than to simply send the data so many times, but they are more complicated and out of scope for this post. Even with very complicated error correction techniques, they still require lots of overhead when compared to error detection.

Also, notice that except for the bytes sent as overhead in case of error correction, error detection is much simpler.

Error Correction vs Error Detection – Which is Better?

We already concluded that error detection is simpler, and with a smaller payload compared to error correction.

So, when would we prefer error correction?

One case might be when we have a one-way link. That is, a network where we can only transfer data in one direction.

For example, say you have a secret agent that you need to send a message to. The agent knows that they need to look up to the sky at exactly midnight, and they will see a series of flashes indicating the secret message.

The secret agent cannot reply, or their location and identity will be revealed. In addition, you don’t want to send the message over and over again, as not to draw much attention, and to make it harder for someone to intercept the message.

In this case, you definitely want your agent to receive the exact message that you’ve sent. Consider a case where you want to send them the message “do not place the bomb”.

_A sensitive message for a secret agent (Source: Brief)_

Of course, you don’t want to risk the unfortunate scenario of the agent reading the message as “do now place the bomb”, due to an error.

_An error may change the meaning of the message substantially (Source: Brief)_

If you use error detection, the agent might be aware that the message they received is invalid in case of an error, but they won’t be able to tell you that they need you to send the message again. As you want the agent to be able to read your message correctly and without sending any data back to us, error correction is preferred.

So, one-way link is one case where we prefer error correction. What about other cases?

Sometimes you just can’t send the data again, perhaps because it has been erased from the memory of your machine. That is, the data is deleted right after it has been sent. In this case, you'd clearly prefer error correction, as sending the data again, as we would do with error detection, is just impossible.

Also, if sending the data again is possible, but extremely expensive, error correction may be preferable.

For example, if you send a message to the moon, say, with a spaceship – it might be really expensive to send it over again in case of an error. Using error correction, you send the data only once and the receiver should be able to deal with it, even if an error occurred.

_Cases where correction is preferred (Source: Brief)_

In general, we prefer error correction when retransmitting the data is costly or impossible.

When would we prefer error detection?

Well, in case we can retransmit the data, we usually prefer error detection since it comes with very little overhead compared to error correction. Especially, when sending the data is relatively cheap.

For example, on the Internet, if an error occurs when you send a frame, no problem – you can simply send it again!

For example, when I covered the Ethernet protocol in a previous post, I mentioned that Ethernet protocol uses change detection, namely CRC32 – that is, 32 bits (or 4 bytes) of a checksum for every frame.

Note that it doesn’t mean that error detection is simply better. It just better fits the Internet than error correction. As mentioned before, error correction is preferable in other cases.

Wrapping Up

In this tutorial, we discussed various methods for handling errors. We looked at error detection, where we only know whether a frame is valid or not. We also considered error correction, where the receiver can restore the correct value of an erroneous frame. We also introduced the term overhead.

We then understood why we use error detection on the Internet, rather than error correction. Stay tuned for more posts in this series about Computer Networks 💪🏻

About the Author

Omer Rosenbaum is Swimm’s Chief Technology Officer. He's the author of the Brief YouTube Channel. He's also a cyber training expert and founder of Checkpoint Security Academy. He's the author of Computer Networks (in Hebrew). You can find him on Twitter.

Additional Resources

How to Use Scapy – Python Networking Tool Explained

Omer Rosenbaum — Wed, 21 Dec 2022 21:02:17 +0000

In this post you will learn about an amazing tool named Scapy. Scapy is a Python library that enables us to send, sniff, and dissect network frames.

It is useful in a variety of use cases, one of which is to actually get some hands-on experience when you learn Computer Networks. Wouldn't it be great if, when learning about Ethernet, for example, you could create, send, sniff and parse Ethernet frames on your own? Scapy is the perfect tool for that.

In addition, you can use Scapy for creating networking-based applications, parsing network traffic to analyze data, and many other cases.

This post assumes you have some background knowledge in Computer Networks, for example about the layers model. It also assumes you have some basic Python knowledge.

What will you learn?

In this post we will start from the very basics – what Scapy is, and how to install it.

You will learn how to sniff data and parse it with Scapy, and how to display it in a meaningful manner.

You will also learn how to create frames or packets, and how to send them. Altogether, you should have a new powerful tool under your belt.

How to Install Scapy

To install Scapy, you can simply use pip install scapy.

If you run into trouble, simply follow the official documentation.

How to Use Scapy

For now, let’s open up the command line and type in scapy.

You should expect something like the following:

_Running Scapy from the CLI (Source: Brief)_

Note that the warning messages are fine.

Since this is a Python environment, dir, help, and any other Python function for information retrieval are available for you. Of course, you can always combine Python code with your Scapy scripts.

How to Work with Packets and Frames in Scapy

Packets and frames in Scapy are described by objects created by stacking different layers. So a packet can have a variable number of layers, but will always describe the sequence of bytes that have been sent (or are going to be sent) over the network.

Let's create a frame that consists of an Ethernet layer, with an IP layer on top:

_Stacking Layers (Source: Brief)_

Look how easy that is! We’ve used the / operator in order to stack the IP layer on top of the Ethernet layer.

Note that when looking at this object, it only tells us non-default values. The type of Ethernet is 0x800 (in hexadecimal base) as this is the type when an IP layer is overloaded.

Let's look more deeply at the fields of the packet:

_With the show method we can observe all fields of the frame (Source: Brief)_

Pretty cool! 😎

How to Sniff with Scapy

Scapy also allows us to sniff the network by running the sniff command, like so:

_Sniffing with the sniff command (Source: Brief)_

After running sniff with count=2, Scapy sniffs your network until 2 frames are received. Then it returns – and in this case, the variable packets will store the frames that have been received.

The return value of sniff can be treated as a list. Therefore packets[0] will contain the first packet received, and packets[1] will contain the second:

_The return value of sniff is an iterable, so it can be accessed as a list (Source: Brief)_

A helper function summary is available too and will provide minimal information regarding the packet collection:

_Using summary we can get some information of the packet collection (Source: Brief)_

When looking at a specific frame, every layer or field can be accessed in a very elegant way. For instance, in order to get the IP section of the packet, we can access it like so:

_Accessing a specific layer (and its payload) (Source: Brief)_

Note that this shows us everything from the IP layer and above (that is, the payload of the IP layer). Let's now observe the source Ethernet address of this frame:

_Accessing a specific field (Source: Brief)_

Nice and easy. Now, you will learn how to run a specific command for every frame that you sniff.

First, create the callback function that will be run on every packet. For example, a function that will just print the source Ethernet address of the received frame:

_Defining a callback function that receives a frame as its argument (Source: Brief)_

Now, we can pass this function to sniff, using the prn argument:

_Run a callback function on every sniffed frame (Source: Brief)_

The Ethernet addresses have been printed as a result of print_source_ethernet being executed, where every time, it receives a sniffed frame as an argument.
Note that you can write the same in Python using a lambda function, as follows:

_Define the callback function using lambda (Source: Brief)_

If you prefer to write an explicit function like the one we’ve written above, that’s perfectly fine.

We usually want to filter traffic that we receive – and look only at relevant frames. Scapy’s sniff function can take a filter function as an argument – that is, a function that will be executed on every frame, and return a boolean value – whether this frame is filtered or not.

For example, say we would like to filter only frames that are sent to broadcast. Let’s write a simple filtering function that does just that:

_A simple filtering function (Source: Brief)_

Now, we can use the lfilter parameter of sniff in order to filter the relevant frames:

_Filtering frames based on a filter function (Source: Brief)_

In order to clarify, let’s draw this process:

_The process of sniffing and filtering with lfilter (Source: Brief)_

A frame f is received by the network card. It is then transferred to lfilter(f). If the filter function returns False, f is discarded. If the filter returns True, then we execute the prn function on f.

So we can now combine these two arguments of sniff, namely lfilter and prn, and print the source address of every frame that is sent to the broadcast address. Let’s do this now using lambda:

_Combining lfilter and prn 💪🏻 (Source: Brief)_

This is equivalent to writing the following line, without lambda:

sniff(count=2, lfilter=is_broadcast_frame, prn=print_source_ethernet)

Readable, quick, and useful. Have you noticed that I love Scapy? 🥰

Alright, so far we’ve learnt how to sniff frames. When sniffing, we know how to filter only relevant frames, and how to execute a function on each filtered frame.

How to Create Frames in Scapy

To create a frame, simply create an Ethernet layer using Ether(). Then, stack additional layers on top of it. For instance, to stack an IP layer:

_Creating a frame with two stacked layers (Source: Brief)_

Alternatively, we can just add raw data, as follows:

_Using Raw data as the payload (Source: Brief)_

If you want to specify a specific value, for instance the destination address of the frame, you can do it when you initially create the frame, like so:

_Creating a frame and specifying specific values (Source: Brief)_

Or, we can modify the specific field after creation:

_Modifying specific values (Source: Brief)_

How can we look at the frame we’ve just created? One way is to observe a frame using show, as we’ve done above. Another way of looking at a frame is by looking at its byte stream, just like in Wireshark. You can do this using the hexdump function:

_Viewing the hexadecimal byte stream (Source: Brief)_

Well, even better – we can just look at it inside Wireshark! By running wireshark(frame).

How to Send Frames in Scapy

You can send frames using sendp, as follows:

_Sending frames with sendp (Source: Brief)_

Let's sniff in wireshark while sending the frame to make sure that it’s actually sent:

_Observing the frame we've sent using Wireshark (Source: Brief)_

Note that we use sendp only when we send an entire frame, using the second layer and above. If you want to send a packet including only the third layer and above, use send instead.

Recap

In this post you got to know an awesome tool called Scapy. You saw how you can sniff, how to filter packets, and how to run a function on sniffed packets. You also learned how to create and send frames.

About the Author

Omer Rosenbaum is Swimm’s Chief Technology Officer. He's the author of the Brief YouTube Channel. He's also a cyber training expert and founder of Checkpoint Security Academy. He's the author of Computer Networks (in Hebrew). You can find him on Twitter.

Additional Resources

Network Devices – How Hubs and Switches Work and How to Secure Them

Omer Rosenbaum — Thu, 27 Oct 2022 14:30:00 +0000

In a previous post I described every bit and byte of the Ethernet protocol. In this post you will learn about two network devices, how they work, and how this knowledge may be used by hackers.

How Classic Ethernet Works

Before describing the network devices, consider a network without special network devices. That is, a network using classic Ethernet where all computers are attached to a single cable.

_Four devices connected using classic Ethernet (Source: Brief)_

In this case, if computer A sends a message to another computer, for instance – B, the message is sent over the shared cable, and all devices receive it.

_With classic Ethernet, If A sends a message to B - all devices (except for A) receive this message (Source: Brief)_

Can you think of some problems with this network structure?

First, overload – all network frames are received by all computers. Let’s say A wants to send a frame to B. C also sees this frame, and has to realize that it is not destined to his address, and thus discard it. This process takes time and resources. The same process happens at machine D, of course.

Second, privacy – if C sees every message sent from A to B and vice versa, this means that the privacy is violated. We would rather have a network where only A and B see the messages sent between them.

Third, extensibility – this network is not really extensible. Assume that up to 10 computers can attach to this cable. What happens when you need to add one more computer? You'd have to replace the entire cable. This is expensive and inconvenient.

Well, the person who actually has to replace the cable is probably the I.T. person - you know, the one who makes sure that everything runs well in your network and is rarely noticed until something bad happens (at least when you work in an organization large enough to have I.T. people).

Just to be clear – we LOVE the I.T. person. We want their life to be good, we don’t want them to be running around buying cables all the time.

Fourth, collisions – let’s say A wants to send a message to B, and C wants to send a message to D. At the same time, both of them might start their transmission, and the messages will collide.

In this case, we get errors – much like the case where two people start to speak at the same time, and it is impossible to understand either of them.

Fifth, this network structure might lead to starvation – let’s say that A is transmitting a frame. If the other stations wish to avoid collisions, they will refrain from sending data. But now, machine A can keep on transmitting forever, thereby taking all the bandwidth to itself and not letting any other station speak. This is called starvation.

_Five major problems with classic Ethernet networks (Source: Brief)_

Well, this doesn’t seem like the best network, does it?

We'll now get to know network devices that help deal with these issues.

How Network Devices Solve These Problems

What is a Hub?

One device that solves only the extensibility issue is called a Hub. A hub is a device with multiple ports that single Ethernet cables are connected to:

_An Ethernet hub is a device with multiple ports, each connected to a single Ethernet cable (Source: Brief)_

So now, instead of having one cable with multiple ports with many computers attached to it, we have instead a single hub, and each computer is connected to it via a single cable. This makes the I.T. person's life much easier.

The hub simply takes the pulse it receives and multiplies it – that is, sends it to all other ports. For example, if A sends a frame to B, the hub will send this frame to B, C and D – all ports except A’s port.

The hub doesn’t understand Ethernet, and doesn’t know anything about MAC addresses. For the hub, all bits are just bits transmitted over the wire, and these bits should get to all other ends.

_A hub simply takes a bitstream and multiplies it to all ports but the source port (Source: Brief)_

Now, if you need to add a new computer to the network, you can simply connect it to the hub.

_To add a new device to the network, we simply connect it to the Hub (Source: Brief)_

What happens if the hub runs out of ports? No problem, we will connect it to another Hub, like so:

_In case you run out of ports, you can add another Hub (Source: Brief)_

Nice! This is a lot easier to maintain than classic Ethernet.

Yet, at least with classic hubs, all other issues still remain. Since all computers receive the frame sent from A to B, there is no privacy, the network is overloaded, collisions may occur, and the network is prone to starvation.

What we really want is a device that, when A sends a frame to B, forwards that frame to B and only B. This device is called a switch.

What is a Switch?

If all the stations are connected via a switch, and A sends a frame to B, only B receives it.

_With a Switch, if A sends a message to B - only B will receive it (Source: Brief)_

Notice that this means that all issues are indeed solved. The devices won’t be overloaded as every frame will get only to the relevant recipients. There are no privacy issues since, apart from the switch, only A and B see the frame. The network is easily extensible by plugging additional switches if needed.

_Similar to working with Hubs, the network is easily extensible by adding multiple Switches (Source: Brief)_

The switch can avoid collisions as every connection between a switch and an endpoint is a single collision domain – that is, the switch will refrain from sending more than one frame on a single wire at the same time.

_Every connection between the Switch and another device forms an independent collision domain (Source: Brief)_

Similarly, there will be no starvation as B and C can communicate with one another while A is sending data. Even if A keeps sending frames destined to the entire network, that is the broadcast address, the switch can allow messages sent by other hosts to be transferred in between.

But, how can this magical switch operate?

Let’s say we have just bought a brand new switch and plugged it into the network. A sends a frame destined to B. How does the switch know where computer B resides?

One option would be to manually configure the switch. That is, have a table mapping between a MAC address and the relevant port, and have someone manually configure that table.

_The Switch may hold a table mapping MAC addresses to physical ports (Source: Brief)_

When we say someone, we usually mean the I.T. person. And, well, we LOVE I.T. people. We wouldn’t want to make them do this tedious job every time.

In addition, I don’t know about you, but most people don’t usually have an I.T. person at home for every time they plug a device into their network.

Another option would be to send a special message from the switch to every port, and then the endpoints will reply with their MAC addresses. The major downside here is that we now have to make all devices aware of the switch. We need to change the devices’ behavior so they reply to that special message.

It would be so much better if the switch were just transparent – no endpoint would need to know that it’s there, but it would still do the job.

Apparently, this can indeed be achieved!

Consider this network, with a brand new switch that has just been added to the network. The switch stores a table, mapping a MAC address to a physical port. This table is empty.

_When a Switch joins a new network, the table mapping MAC addresses to physical ports is empty (Source: Brief)_

Now, A sends a frame to B.

The switch understands Ethernet, and can look at the Frame’s header and read the source address. Since this source address maps to “A”, and since the message has been sent from physical port number 2, the switch adds the mapping of A’s MAC address and port number 2 to its table.

_When machine A sends a frame, the Switch inspects the frame, reads the source address, and maps it with the corresponding physical port (Source: Brief)_

But what will the switch do with the frame? Well, for now, the switch doesn’t know where B resides, so the switch simply multiplies the frame and sends it to all ports, just like a hub would do. So for now, B, C and D all get the frame.

_Since the Switch's table doesn't include a record for B, a frame destined to B is actually sent to all ports but the source port - the same as a Hub would do (Source: Brief)_

Next, A sends another message to B. The switch looks at it, and already knows that A’s MAC address is plugged to port number 2. It still doesn’t know B, so this frame is sent to all other ports as well.

Now, C sends a frame to A. The switch looks at the source address, and adds the mapping between C’s MAC address and port number 5 to its table.

_Upon receiving a frame from C, the Switch parses its header, extracts the source address, and associates it with the corresponding physical port - port number 5 (Source: Brief)_

This time, since the frame is destined to A’s MAC address, and since the switch knows that address – the frame can be forwarded to port number 2, and port number 2 only. Yay! 👏🏻👏🏻👏🏻

Now, B sends a message to C. The switch creates a mapping between port number 7 and B’s MAC address, which appears at the source address field.

_The Switch keeps on learning the addresses gradually, filling in its internal mappings (Source: Brief)_

The switch can also forward the message to C, as it already knows C's address.

So, in general, the switch uses the source address field of Ethernet frames to dynamically learn what addresses reside behind every port.

Now, a question for you: Is it possible for two different addresses to map to a single port? For example, to have the address of computer A map to port number 3, and also have the address of computer B map to port number 3? 🤔

Well, the answer is yes. Consider the following network:

_A network diagram with five endpoints and three Switches (Source: Brief)_

Now, given that the switches know the network, when A sends a message to D, it will be sent to Switch 1, and then to Switch 2, and finally forwarded by Switch 2 to D. When Switch 2 sees the frame, what address does it see in the source address field?

The address of computer A, of course. Notice that switches are transparent, and never modify the MAC addresses. So Switch 2 learns that the MAC address of computer A resides behind port number 3.

Next, when computer B sends a frame to computer C, this message will also be transferred via switch 1 and then switch 2. So now, switch 2 learns that the MAC address of computer B resides behind port number 3 as well. So, in this case, both the MAC address of A and that of B reside behind port number 3.

_Given this network diagram, switch 2 registers both the MAC address of A as well as that of B - with port number 3 (Source: Brief)_

NOTE that a switch is not an additional hop! We are not talking about routing here. As we’ve said earlier, a switch is a transparent device. From the endpoints’ perspective, there is no switch – A “feels” as if it were directly connected to B, C and D.

All devices that are connected via one hop are said to be in the same network segment. So here, all computers and switches – A, B, C, D, switch 1 and switch 2 – all reside within the same segment.

In the resources section below, I’ve added a link to an exercise about hubs and switches. You are welcome to solve it in order to make sure everything is clear. If you have any questions, feel free to reach out 😊

Interim Summary

So far you learned about two network devices. First, a hub, which is basically a first layer device. That is, it only transmits bits from one port to other ports, without understanding any protocols.

Second, you got to know a second layer network device, namely a switch, which already "understands" the Ethernet protocol and MAC addresses. It uses that knowledge in order to transfer frames only to relevant ports, at least once it knows the network.

Security Twist 😈

Now that you understand how hubs and switches work under the hood, it's time to consider their security implications.

Assume that I am connected to a certain Ethernet segment, and you run on computer A. B sends a message to C. Is it possible for you to see that message?

_Four PCs, B is sending a frame to C (Source: Brief)_

In case the computers are connected via a hub, you certainly will see the message, as the hub simply forwards the frame to all ports (except for the source port) regardless of the destination address.

_A hub would simply multiply the frame and send it to A, C and D (Source: Brief)_

Furthermore, if the computers are connected via a switch, but the switch has not yet learned the address of the destination, this message will also be sent to your port – and, in general to all ports other than the source port, just like a hub would act.

_A new switch acts just like a hub until it learns the destination address (Source: Brief)_

So, in these cases, your network card will receive the frames, but will it handle them?

As I covered in a previous tutorial, the first field of an Ethernet frame is the destination address. By default, the network card will discard frames that are not destined to its address, or to a group which its system belongs to, such as the broadcast address.

_Ethernet frame structure - the devices first consider the destination address (Source: Brief)_

So, by default, if your network card happens to receive a frame that was not destined to it, the frame will be discarded. This is exactly where promiscuous mode comes in handy. When the network card is in promiscuous mode, it will not discard frames based on their destination MAC addresses.

Now, consider a network with a switch, and that switch has already learned all addresses of the network, thereby achieving privacy.

Let’s say that a malicious person works from computer C, and wants to see the communication being sent to computer B, even though the switch forwards those frames to B only.

_A network with a switch that has already learned the MAC addresses and their corresponding ports. Can a malicious person see private communication? (Source: Brief)_

Can the malicious person do something in order to steal the data?

Well, the malicious person can pretend that they have B’s address. That is, the malicious person will send a frame with the source address of B. It doesn’t really matter what the destination address of that frame would be.

_The malicious person sends a frame and impersonates B by specifying B's MAC address as the source address of the frame (Source: Brief)_

Now, the switch sees a frame being sent from B’s address and from C’s port, in our diagram, port 5, and changes the mapping of B’s address to port 5.

_As a result, the Switch changes the port associated with B's address (Source: Brief)_

As I mentioned earlier, it is indeed possible to have two different MAC addresses map to the same port number (for instance in case of an additional switch that connects the devices that have these addresses). But it is not possible to have B’s address mapped to two different ports.

_As far as the Switch is concerned, B and C may indeed both be attached to it via port 5, perhaps through another Switch (Source: Brief)_

Now, if A sends a message to B, it will actually get to C, but not to B! 😨

This technique is called MAC SPOOFING. The malicious entity is said to spoof B’s MAC address.

Is this technique very useful for the attacker? 🤔

Well, not really. Once B sends any frame at all to the network, the switch will replace the entry for B’s MAC address to that of the correct port number. So, for the attacker to keep receiving data, they will have to keep sending more frames on B’s behalf, thereby causing the switch to rewrite the table entry again and again.

This way, C will send a frame using B’s address, and the switch will map B’s MAC address to C’s port. Then, B will send a frame, and the switch will map B’s MAC address to B’s port again.

_Once B send any frame, the Switch will overwrite its entry and the original value will be restored (Source: Brief)_

Hence, B will receive some of the traffic, and this attack is easily noticeable.

There are many ways to defend a switch from such attacks. One would be to set the port with a maximum number of MAC addresses that are attached to it. For instance, if no other switch is supposed to be connected to a certain port, the maximum number of linked MAC addresses can be set to one.

How cool is that?! By understanding how a switch operates, we are able to estimate security issues that stem from its way of operation, as well as relevant countermeasures. 🤯

Conclusion

In this post you learned about two important network devices, a hub and a switch.

You learned that a hub simply multiplies the bitstream it receives to all ports other than the port that received the bitstream, whereas a switch forwards the frame only to the right port (once it has learned the network). You also learned how switches are able to achieve this ability automatically.

Lastly, you learned about a security problem that arises from the way switches operate, and how it may be mitigated.

About the Author

Omer Rosenbaum is Swimm’s Chief Technology Officer. He's the author of the Brief YouTube Channel. He's also a cyber training expert and founder of Checkpoint Security Academy. He's the author of Computer Networks (in Hebrew). You can find him on Twitter.

Additional Resources

How the Ethernet Protocol Works – A Complete Guide

Omer Rosenbaum — Fri, 21 Oct 2022 17:12:40 +0000

Whether you’ve been aware of it or not, you’ve probably used the Ethernet in the past. Does this cable look familiar?

_(Source: Wikipedia)_

Ethernet is extremely popular, and is the most widely used Data Link Layer protocol, at least where the devices are linked by physical cables (rather than wireless).

If you need a reminder about the Data Link Layer and its role within the Layers Model, check out my previous post.

In this tutorial, you will learn everything about Ethernet – its history, as well as every bit and byte of the Ethernet frame. You will also get to know how protocols are formed, why it is so hard to change them after they are published, and what lessons can be learned for other protocols.

Here's what we'll cover:

Some Ethernet History
Ethernet Frame Overview
– Before the frame – preamble (8 bytes)
– Destination Address and Source Address (6 bytes each)
– Type / Length field – Ethernet II (Type) (2 bytes)
– Data and Pad (46-1500 bytes)
– Checksum – CRC32 (4 bytes)
– The Problem with the Type / Length Field
How Ethernet Addresses Work
– Unicast and Multicast Bits
– Globally Unique / Locally Administered Bit
Why Does an Ethernet Frame Have a Minimum Length?
– How are Collisions Handled in Ethernet?
Conclusion

Some Ethernet History

The first version of Ethernet was implemented in 1976. In 1978 a second version was published by DEC, Intel, and Xerox who worked together to publish DIX (which stands for DEC, Intel and Xerox). This was also called "Ethernet II".

In 1983, with a change that we will discuss soon, a new Ethernet version was released – the IEEE 802.3 standard, by the IEEE standards association.

Both Ethernet II and IEEE 802.3 are widely used, so we will cover them both. As you will see, they are almost identical. Usually, both are simply referred to as “Ethernet”.

For this tutorial, in order to be precise about what we mean, I will explicitly state whether I'm talking about Ethernet II or IEEE 802.3.

_The versions of Ethernet (Source: Brief)_

Ethernet Frame Overview

Let's consider the Ethernet Frame format:

_Ethernet Frame Header and Trailer (Source: Brief)_

Before the Frame – Preamble (8 bytes)

First comes a Preamble consisting of 8 bytes, each containing the bit pattern of alternating 1s and 0s, that is, 10101010.

In Ethernet II, all 8 bytes had this pattern. In 802.3, the seven first bytes carry the value 10101010, yet the last bit of the last byte is set to 1, so the byte carries the value of 1010101**1**.

This last byte is called the Start of Frame. The last two 1 bits tell the receiver that the rest of the frame is about to start.

Sending this bit pattern before a new frame allows devices on the network to easily synchronize their receiver clocks. Note that the preamble is not really a part of the actual frame – it only precedes every frame, and thus you won't see it on many diagrams of the Ethernet protocol.

_Ethernet Preamble (Source: Brief)_

Destination Address and Source Address (6 bytes each)

Next, we have two addresses, each consisting of 6 bytes. I'll describe Ethernet Addresses in more detail later on in this post, but for now, let's notice that a frame starts with a destination address, followed by the source address.

Why would the frame start with the destination address? Is there a reason for that?

Well, there is. The very first thing a device is likely to do with a frame it has received is to check whether this frame is destined to it, or not. If the frame is not destined to this device, it can be simply dropped. Therefore, the destination address comes in first.

Why is the source address important? Well, to know to whom the receiver should send a reply, if necessary. This source address also plays a role in the way some network devices are implemented, as we will see in future posts.

Type / Length field – Ethernet II (Type) (2 bytes)

Next comes a quite problematic field, called the Type or Length field.

In Ethernet II, this field is called Type, and tells the receiver what payload this frame carries.

For instance, if this frame carries an IP layer (that is, the data of the Ethernet layer is an IP packet), then the receiving network card should forward the frame’s payload to the IP handler. If the frame’s payload is ARP, then the ARP handler should deal with it.

By handler I mean the code that handles this protocol, for instance the code that parses ARP.

We will come back to the need for Length and how it is dealt within IEEE 802.3 shortly.

_In Ethernet II, the Type field carries the type of the payload (Source: Brief)_

Data and Pad (46-1500 bytes)

After this field, we get up to 1500 bytes of Data. This number was chosen because RAM was expensive back in 1978, and a receiver would have needed more RAM if the frame had been bigger.

This means that if the third layer wants to send more than 1500 bytes of data over Ethernet, it must be sent across multiple frames.

There is also a minimum length of data, which is 46 bytes. Together with the other fields of the frame, the minimum length of an Ethernet frame is 64 bytes in total.

Why would we need a minimum frame length? We will discuss this in a subsequent section.

For now, given that we have a minimum length for an Ethernet frame, what happens if the sender wants to send a very short message, let’s say just one byte?

In that case, the sender has to pad the message, for instance with 0s until reaching the minimum length. For example, if the sender wants to send only 1 byte of data, such as the letter A, they will have to add 45 bytes of 0s.

_46-1500 bytes of data, with padding if needed (Source: Brief)_

Checksum – CRC32 (4 bytes)

Last but not least, we have a Checksum. This is a 32-bit CRC checksum, used to determine whether the bits of the frame have been received correctly. In case of an error, the frame is dropped.

The CRC is computed on the entire frame – that is, including the header. Notice that it doesn’t include the preamble, as it is not really a part of the frame.

When we use CRC-32 for the checksum, we set a fixed overhead of 32 bits, or 4 bytes, regardless of the length of the data. In other words, if we send only 1 byte of data, we get a 32-bit checksum, and if we send a thousand bytes of data – we still get 32-bits of checksum.

The Problem with the Type / Length Field

Earlier, we mentioned that the Data field has to be at least 46 bytes long, and if not, we pad it. For simplicity’s sake, let’s assume we pad with 0s, as the standard indicates.

Well, we actually have a problem here.

Let’s say the sender wants to send a single byte, consisting of the character A. So they will send an A followed by 45 0s.

What happens in case the sender wants to send A and zero? That is, the data actually consists of A0. In this case, they would also send an A, followed by 45 0s. But this time, the first zero is actually part of the data, and not the padding.

_Whether the sender would like to send A as data or A0 as the data, due to padding the frame consists of A and 45 0s (Source: Brief)_

As a receiver, you'd need a way to differentiate these cases, and understand which bytes belong to the padding, and which bytes belong to the data, in case of a short frame.

Ethernet II dealt with this problem by… Well, not handling it. That is, the third layer will receive the data and the padding, which would be an A followed by 45 0s in this example. It will then have to figure out on its own which bytes belong to the data and which don’t.

This is doable, of course, if the third layer includes a length field. However, this solution is far from elegant – why would the third layer deal with a padding problem that should be dealt with by the second layer?

This is a clear violation of our layers model (if you would like to see an overview about the Layers Model, refer to this tutorial).

For this reason, IEEE decided to change the Type field into a Length field in IEEE 802.3 . So, for example, a frame carrying a single byte of data, A, will have the Length field set to 1, whereas a fame carrying two bytes of data, A0, will have the Length field set to 2.

_In 802.3, the Length field sets the data apart from the padding (Source: Brief)_

This is an elegant solution, but now two issues arise:

First, if you receive an Ethernet frame, how do you know if it’s an Ethernet II frame, where this field means Type, or an IEEE 802.3 frame, where this field means length?

Second, what happens with the Type field? How would the receiver know what protocol is carried inside the frame?

Let's start with the first question. Just to clarify, by the time IEEE 802.3 was published, many Ethernet cards had already been in use. People didn’t want to replace their network cards just because a new standard was published.

Think about it, would you want to buy a new network card? Or perhaps your friends who are not programmers – would they get a new card as someone told them that "the internet geeks" decided that there was "a new standard" (whatever that means?).

The solution was to allow both Ethernet II and IEEE 802.3 to operate on the same network.

Fortunately, all the Type values used at that time had greater values than 1500. The solution is thus straightforward: in case this field has a value less than or equal to 1500, it actually means Length. In case it has a value greater than or equal to 1536, it means Type. The values in between currently have no meaning.

_The Type/Length field is divided: values equal to or lower than 1500 are Length values, and values equal to or greater than 1536 are Type values. (Source: Brief)_

For example, if we see a frame where the value of this field is 400, it is clear that we have an IEEE 802.3 frame, which is 400 bytes long.

Now you try: in case we see a frame where this field is set to 20, is it an Ethernet II frame or IEEE 802.3 frame?

Indeed, this is an IEEE 802.3 frame, which has 20 bytes of data, and thus 26 bytes of padding. And… in case we see a frame where this field is set to 2000?

In this case we know that this is an Ethernet II frame, and 2000 is the Type.

So this is how we know whether we are dealing with an Ethernet 2 or an IEEE 802.3 frame.

Next, how does an IEEE 802.3 frame contain the Type information? That is, given that IEEE 802.3 overrode the Type field, there was no way for the receiver to figure out what to do with an incoming frame. Thus, IEEE 802.3 adds another header of the 802.2 LLC (Logical Link Control) protocol right before the data. This header conveys the type information.

So an IEEE 802.3 frame will have a destination address field, then a source field, then a length field, and then an LLC header, followed by the data and the checksum.

Wait, wasn't IEEE 802.3 published in 1983? Why is it relevant? 🤔

As mentioned beforehand, in 1978, Ethernet II was published. Not so long later, in 1983, a new format came out – and its authors allowed for backward compatibility, probably believing that in a few years, all devices would be upgraded to the new standard.

Oh, were they wrong.

If you check your own network (given that you are connected to an Ethernet one), I bet you will see Ethernet II frames.

Your device probably supports both versions, but by default it will transmit Ethernet II frames, rather than 802.3. After all, it is guaranteed that any device connected to an Ethernet network can read Ethernet II frames, and it's not guaranteed that the device can read 802.3 ones. If Ethernet II works, why not use it?

All third-layer protocols had to account for the fact that Ethernet doesn't solve the problem of differentiating data from padding. So if all protocols already deal with that, why don't we just...keep things the way they are?

Endpoint devices (such as personal computers) almost always communicate over Ethernet II. IEEE 802.3 is also very common, though, and it's used by default on most modern network devices (such as switches).

This story actually entails a really important lesson.

It is very, very hard to replace protocols after the fact, especially when they are implemented on hardware devices (such as network cards).

What's an Interpacket Gap?

After an Ethernet frame is sent, transmitters wait a very short period of time before transmitting the next frame, in order to allow the receiver to know that the transmission of a frame is over. This idle time between frames is called the “Interpacket gap”.

How Ethernet Addresses Work

Every Ethernet frame carries two addresses – first, the destination, and second, the source. We mentioned that the destination address appears first so the receiver will be able to tell whether the frame is relevant for it. If not, the frame will be discarded.

_Ethernet addresses within the Ethernet Frame (Source: Brief)_

What does an Ethernet address look like?

An Ethernet address consists of 6 bytes – that is, 48 bits. Usually, they are presented in hexadecimal base, delimited either by dashes or colons, as you can see in these examples:

_Two representations of the same Ethernet Address (Source: Brief)_

00:01:42:a9:c2:dd
00-01-42-a9-c2-dd

These are two representations of the exact same Ethernet address, and there is no real difference between the two.

In general, Ethernet addresses are supposed to be globally unique. That is, no two Ethernet devices share the same address (at least, in theory).

The first 3 bytes of any address is called the OUI – Organizationally Unique Identifier. To make sure the addresses are unique, IEEE assigns these OUIs to various manufacturers, such as Dell, HP or IBM.

This part of the address is also called the Vendor ID (with the exception of the two least significant bits, as we will see). Then, the manufacturers assign the remaining 3 bytes to specific hosts. This part is also called the Host ID.

_The most significant 3 bytes are the Vendor ID, and the least significant 3 bytes are the Host ID (Source: Brief)_

For example, the OUI 00:01:42 belongs to Cisco. Now, Cisco can manufacture a network card and assign it the address 00:01:42:00:00:01. Next, it can manufacture another card and assign it the address 00:01:42:00:00:02, and so on. These two addresses share the same Vendor ID, but have different Host IDs.

Since a single OUI leaves 3 bytes to be used for the host IDs, we have 2^24 host IDs per OUI – that is, 16,777,216 host IDs. Of course, big manufacturers need many more addresses, and thus they are assigned additional OUIs. For example, 00:01:64 is another OUI that belongs to Cisco.

Unicast and Multicast Bits

Ethernet addresses also consist of two special bits.

The first special bit indicates whether the address is a unicast or a multicast address. Unicast means that the address represents a single device. Multicast addresses represent a group of devices – such as all printers on the network, or all devices in the same local network.

The bit representing whether the address is unicast or multicast is the least significant bit within the most significant byte. Wait, what?

Consider the following Ethernet address:

06:b2:d9:a2:32:9e

The most significant byte is 06.

Let’s convert this to binary:

00000110

Now we look at the least significant bit – that is, this 0:

_When the least significant bit within the most significant byte is set to 0, this is a Unicast address (Source: Brief)_

This bit is off. This means that this is a unicast address. In other words, it belongs to a single device, such as a computer’s network card.

Let’s consider another address:

11:c0:ff:ee:d8:ab

The most significant byte is 11 (in hexadecimal base).

Let’s convert this to binary:

00010001

_When the least significant bit within the most significant byte is set to 1, this is a Multicast address (Source: Brief)_

The least significant bit is this one. Since it is on, we can tell that this is a multicast address. That is, it’s an address of a group. You can send a frame to this address, and all devices that belong to this group will consider the frame as sent to them.

One very famous multicast address is called the broadcast address, that is – the group that contains all machines. The address of this group is:

FF:FF:FF:FF:FF:FF
In other words, the address where all bits are on.

All the machines are part of the broadcast group.

Globally Unique / Locally Administered Bit

The second special bit indicates whether the address is indeed globally unique. This bit is the second least significant bit within the most significant byte. Um, what?

Well, again, consider the first address from before:

06:b2:d9:a2:32:9e

The first byte is 06.

Converted to binary, we get:

00000110

So the second least significant bit is the one right here:

_When the second least significant bit within the most significant byte is set to 1, this address is not globally unique (Source: Brief)_

This bit is on, and thus we know that this address is actually not globally unique. IEEE will never assign this address to any vendor. So what is this address? Well, in this case it’s just one that I’ve made up. If I wanted to, I could assign it to a specific device. The fact that this bit is on declares that it is not globally unique.

Consider another address:

00:01:42:a9:c2:dd

The first byte is 00, so the second least significant bit is 0.

This is indeed a globally unique address, assigned to Cisco.

Ethernet Addresses – Recap

So, all in all, an Ethernet address has two main parts: The vendor ID, and the host ID.

There are also two special bits: the least significant bit within the most significant byte states whether the address is unicast or multicast. The second least significant bit within the most significant byte states whether the address is globally unique.

Why Does an Ethernet Frame Have a Minimum Length?

This is more of a "bonus" part of this post, and concerns collisions. Collisions is a very interesting topic, but since this post focuses on the Ethernet protocol, collisions will not be of our focus. I will therefore address this issue just briefly. While it's not crucial to understand in order to understand Ethernet frames, I promised a complete overview of the Ethernet protocol.

In the overview, I mentioned that an Ethernet frame consists of minimum 46 bytes of data and maximum 1500 of data. I already explained why we have that maximum limit, but what about the minimum?

To simplify our discussion, let's consider a network using classic Ethernet where all computers are attached to a single cable.

_A "classic Ethernet" network with four devices connected via a single cable (Source: Brief)_

Let’s say A wants to send a message to B, and C wants to send a message to D. Let’s say that while A is transmitting its frame, C is also transmitting its frame. In this case, the frames will collide.

_In case two devices transmit data on the same time - their frames will collide (Source: Brief)_

When this happens, we get errors – much like the case where two people start to speak in the same time, and it is impossible to understand either of them.

How are collisions handled in Ethernet?

Ethernet uses two main mechanisms to deal with collisions. The first is called CSMA, which stands for Carrier Sense Multiple Access. This basically means that when a station wants to transmit data, it first senses the channel to see if anyone else is transmitting by checking the signal level of the line. If the channel is in use, the station will wait and try again.

So, if A is transmitting, and C wants to send data, C will wait until A finishes its transmission before starting to transmit.

This is just like the case in a human conversation, where one person waits until the other stops talking, and only then does that person talk.

Yet, just like the case where two people might start talking at the same time, two Ethernet machines might start transmitting data at the same time. In this case, CD – Collision Detection – comes into play. Collision Detection means that the transmitting devices detect the fact that a collision has occurred. This is achieved by listening to the channel while transmitting.

For example, assume that station A transmits the bit stream 11001010. While transmitting, A is also listening to the channel. If no collision occurred, A would also read the signal 11001010 from the line.

_With Collision Detection, A is listening to the channel while transmitting data. In case no collision occurred, A will sense exactly the bitstream it has sent (Source: Brief)_

If, however, a collision occurred, say with a frame sent by C, then A would read something different from the line – for instance, 11011010. This way, machine A realizes that its frame has collided.

_With Collision Detection, A is listening to the channel while transmitting data. In case of a collision, A reads a different bitstream than that it has sent (Source: Brief)_

Machine A can realize that a collision has occurred even before it finished transmitting the frame. Then, machine A stops transmitting and issues a JAM signal to tell the other station that a collision has occurred. As a result, both stations stop transmitting and wait a random interval of time before trying to submit again.

The amount of time that the stations wait increases with the number of collisions in the network. So on the first collision, A and C wait for a relatively short amount of time before transmitting again. If another collision occurs, they might wait longer.

_After a collision occurs, the amount of time that the stations wait increases with the number of collisions in the network (Source: Brief)_

Now, back to Ethernet. Ethernet requires that valid frames must be at least 64 bytes long, from destination address to checksum, including both. So, that data has to be at least 46 bytes long. If the frame is too short, then it must be padded.

_The minimum length of an Ethernet frame consists of 46 bytes of data, or 64 bytes overall (Source: Brief)_

One reason for having this minimum is directly related to the collision detection mechanism stated above.

Let's consider the following scenario. Host A wants to transmit a really really short frame to B, a frame that is only 1 byte long. I am exaggerating of course, this can’t really happen in Ethernet, but it will be helpful for the explanation.

Host A transmits this frame, which consists of 8 1s. Then, A listens to the channel while transmitting, and also reads 8 1s from it, reaching the conclusion that the frame has been transmitted successfully.

However, before the frame reaches the other end of the network, D starts transmitting a very short frame, one byte long, consisting of 8 0s. D listens to the channel while transmitting, and also reads 8 0s from it, concluding that the frame has been transmitted successfully.

_Both A and D send a really short frame, and they finish transmitting without realizing a collision is liable to take place (Source: Brief)_

Now, these two really short frames collide. Yet, neither A nor D are aware of this collision, as they have already concluded that the frame has been successfully delivered.

In order to avoid such cases, the frame must be long enough to prevent a station from completing its transmission before the first bit of the frame reaches the far end of the line. Having a minimum length for Ethernet frames solves this issue.

This was a very short discussion of collisions. If you’d like to know more about this topic, refer to the "additional resources" section below.

Conclusion

In this tutorial, we covered every bit and byte of the Ethernet protocol. You should now have a good understanding of this protocol, as well as a reference to consult when needed.

About the Author

Omer Rosenbaum is Swimm’s Chief Technology Officer. He's the author of the Brief YouTube Channel. He's also a cyber training expert and founder of Checkpoint Security Academy. He's the author of Computer Networks (in Hebrew). You can find him on Twitter.

Additional References

What is the Five Layers Model? The Framework of the Internet Explained

Omer Rosenbaum — Mon, 17 Oct 2022 13:37:19 +0000

Computer Networks are a beautiful, amazing topic. Networks involve so much knowledge from different fields, from physics to algorithms.

When dealing with Computer Networks, there is one framework that puts everything into place – and that is the layers model.

In this post you'll learn why we need layers, as well as what the five layers model is. You will also understand the role of each layer in this model.

Why Layers?

Imagine you are given the task to design and implement the Internet! Where do you start? What do we actually want from a network, and an important one such as the Internet?

Well, we actually want quite a lot of things. To name a few:

We want it to be fast – that is, allow fast communication. We don’t want to wait long for a message to get from one host to another.
It should also be reliable – when sending a message, we want the receiver to actually receive it.
The network should be extendable – that is, allow more devices to join. We wouldn’t want to start with two computers, and then not bee able to add a third one.
The network should support different devices and connections – it should be able to connect a wired PC, wireless laptop, and a cellphone, for example.

And this is just a partial list.

So, how do we go about implementing the internet when we want to achieve so many different things?

Computer Networks are complex (Source: XKCD)

In order to simplify things and make networks flexible, the communication is divided into layers.

Each layer has its own responsibility. It provides services to an upper layer, and uses services provided by a lower layer.

Consider an example network consisting of three devices:

_An example network with three devices (Source: Brief)_

We have two layers:

Layer Alpha is responsible for transmitting data between hosts that are directly connected to each other. In the diagram above, it's between hosts A and B, or between hosts B and C.

Layer Beta is responsible for transmitting data between distant hosts. In the diagram, it's between hosts A and C.

What did we gain from this division? We gained a lot of flexibility.

Each layer can be developed and implemented by different people. The upper layer doesn’t care about the implementation of the lower layer, and vice versa.

For instance, the connection between hosts A and B could be a WiFi connection, while the connection between B and C could consist of a carrier pigeon. These are (completely) different implementations of Layer Alpha.

Notice that this way also enables us to have different specializations and expertise – an expert in training carrier pigeons does not necessarily have to be qualified at building solid WiFi network cards, or vice versa.

_The Alpha Layer may have different implementations on the same network (Source: Brief)_

Developers of Layer Beta don’t need to bother themselves with this difference. At this layer, host A needs to know that in order to reach host C, it first needs to send his message to host B, rather than, say, host D. Then, host B will forward it to host C.

This way, Layer Beta is only responsible for finding the route to send the message. It uses the service provided by Layer Alpha – transmitting data between directly connected hosts.

In general, networks are very complicated, and have various requirements. Dividing the communication into layers will allow us to simplify things and make communication more flexible.

Now that you understand why we need layers, we can go on to learn about the layers that are actually used in networks.

What is the Five Layers Model?

There have been a few layer models proposed along the years – most notably, the five layers model, the 7 layers model (aka OSI model), or the 4 layers model (aka the TCP/IP model).

They are way more similar than different, and I choose to focus on the five layers model as it is the most practical of all – and best describes the way the Internet actually works.

The First Layer – The Physical Layer

The first layer is responsible for transmitting a single bit – 0 or 1 – over the network.

To get some intuition as to what this layer is responsible for, consider the time of transmission. Assume that we have some kind of cable to transmit our data, and we use +5 Voltage to transmit 1, and -5 Voltage to transmit 0. What bits does the following diagram represent?

_A physical layer implementation encoding 1 as +5 Voltage and 0 as -5 Voltage (Source: Brief)_

Well, it might be 1001. That is the case if it takes this long to transmit a single bit (demonstrated by the dashed orange line in the diagram below):

_An example bitstream encoded by this signal (Source: Brief)_

However, it might also represent other bit streams. For instance, if it only takes half the time to transmit a single bit (demonstrated by the dashed green line below), then the bit stream might be 11000011:

_Another possible bitstream encoded by the same signal (Source: Brief)_

The difference lies in the time dedicated for transmitting a single bit. This is called the bitrate – that is, the number of bits that are conveyed per unit of time.

Of course, achieving a high bitrate is preferable, as it means we can send many bits in a short timeframe. But it is hard to achieve high bitrates without getting many errors.

This is only one of the things that the first layer needs to take into consideration. The important thing for now is the goal of this layer: to transmit and receive a single bit.

The Second Layer – The Data Link Layer

The second layer is responsible for transmitting data between two hosts that are directly linked, despite possible errors.

What do we mean by “directly linked”? For now, imagine that there is no device in between the two devices. So, if we have two computers here – computer A and computer B, and they are connected via computer M – then computer A and computer B are NOT directly linked. But computer A and computer M are directly linked, and so are computer M and computer B.

_Two remote hosts connected via another device (Source: Brief)_

Another way to put it is that computer A and computer M are one hop away from one another, whereas computer A and computer B are two hops away.

That is, in order to get from computer A to computer B we need two “hops” – one hop from A to M, and another hop from M to B.

_Every direct connection is called a Hop (Source: Brief)_

Going back to the second layer's responsibility – we mentioned it is responsible for transmitting data between two hosts that are directly linked, despite possible errors.

What do we mean by errors? The physical layer might provide erroneous data. For example, 1 instead of 0. So a stream of bits such as 000110, might be received as 001110.

Many reasons might cause these kind of errors. For instance, we can think of a truck literally running over the wire where the bits are transmitted, causing some problem. Regardless of the reason, the second layer must handle the communication despite these errors.

The second layer sends data in datagrams, that is, in chunks. Datagrams in this layer are called Frames. Frames will usually contain MAC addresses, which are physical addresses, one identifying the sender, and another identifying the receiver.

Why would we need a MAC address?

First, the receiving devices would like to know whether the frame is intended for them. The receiver wouldn’t like to waste precious time reading data intended for someone else. If the frame contains a MAC address that doesn’t belong to a receiver's device, that device can simply ignore this frame.

Second, for privacy reasons - we would like messages to arrive only at intended receivers, so only they can read the data.

Third, the sender would like the receiver to know who sent the frame. That way, the receiver will be able to send their response back to the sender, and not to someone else.

Note that we would like these addresses to be unique. That is, we want one address to identify a single device. That way, we know that if we send a message to a specific address it will be sent to the intended device only.

The Third Layer – The Network Layer

The third layer is responsible for routing – that is, determining the path where the data will “travel”.

You can think of this layer as the successful routing app, Google Maps. When you get in the car and use Google Maps, you tell the app your destination, and Google Maps finds out the best route for you to drive in.

Notice that Google Maps is dynamic – it won’t necessarily pick the same route each time. Sometimes, one path will have a traffic jam, so Google Maps will prefer another route.

We said that the second layer has physical addresses, called MAC addresses. The third layer is responsible for logical addresses, such as IP addresses.

In this layer, datagrams are called packets.

Consider the following network diagram:

_A network diagram with Computer A in France, Computer B in the US, and 10 routers in between (Source: Brief)_

We have two computers here – one in France, and one in the United States. Of course, they are not directly linked. Rather, they are linked via third layer devices called routers.

Which layer is responsible for each connection?

Consider the connection between Computer A and Router 1. The second layer is responsible for this connection. What about the connection between Router 2 and Router 5? Right, again, this is the second layer. The same applies for each connection between two directly linked devices.

The third layer is responsible for defining the route – that the message sent from Computer A to Computer B will go through Routers 1, 2, 5, 8 and 10, and not in another way.

Note that there may be different implementations for each layer. For instance, we may have different implementations of the second layer. So while the connection between computer A and Router 1 might be over an Ethernet cable, the connection between Router 1 and 2 might be wireless and use WiFi. The connection between Router 2 and Router 5 might use a carrier pigeon, while the connection between router 5 and 9 will also use WiFi.

_The second layer may be implemented differently on every link (Source: Brief)_

The third layer does not care about these changes, but the second layer definitely does. If the carrier pigeon that transmits data from Router 2 to Router 5 is sick, the second layer will have to handle it. The data link layer will also have to make sure the data transmitted over the air between routers 1 and 5 is valid and without errors.

Interim Summary

So far we have covered three of the five layers. To recap:

The physical layer is responsible for transmitting a single bit, 1 or 0, over the network.
The data link layer is responsible for transmitting data between directly linked devices, that is – devices connected via a single hop.
The third layer is responsible for transferring data between hosts that are connected via multiple hops. It determines the route, the path that the packets will travel.

The Fourth Layer – The Transportation Layer

The fourth layer is an end-to-end layer. That is, it is responsible for communication from the source, all the way to the ultimate destination.

It allows multiplexing of multiple services. For example, one server may serve as a Web server, as well as a Mail server. When a client turns to that server, the client should be able to specify which service it would like to access. While the third layer specifies the address of the server, the transport layer identifies which service is relevant for the current communication.

In addition, the transport layer may ensure reliability. So when this layer receives data from the upper layer, it splits it into chunks, sends them, and makes sure that all those chunks arrive correctly at the other end.

Notice that the network layer is usually not reliable. Packets may arrive in incorrect order, they can arrive with incorrect data, or even not arrive at all. A reliable transportation layer makes sure that the data is correctly received.

In this layer, datagrams are called segments.

Consider the following network diagram once more:

_The network diagram again (Source: Brief)_

Which layer is responsible for what?

We have already said that the network layer is responsible for the route, that is, the path in which the packets travel. We also mentioned that the second layer is responsible for the transmission of the data between two, directly connected devices. For example, the link between Router 1 and Router 2.

The fourth layer views all of this network diagram as an abstract cloud. It doesn’t know the routers, and it doesn’t care about the structure of the network, or the routing. It assumes that the network can send a packet from one end to another:

_The fourth layer sees the network as an abstract cloud (Source: Brief)_

The transportation layer makes sure that the endpoints can communicate over different services – for example, web and email. In addition, it might make sure that the connection is reliable.

One example would be to acknowledge every received segment. For instance, when computer A sends a segment to computer B, computer B will send a special Acknowledgement segment, announcing that it has received the packet.

The Fifth Layer – The Application Layer

Last but definitely not least, we have the fifth layer, or Application Layer. This layer provides the service to the user’s application – web service, Voice over IP (VoIP), network games, streaming, and so on.

According to the layers model, the fifth layer doesn’t care at all about the network. It relies on the fourth layer, as well as the lower layers, to transmit the data from one endpoint to another. The fifth layer will use this service for the various needs of the application.

Different protocols will be used for different applications. For instance, HTTP protocol is commonly used for serving web pages on the World Wide Web. SMTP is a protocol used for emails, FTP for exchanging files, and there are many, many more.

What is Encapsulation?

The goal of networks is to transmit data from one host to another.

To achieve this goal, each layer adds its own header to the data. A header contains information specific for that layer, and it precedes the data itself.

Consider a case where we have a lookup service, used in order to find a person’s phone number, given the person's name. The data consists of the person’s first and last name.

Before the packet is sent, the fifth layer might add its own header, describing that this is a REQUEST packet. The header might also specify that this is a request to map from a person’s name to a phone number, and not vice versa.

_Header of the 5th layer, with data (Source: Brief)_

Then, the fifth layer passes the data to the fourth layer. Note, that the fourth layer regards everything as data – ones and zeroes. It doesn’t care if the fifth layer added a header, or what is written inside that header.

The fourth layer then adds its own header. For instance, it might specify that the requested service is the names-and-phones service. It may also include a sequential number for the packet, so it can be identified later.

_Header of the 4th layer, with data which includes the 5th layer's header (Source: Brief)_

Afterwards, the fourth layer will pass the packet to the third layer. Again, the third layer will regard everything it has received – including the data itself, the header added by the fifth layer, and the header added by the fourth layer – simply as a chunk of data.

Then, the third layer will add its own header. For instance, it may include the source address and destination address of the packet.

_Header of the 4th layer, with data which includes the 4th layer's header and data (Source: Brief)_

This process goes on. So, each layer adds its own header to the packet. This process is called *encapsulation.

On the other end, the receiver gets the packet and needs to read and remove the headers.

The second layer may also include a trailer – an additional chunk of bits following the data, with some information.

Putting it All Together

Now that we have covered the five layers, let’s have one example using all of them together.

Let’s say we would like to send a video file to our friend who lives in France, while we are enjoying a trip in Argentina. For that, we are using an email service.

The fifth layer defines how the email will be transmitted. For example, it includes the email address of the sender, as well as the receiver. It contains a title, and the body of the message. It requires that we follow a specific template of an email address, that will be included in the header of this layer.

_The five layers model, with an example of sending an email (Source: Brief)_

Then, the fifth layer uses the fourth layer in order to split the email into chunks. Of course, each chunk will also carry the fourth layer's header. It is also used in order to specify that we are currently using an email service.

In this case, we definitely want the connection to be reliable – so the receiver will be able to play our video file correctly. Thus, the fourth layer will also handle reliability. On the receiver’s end, it might send an acknowledgment packet for every packet it receives.

The third layer will define the best route for every packet to be sent. It might choose different routes for different packets. Among other things, its header will contain the source and destination addresses for the packet.

The second layer will be responsible for every link between two directly connected devices. Its header will include the MAC addresses for each device.

The first layer is responsible for encoding all the ones and zeros, and to pass them over the line. And then, decoding and reading those ones and zeroes on the other end. On this layer, we don't really have a header, as it consists of single bits only.

This way, every layer uses the services provided by the lower layers, and the huge problem of transmitting data over the network becomes doable. How amazing is that?

Summary

In this post you learned what the five layers model is and why we need layers. You should now understand what each layer is responsible for, and you can fit every topic you encounter in Computer Networks into this model.

About the Author

Omer Rosenbaum is Swimm’s Chief Technology Officer. He's the author of the Brief YouTube Channel. He's also a cyber training expert and founder of Checkpoint Security Academy. He's the author of Computer Networks (in Hebrew). You can find him on Twitter.

Additional References

A Visual Guide to Git Internals — Objects, Branches, and How to Create a Repo From Scratch

Omer Rosenbaum — Mon, 14 Dec 2020 22:30:27 +0000

Many of us use git on a daily basis. But how many of us know what goes on under the hood?

For example, what happens when we use git commit? What is stored between commits? Is it just a diff between the current and previous commit? If so, how is the diff encoded? Or is an entire snapshot of the repo stored each time? What really happens when we use git init ?

Many people who use git don’t know the answers to the questions above. But does it really matter?

First, as professionals, we should strive to understand the tools we use, especially if we use them all the time — like git .

But even more acutely, I've found that understanding how git actually works is useful in many scenarios — whether it’s resolving merge conflicts, looking to conduct an interesting rebase, or even just when something goes slightly wrong.

You’ll benefit from this post if you’re experienced enough with git to feel comfortable with commands such as git pull ,git push ,git add or git commit.

Still, we will start with an overview to make sure we are on the same page regarding the mechanisms of git, and specifically, the terms used throughout this post.

I also uploaded a YouTube series covering this post — you are welcome to watch it here.

What to expect from this tutorial

We will get a rare understanding of what goes on under the hood of what we do almost daily.

We will start by covering objects — blobs, trees, and commits. We will then briefly discuss branches and how they are implemented. We will dive into the working directory, staging area and repository.

And we will make sure we understand how these terms relate to the git commands we know and use to create a new repository.

Next, will create a repository from scratch — without using git init, git add, or git commit. This will allow us to deepen our understanding of what is happening under the hood when we work with git.

We will also create new branches, switch branches, and create additional commits — all without using git branch or git checkout.

By the end of this post, you will feel like you understand **git**. Are you up for it? 😎

Git Objects — blob, tree and commit

It is very useful to think about git as maintaining a file system, and specifically — snapshots of that system in time.

A file system begins with a root directory (in UNIX-based systems, /), which usually contains other directories (for example, /usr or /bin). These directories contain other directories, and/or files (for example, /usr/1.txt).

In git, the contents of files are stored in objects called blobs, binary large objects.

The difference between blobs and files is that files also contain meta-data. For example, a file “remembers” when it was created, so if you move that file into another directory, its creation time remains the same.

Blobs, on the other hand, are just contents — binary streams of data. A blob doesn’t register its creation date, its name, or anything but its contents.

Every blob in git is identified by its SHA-1 hash. SHA-1 hashes consist of 20 bytes, usually represented by 40 characters in hexadecimal form. Throughout this post we will sometimes show just the first characters of that hash.

In git, the equivalent of a directory is a tree. A tree is basically a directory listing, referring to blobs as well as other trees.

Trees are identified by their SHA-1 hashes as well. Referring to these objects, either blobs or other trees, happens via the SHA-1 hash of the objects.

Note that the tree CAFE7 refers to the blob F92A0 as pic.png. In another tree, that same blob may have another name.

The diagram above is equivalent to a file system with a root directory that has one file at /test.js, and a directory named /docs with two files: /docs/pic.png and /docs/1.txt.

Now it’s time to take a snapshot of that file system — and store all the files that existed at that time, along with their contents.

In git, a snapshot is a commit. A commit object includes a pointer to the main tree (the root directory), as well as other meta-data such as the committer, a commit message and the commit time.

In most cases, a commit also has one or more parent commits — the previous snapshot(s). Of course, commit objects are also identified by their SHA-1 hashes. These are the hashes we are used to seeing when we use git log.

Every commit holds the entire snapshot, not just diffs from the previous commit(s).

How can that work? Doesn’t that mean that we have to store a lot of data every commit?

Let’s examine what happens if we change the contents of a file. Say that we edit 1.txt, and add an exclamation mark — that is, we changed the content from HELLO WORLD, to HELLO WORLD!.

Well, this change would mean that we have a new blob, with a new SHA-1 hash. This makes sense, as sha1("HELLO WORLD") is different from sha1("HELLO WORLD!").

Since we have a new hash, then the tree’s listing should also change. After all, our tree no longer points to blob 73D8A, but rather blob 62E7A instead. As we change the tree’s contents, we also change its hash.

The tree that points to the changed blob needs to change as well

And now, since the hash of that tree is different, we also need to change the parent tree — as the latter no longer points to tree CAFE7, but rather tree 24601. Consequently, the parent tree will also have a new hash.

Almost ready to create a new commit object, and it seems like we are going to store a lot of data — the entire file system, once more! But is that really necessary?

Actually, some objects, specifically blob objects, haven’t changed since the previous commit — blob F92A0 remained intact, and so did blob F00D1.

So this is the trick — as long as an object doesn’t change, we don’t store it again. In this case, we don’t need to store blob F92A0 and blob F00D1 once more. We only refer to them by their hash values. We can then create our commit object.

Blobs that remained intact are referenced by their hash values

Since this commit is not the first commit, it has a parent — commit A1337.

So to recap, we introduced three git objects:

blob — contents of a file.
tree — a directory listing (of blobs and trees).
commit — a snapshot of the working tree.

Let us consider the hashes of these objects for a bit. Let’s say I wrote the string git is awesome! and created a blob from it. You did the same on your system. Would we have the same hash?

The answer is — Yes. Since the blobs consist of the same data, they’ll have the same SHA-1 values.

What if I made a tree that references the blob of git is awesome!, and gave it a specific name and metadata, and you did exactly the same on your system. Would we have the same hash?

Again, yes. Since the trees objects are the same, they would have the same hash.

What if I created a commit of that tree with the commit message Hello, and you did the same on your system. Would we have the same hash?

In this case, the answer is — No. Even though our commit objects refer to the same tree, they have different commit details — time, committer etc.

Branches in Git

A branch is just a named reference to a commit.

We could always reference a commit by its SHA-1 hash, but humans usually prefer other forms to name objects. A branch is one way to reference a commit, but it’s really just that.

In most repositories, the main line of development is done in a branch called master. This is just a name, and it’s created when we use git init, making it is widely used. However, it’s by no means special, and we could use any other name we’d like.

Typically, the branch points to the latest commit in the line of development we are currently working on.

To create another branch, we usually use the git branch command. By doing that, we actually create another pointer. So if we create a branch called test, by using git branch test, we are actually creating another pointer that points to the same commit as the branch we are currently on.

Using git branch creates another pointer

How does git know what branch we’re currently on? It keeps a special pointer called HEAD. Usually, HEAD points to a branch, which in turns points to a commit. In some cases, HEAD can also point to a commit directly, but we won’t focus on that.

HEAD points to the branch we are currently on.

To switch the active branch to be test, we can use the command git checkout test. Now we can already guess what this command actually does — it just changes HEAD to point to test.

git checkout test changes where HEAD points

We could also use git checkout -b test before creating the test branch, which is the equivalent of running git branch test to create the branch, and then git checkout test to move HEAD to point to the new branch.

What happens if we make some changes and create a new commit using git commit? Which branch will the new commit be added to?

The answer is the test branch, as this is the active branch (since HEAD points to it). Afterwards, the test pointer will move to the newly added commit. Note that HEAD still points to test.

Every time we use git commit, the branch pointer moves to the newly created commit.

So if we go back to master by git checkout master, we move HEAD to point to master again.

Now, if we create another commit, it will be added to the master branch (and its parent would be commit B2424).

How to Record Changes in Git

Usually, when we work on our source code we work from a working dir. A working dir(ectrory) (or working tree) is any directory on our file system which has a repository associated with it. It contains the folders and files of our project, and also a directory called .git that we will talk more about later.

After we make some changes, we want to record them in our repository. A repository (in short: repo) is a collection of commits, each of which is an archive of what the project’s working tree looked like at a past date, whether on our machine or someone else’s.

A repository also includes things other than our code files, such as HEAD, branches, and so on.

Unlike other, similar tools you may have used, git does not commit changes from the working tree directly into the repository. Instead, changes are first registered in something called the index, or the staging area.

Both of these terms refer to the same thing, and they are used often in git’s documentation. We will use these terms interchangeably throughout this post.

When we checkout a branch, git populates the index with all the file contents that were last checked out into our working directory and what they looked like when they were originally checked out. When we use git commit, the commit is created based on the state of the index.

The use of the index allows us to carefully prepare each commit. For example, we may have two files with changes since our last commit in our working dir. We may only add one of them to the index (using git add), and then use git commit to record this change only.

Files in our working directory can be in one of two states: tracked or untracked.

Tracked files are files that git knows about. They either were in the last snapshot (commit), or they are staged now (that is, they are in the staging area).

Untracked files are everything else — any files in our working directory that were not in our last snapshot (commit) and are not in our staging area.

How to Create a Repo — The Conventional Way

Let’s make sure that we understand how the terms we’ve introduced relate to the process of creating a repository. This is just a quick high-level view, before we dive much deeper into this process.

Note — most posts with shell commands show UNIX commands. I will provide commands for both Windows and UNIX, with screenshots from Windows, for the sake of variance. When the commands are exactly the same, I will provide them only once.

We will initialize a new repository using git init repo_1, and then change our directory to that of the repository using cd repo_1. By using tree /f .git we can see that running git init resulted in quite a few sub-directories inside .git. (The flag /f includes files in tree’s output).

Let's create a file inside the repo_1 directory:

On a Linux system:

This file is within our working directory. Yet, since we haven’t added it to the staging area, it is currently untracked. Let's verify using git status:

The new file is untracked as we haven’t added it to the staging area, and it wasn’t included in a previous commit

We can now add this file to the staging area by using git add new_file.txt. We can verify that it has been staged by running git status:

Adding the new file to the staging area

We can now create a commit using git commit:

Has something changed within .git directory? Let’s run tree /f .git to check:

A lot of things have changed within .git

Apparently, quite a lot has changed. It's time to dive deeper into the structure of .git and understand what is going on under the hood when we run git init, git add or git commit.

Time to get hard core

So far we've covered some Git fundamentals, and now we’re ready to really Git going.

In order to deeply understand how git works, we will create a repository, but this time — we'll build it from scratch.

We won’t use git init, git add or git commit which will enable us to get a better hands-on understanding of the process.

How to Set Up `.git`

Let’s create a new directory, and run git status within it:

Alright, so git seems unhappy as we don’t have a .git folder. The natural thing to do would be to simply create that directory:

Apparently, creating a .git directory is just not enough. We need to add some content to that directory.

A git repository has two main components:

A collection of objects — blobs, trees, and commits.
A system of naming those objects — called references.

A repository may also contain other things, such as git hooks, but at the very least — it must include objects and references.

Let’s create a directory for the objects at .git\objects and a directory for the references (in short: refs) at .git\refs (on UNIX -based systems — .git/objects and .git/refs, respectively).

One type of reference is branches. Internally, git calls branches by the name heads. So we will create a directory for them — .git\refs\heads.

This still doesn’t change our git status:

How does git know where to start when looking for a commit in the repository? As I explained earlier, it looks for HEAD, which points to the current active branch (or commit, in some cases).

So, we need to create the HEAD, which is just a file residing at .git\HEAD. We can apply the following:

On Windows: > echo ref: refs/heads/master > .git\HEAD

On UNIX: $ echo "ref: refs/heads/master" > .git/HEAD

⭐ So we now know how HEAD is implemented — it’s simply a file, and its contents describe what it points to.

Following the command above, git status seems to change its mind:

HEAD is just a file

Notice that git believes we are on a branch called master, even though we haven’t created this branch. As mentioned before, master is just a name. We could also make git believe we are on a branch called banana if we wanted to:

🍌

We will switch back to master for the rest of this post, just to adhere to the normal convention.

Now that we have our .git directory ready, can we work our way to make a commit (again, without using git add or git commit).

Plumbing vs Porcelain Commands in Git

At this point, it would be helpful to make a distinction between two types of git commands: plumbing and porcelain. The application of the terms oddly comes from toilets (yeah, these — 🚽), traditionally made of porcelain, and the infrastructure of plumbing (pipes and drains).

We can say that the porcelain layer provides a user-friendly interface to the plumbing. Most people only deal with the porcelain. Yet, when things go (terribly) wrong, and someone wants to understand why, they would have to roll-up their sleeves to check the plumbing. (Note: these terms are not mine, they are used very widely in git).

git uses this terminology as an analogy to separate the low-level commands that users don’t usually need to use directly (“plumbing” commands) from the more user-friendly high level commands (“porcelain” commands).

So far, we have dealt with porcelain commands — git init, git add or git commit. Next, we transition to plumbing commands.

How to Create Objects in Git

Let's start with creating an object and writing it into the objects’ database of git, residing within .git\objects. We'll find the SHA-1 hash value of a blob by using our first plumbing command, git hash-object, in the following way:

On Windows:

> echo git is awesome | git hash-object --stdin

On UNIX:

$ echo "git is awesome" | git hash-object --stdin

By using --stdin we are instructing git hash-object to take its input from the standard input. This will provide us with the relevant hash value.

In order to actually write that blob into git’s object database, we can simply add the -w switch for git hash-object. Then, we can check the contents of the .git folder, and see that they have changed.

Writing a blob to the objects’ database

We can now see that the hash of our blob is — 54f6...36. We can also see that a directory has been created under .git\objects, a directory named 54, and within it, a file by the name of f6...36.

So git actually takes the first two characters of the SHA-1 hash and uses them as the name of a directory. The remaining characters are used as the filename for the file that actually contains the blob.

Why is that so? Consider a fairly big repository, one that has 300,000 objects (blobs, trees, and commits) in its database. To look up a hash inside that list of 300,000 hashes can take a while. Thus, git simply divides that problem by 256.

To look up the hash above, git would first look for the directory named 54 inside the directory .git\objects, which may have up to 256 directories (00 through FF). Then, it will search that directory, narrowing down the search as it goes.

Back to our process of generating a commit. We have now created an object. What is the type of that object? We can use another plumbing command, git cat-file -t (-t stands for “type”), to check that out:

Not surprisingly, this object is a blob. We can also use git cat-file -p (-p stands for “pretty-print”) to see its contents:

This process of creating a blob usually happens when we add something to the staging area — that is, when we use git add.

Remember that git creates a blob of the entire file that is staged. Even if a single character is modified or added (as we added ! in our example before), the file has a new blob with a new hash.

Will there be any change to git status?

Apparently, no. Adding a blob object to git’s internal database doesn’t change the status, as git doesn’t know of any tracked or untracked files at this stage.

We need to track this file — add it to the staging area. To do that, we can use the plumbing command git update-index, like so: git update-index --add --cacheinfo 100644 .

Note: (The cacheinfo is a 16-bit file mode as stored by git, following the layout of POSIX types and modes. This is not within the scope of this post).

Running the command above will result in a change to .git's contents:

Can you spot the change? A new file by the name of index was created. This is it — the famous index (or staging area), is basically a file that resides within .git\index.

So now that our blob has been added to the index, we expect git status to look different, like this:

That’s interesting! Two things happened here.

First, we can see that new_file.txt appears in green, in the Changes to be committed area. That is so because the index now has new_file.txt, waiting to be committed.

Second, we can see that new_file.txt appears in red — because git believes the file my_file.txt has been deleted, and the fact that the file has been deleted is not staged.

This happens as we added the blob with the contents git is awesome to the objects’ database, and told the index that the file my_file.txt has the contents of that blob, but we never actually created that file.

We can easily solve this by taking the contents of the blob, and writing them to our file system, to a file called my_file.txt:

As a result, it will no longer appear in red by git status:

So now it’s time to create a commit object from our staging area. As explained above, a commit object has a reference to a tree, so we need to create a tree.

We can do this with the command git write-tree, which records the contents of the index in a tree object. Of course, we can use git cat-file -t to see that it’s indeed a tree:

Creating a tree object of the index

And we can use git cat-file -p to see its contents:

Great, so we created a tree, and now we need to create a commit object that references this tree. To do that, we can use git commit-tree -m :

You should now feel comfortable with the commands used to check the created object’s type, and print its contents:

Creating a commit object

Note that this commit doesn’t have a parent, because it’s the first commit. When we add another commit we will have to declare its parent — we will do so later.

The last hash that we got — 80e...8f – is a commit’s hash. We are actually very used to using these hashes — we look at them all the time. Note that this commit owns a tree object, with its own hash, which we rarely specify explicitly.

Will something change in git status?

Nope 🤔.

Why is that? Well, to know that our file has been committed, git needs to know about the latest commit. How does git do that? It goes to the HEAD:

Looking at HEAD on Windows

Looking at HEAD on UNIX

HEAD points to master, but what is master? We haven’t really created it yet.

As we explained earlier in this post, a branch is simply a named reference to a commit. And in this case, we would like master to refer to the commit with the hash 80e8ed4fb0bfc3e7ba88ec417ecf2f6e6324998f.

We can achieve this by simply creating a file at \refs\heads\master, with the contents of this hash, like so:

⭐ In sum, a branch is just a file inside .git\refs\heads, containing a hash of the commit it refers to.

Now, finally, git status and git log seem to appreciate our efforts:

We have successfully created a commit without using porcelain commands! How cool is that? 🎉

How to Work with Branches in Git — Under the Hood

Just as we’ve created a repository and a commit without using git init, git add or git commit, now we will create and switch between branches without using porcelain commands (git branch or git checkout).

It’s perfectly understandable if you are excited, I am too 🙂

Let’s start:

So far we only have one branch, named master. To create another one with the name test (as the equivalent of git branch test), we would need to simply create a file named test within .git\refs\heads, and the contents of that file would be the same commit’s hash as the master points to.

If we use git log, we can see that this is indeed the case — both master and test point to this commit:

Let’s also switch to our newly created branch (the equivalent of git checkout test). For that, we should change HEAD to point to our new branch:

Switching to branch test by changing HEAD

As we can see, both git status and git log confirm that HEAD now points to test, which is, therefore, the active branch.

We can now use the commands we have already used to create another file and add it to the index:

Using the commands above, we have created a file named test.txt, with the content of Testing, created a corresponding blob, and added it to the index. We also created a tree representing the index.

It’s now time to create a commit referencing this tree. This time, we should also specify the parent of this commit — which would be the previous commit. We specify the parent using the -p switch of git commit-tree:

We have just created a commit, with a tree as well as a parent, as we can see:

Will git log show us the new commit?

As we can see, git log doesn’t show anything new. Why is that? 🤔 Remember that git log traces the branches to find relevant commits to show. It shows us now test and the commit it points to, and it also shows master which points to the same commit.

That’s right — we need to change test to point to our new commit. We can do that by simply changing the contents of .git\refs\heads\test:

It worked! 🎉🥂

git log goes to HEAD, which tells it to go to the branch test, which points to commit 465...5e, which links back to its parent commit 80e...8f.

Feel free to admire the beauty, we git you. 😊

Summary

This post introduced you to the internals of git. We started by covering the basic objects — blobs, trees, and commits.

We learned that a blob holds the contents of a file. A tree is a directory-listing, containing blobs and/or sub-trees. A commit is a snapshot of our working directory, with some meta-data such as the time or the commit message.

We then discussed branches and explained that they are nothing but a named reference to a commit.

We went on to describe the working directory, a directory that has a repository associated with it, the staging area (index) which holds the tree for the next commit, and the repository, which is a collection of commits.

We clarified how these terms relate to git commands we know by creating a new repository and committing a file using the well-known git init, git add, and git commit.

Then, we fearlessly deep-dived into git. We stopped using porcelain commands and switched to plumbing commands.

By using echo and low-level commands such as git hash-object, we were able to create a blob, add it to the index, create a tree of the index, and create a commit object pointing to that tree.

We were also able to create and switch between branches. Kudos to those of you who tried this on their own!👏

Hopefully, after following this post you feel you’ve deepened your understanding of what is happening under the hood when working with git.

Thanks for reading! If you enjoyed this article, you can read more on this topic on the swimm.io blog.

Omer Rosenbaum, Swimm’s Chief Technology Officer. Cyber training expert and Founder of Checkpoint Security Academy. Author of Computer Networks (in Hebrew).

Visit My YouTube Channel.

Additional References

A lot has been written and said about git. Specifically, I found these references to be useful:

Mutable vs Immutable Objects in Python – A Visual and Hands-On Guide

Omer Rosenbaum — Wed, 11 Nov 2020 19:01:31 +0000

Python is an awesome language. Because of its simplicity, many people choose it as their first programming language.

Experienced programmers use Python all the time as well, thanks to its wide community, abundance of packages, and clear syntax.

But there's one issue that seems to confuse beginners as well as some experienced developers: Python objects. Specifically, the difference between mutable and immutable objects.

In this post we will deepen our knowledge of Python objects, learn the difference between mutable and immutable objects, and see how we can use the interpreter to better understand how Python operates.

We will use important functions and keywords such as id and is, and we'll understand the difference between x == y and x is y.

Are you up for it? Let's get started.

In Python, everything is an object

Unlike other programming languages where the language supports objects, in Python really everything is an object – including integers, lists, and even functions.

We can use our interpreter to verify that:

>>> isinstance(1, object)
True

>>> isinstance(False, object)
True

def my_func():
   return "hello"

>>> isinstance(my_func, object)
True

Python has a built-in function, id, which returns the address of an object in memory. For example:

>>> x = 1
>>> id(x)
1470416816

Above, we created an object by the name of x, and assigned it the value of 1. We then used id(x) and discovered that this object is found at the address 1470416816 in memory.

This allows us to check interesting things about Python. Let's say we create two variables in Python – one by the name of x, and one by the name of y – and assign them the same value. For example, here:

>>> x = "I love Python!"
>>> y = "I love Python!"

We can use the equality operator (==) to verify that they indeed have the same value in Python's eyes:

>>> x == y
True

But are these the same object in memory? In theory, there can be two very different scenarios here.

According to scenario (1), we really have two different objects, one by the name of x, and another by the name of y, that just happen to have the same value.

Yet, it could also be the case that Python actually stores here only one object, which has two names that reference it – as shown in scenario (2):

We can use the id function introduced above to check this:

>>> x = "I love Python!"
>>> y = "I love Python!"
>>> x == y
True

>>> id(x)
52889984

>>> id(y)
52889384

So as we can see, Python's behavior matches scenario (1) described above. Even though x == y in this example (that is, x and y have the same values), they are different objects in memory. This is because id(x) != id(y), as we can verify explicitly:

>>> id(x) == id(y)
False

There is a shorter way to make the comparison above, and that is to use Python's is operator. Checking whether x is y is the same as checking id(x) == id(y), which means whether x and y are the same object in memory:

>>> x == y
True

>>> id(x) == id(y)
False

>>> x is y
False

This sheds light on the important difference between the equality operator == and the identity operator is.

As you can see in the example above, it is completely possible for two names in Python (x and y) to be bound to two different objects (and thus, x is y is False), where these two objects have the same value (so x == y is True).

How can we create another variable that points to the same object that x is pointing to? We can simply use the assignment operator =, like so:

>>> x = "I love Python!"
>>> z = x

To verify that they indeed point to the same object, we can use the is operator:

>>> x is z
True

Of course, this means they have the same address in memory, as we can verify explicitly by using id:

>>> id(x)
54221824

>>> id(z)
54221824

And, of course, they have the same value, so we expect x == z to return True as well:

>>> x == z
True

Mutable and immutable objects in Python

We have said that everything in Python is an object, yet there is an important distinction between objects. Some objects are mutable while some are immutable.

As I mentioned before, this fact causes confusion for many people who are new to Python, so we are going to make sure it's clear.

Immutable objects in Python

For some types in Python, once we have created instances of those types, they never change. They are immutable.

For example, int objects are immutable in Python. What will happen if we try to change the value of an int object?

>>> x = 24601
>>> x
24601

>>> x = 24602
>>> x
24602

Well, it seems that we changed x successfully. This is exactly where many people get confused. What exactly happened under the hood here? Let's use id to further investigate:

>>> x = 24601
>>> x
24601

>>> id(x)
1470416816

>>> x = 24602
>>> x
24602

>>> id(x)
1470416832

So we can see that by assigning x = 24602, we didn't change the value of the object that x had been bound to before. Rather, we created a new object, and bound the name x to it.

So after assigning 24601 to x by using x = 24601, we had the following state:

And after using x = 24602, we created a new object, and bound the name x to this new object. The other object with the value of 24601 is no longer reachable by x (or any other name in this case):

Whenever we assign a new value to a name (in the above example - x) that is bound to an int object, we actually change the binding of that name to another object.

The same applies for tuples, strings (str objects), and bools as well. In other words, int (and other number types such as float), tuple, bool, and str objects are immutable.

Let's test this hypothesis. What happens if we create a tuple object, and then give it a different value?

>>> my_tuple = (1, 2, 3)
>>> id(my_tuple)
54263304

>>> my_tuple = (3, 4, 5)
>>> id(my_tuple)
56898184

Just like an int object, we can see that our assignment actually changed the object that the name my_tuple is bound to.

What happens if we try to change one of the tuple's elements?

>>> my_tuple[0] = 'a new value'
Traceback (most recent call last):
  File "", line 1, in 
TypeError: 'tuple' object does not support item assignment

As we can see, Python doesn't allow us to modify my_tuple's contents, as it is immutable.

Mutable objects in Python

Some types in Python can be modified after creation, and they are called mutable. For example, we know that we can modify the contents of a list object:

>>> my_list = [1, 2, 3]
>>> my_list[0] = 'a new value'
>>> my_list
['a new value', 2, 3]

Does that mean we actually created a new object when assigning a new value to the first element of my_list? Again, we can use id to check:

>>> my_list = [1, 2, 3]
>>> id(my_list)
55834760

>>> my_list
[1, 2, 3]

>>> my_list[0] = 'a new value'
>>> id(my_list)
55834760

>>> my_list
['a new value', 2, 3]

So our first assignment my_list = [1, 2, 3] created an object in the address 55834760, with the values of 1, 2, and 3:

We then modified the first element of this list object using my_list[0] = 'a new value', that is - without creating a new list object:

Now, let us create two names – x and y, both bound to the same list object. We can verify that either by using is, or by explicitly checking their ids:

>>> x = y = [1, 2]
>>> x is y
True

>>> id(x)
18349096

>>> id(y)
18349096

>>> id(x) == id(y)
True

What happens now if we use x.append(3)? That is, if we add a new element (3) to the object by the name of x?

Will x by changed? Will y?

Well, as we already know, they are basically two names of the same object:

Since this object is changed, when we check its names we can see the new value:

>>> x.append(3)
>>> x
[1, 2, 3]

>>> y
[1, 2, 3]

Note that x and y have the same id as before – as they are still bound to the same list object:

>>> id(x)
18349096

>>> id(y)
18349096

In addition to lists, other Python types that are mutable include sets and dicts.

Implications for dictionary keys in Python

Dictionaries (dict objects) are commonly used in Python. As a quick reminder, we define them like so:

my_dict = {"name": "Omer", "number_of_pets": 1}

We can then access a specific element by its key name:

>>> my_dict["name"]
'Omer'

Dictionaries are mutable, so we can change their content after creation. At any given moment, a key in the dictionary can point to one element only:

>>> my_dict["name"] = "John"
>>> my_dict["name"]
'John'

It is interesting to note that a dictionary's keys must be immutable:

>>> my_dict = {[1,2]: "Hello"}
Traceback (most recent call last):
  File "", line 1, in 
TypeError: unhashable type: 'list'

Why is that so?

Let's consider the following hypothetical scenario (note: the snippet below can't really be run in Python):

>>> x = [1, 2]
>>> y = [1, 2, 3]
>>> my_dict = {x: 'a', y: 'b'}

So far, things don't seem that bad. We'd assume that if we access my_dict with the key of [1, 2], we will get the corresponding value of 'a', and if we access the key [1, 2, 3], we will get the value 'b'.

Now, what would happen if we attempted to use:

>>> x.append(3)

In this case, x would have the value of [1, 2, 3], and y would also have the value of [1, 2, 3]. What should we get when we ask for my_dict[[1, 2, 3]]? Will it be 'a' or 'b'? To avoid such cases, Python simply doesn't allow dictionary keys to be mutable.

Taking things a bit further

Let's try to apply our knowledge to a case that is a bit more interesting.

Below, we define a list (a mutable object) and a tuple (an immutable object). The list includes a tuple, and the tuple includes a list:

>>> my_list = [(1, 1), 2, 3]
>>> my_tuple = ([1, 1], 2, 3)
>>> type(my_list)
<class 'list'>

>>> type(my_list[0])
<class 'tuple'>

>>> type(my_tuple)
<class 'tuple'>

>>> type(my_tuple[0])
<class 'list'>

So far so good. Now, try to think for yourself – what will happen when we try to execute each of the following statements?

(1) >>> my_list[0][0] = 'Changed!'

(2) >>> my_tuple[0][0] = 'Changed!'

In statement (1), what we are trying to do is change my_list's first element, that is, a tuple. Since a tuple is immutable, this attempt is destined to fail:

>>> my_list[0][0] = 'Changed!'
Traceback (most recent call last):
  File "", line 1, in 
TypeError: 'tuple' object does not support item assignment

Note that what we were trying to do is not change the list, but rather – change the contents of its first element.

Let's consider statement (2). In this case, we are accessing my_tuple's first element, which happens to be a list, and modify it. Let's further investigate this case and look at the addresses of these elements:

>>> my_tuple = ([1, 1], 2, 3)
>>> id(my_tuple)
20551816

>>> type(my_tuple[0])
<class 'list'>

>>> id(my_tuple[0])
20446248

When we change my_tuple[0][0], we do not really change my_tuple at all! Indeed, after the change, my_tuple's first element will still be the object whose address in memory is 20446248. We do, however, change the value of that object:

>>> my_tuple[0][0] = 'Changed!'
>>> id(my_tuple)
20551816

>>> id(my_tuple[0])
20446248

>>> my_tuple
(['Changed!', 1], 2, 3)

Since we only modified the value of my_tuple[0], which is a mutable list object, this operation was indeed allowed by Python.

Recap

In this post we learned about Python objects. We said that in Python everything is an object, and got to use id and is to deepen our understanding of what's happening under the hood when using Python to create and modify objects.

We also learned the difference between mutable objects, that can be modified after creation, and immutable objects, which cannot.

We saw that when we ask Python to modify an immutable object that is bound to a certain name, we actually create a new object and bind that name to it.

We then learned why dictionary keys have to be immutable in Python.

Understanding how Python "sees" objects is a key to becoming a better Python programmer. I hope this post has helped you on your journey to mastering Python.

Omer Rosenbaum, Swimm’s Chief Technology Officer. Cyber training expert and Founder of Checkpoint Security Academy. Author of Computer Networks (in Hebrew). Visit My YouTube Channel.

Omer Rosenbaum - freeCodeCamp.org

Stop Staring at a Blank Deck: How I Use Claude Code + Marp to Think Through Presentations

We'll cover:

The Workflow

Brainstorm

React

Iterate

Export

Editable PPTX

Get Started in 5 Minutes

1. Install Marp CLI:

2. Install the skill (via skills.sh):

3. Run it:

4. Iterate:

Use Case: This Very Article

Under the Hood

Marp: Markdown to Slides

The Skill File

Section Dividers

Breadcrumb Navigation

Wrapping Up

About the Author

Product-Led Research: A Practical Guide for R&D Leaders [Full Book]

What You'll Learn

Who Is This Book For?

Who Am I?

The Approach of This Book

Structure of This Book

Why Is This Book Publicly Available?

Feedback Is Welcome

Table of Contents

Part 1: Foundations

Part 2: Research Management Methods

Part 3: Ensuring Product Impact

Book Summary

Note

Part 1: Foundations

Chapter 1 - What is Research?

The Difference

Schoenfeld's Framework for Problem Solving

Why This Matters for R&D Leaders

Notes

References

Chapter 2 - Research and Development

Defining Research vs. Development

A Quick Test

Key Indicators You're Doing Research

Common Misconceptions

Your Role as Research Leader

Part 1 - Summary

Part 2: Research Management Methods

Chapter 3 - Why Methodology Matters: A True Story

The Lesson

What This Part Covers

Chapter 4 - The Research Tree

What Is a Research Tree?

Your First Research Tree

The Power of Stopping to Think

Choosing Your First Path

What If Your First Choice Doesn't Work?

Choosing When Everything Seems Equal

Heuristics Can Be Combined

How Answers Lead to New Questions

The Complete Picture

Color-Coding Status

Additional Tips

The Research Tree Prevents Common Pitfalls

Time to Practice

Using the Tree with Your Team

Tools

Pro Tips

Recap - The Research Tree

Chapter 5 - Time-Boxing Research Explorations

The Problem: Research Without Time Limits

Time-Boxing: Creating Decision Points

The Decision Point

Example: Detecting God Objects

How to Set Time Boxes

Using Time-Boxing with Your Team

Integration with the Research Tree