research - freeCodeCamp.org

Stop Staring at a Blank Deck: How I Use Claude Code + Marp to Think Through Presentations

Omer Rosenbaum — Thu, 19 Mar 2026 17:36:41 +0000

The hard part of building a presentation is figuring out the story. What are you trying to say? What’s the structure? Which sections build on which? Where does the data go, table or bullets? Before the comparison or after?

What would help is having something to react to. Starting from zero is hard. Reacting to a draft is fast. “Move this before that” is way easier than “what should I say?”

That’s the workflow I want to show you. I use Claude Code + Marp to think through presentations. Claude helps me brainstorm the story, gives me a first draft to react to, and then I iterate, either through “conversation” or by editing the Markdown directly. The whole thing is a text file. 🎉

(I used a deck to think through this post. You can find it here.)

We'll cover:

The Workflow
- Brainstorm
- React
- Iterate
- Export
- Editable PPTX
Get Started in 5 Minutes
Use Case: This Very Article
Under the Hood
Wrapping Up

The Workflow

The process has four stages: Brainstorm, React, Iterate, Export. Let me walk through each one.

Brainstorm

You kick things off with a slash command (I will provide you with this command later):

/create-marp-deck API rate limiting

Claude starts by interviewing you, asks about the goal, audience, key points, any data you want to include. This partforces you to articulate the story before a single slide exists.

~/projects/my-app $ claude
> /create-marp-deck API rate limiting
─────────────────────────────────────────────

Claude: What's this presentation about? What should the audience
        walk away knowing?

User:   How our rate limiting works, which algorithm we chose and
        why, and what they need to know to configure it for their
        services.

Claude: Who's the audience?

User:   Platform engineering team — they know our stack but haven't
        seen the rate limiting internals.

Claude: What are the key points or sections you want to cover?

User:   1. Why we need rate limiting (the incident last month)
        2. Token bucket vs sliding window — our decision
        3. How to configure it per-service
        4. Monitoring dashboard walkthrough

Think of it as a lightweight brainstorm: you talk through what you’re trying to say, and Claude helps you structure it.

React

Once you’ve aligned on the structure, Claude generates the full Marp Markdown file and exports it. You get a solid first draft you can react to and reshape.

That title slide came from this Markdown:

# API Rate Limiting
## A Technical Deep Dive

**Team**: Platform Engineering
**Date**: February 2026

Is it perfect? Probably not. But now you have something concrete, with sections, structure, and a story, that you can push around. That’s so much faster than starting from a blank canvas.

When you go through the slides, you feel if the story is coherent and clear.

Iterate

While reviewing the draft, it'll inevitably spark ideas: “oh, I should add a comparison table here,” “this section is too dense, maybe split it into two,” “move this summary up to the top.”

One way to make such edits is to ask Claude Code to do that:

"Slide 6 is too dense. Split the algorithm comparison into
two slides, one for token bucket, one for sliding window."

> Slide 6 is too dense. Split the algorithm comparison into two
  slides — one for token bucket, one for sliding window.

Claude: I'll split slide 6 into two separate slides, one per algorithm.

  Edit presentations/api-rate-limiting.md
  ───────────────────────────────────────
  - # Algorithm Comparison
  - | Feature | Token Bucket | Sliding Window |
  + # Token Bucket
  + Tokens refill at a steady rate...
  + ---
  + # Sliding Window
  + Track exact timestamp of every request...

> Add a "Why we chose token bucket" callout to that first slide

Claude: Added a blockquote callout explaining the decision.

  Edit presentations/api-rate-limiting.md
  ───────────────────────────────────────
  + > We chose token bucket because it handles bursty traffic
  + > from our mobile clients without penalizing steady callers

You can also edit in VS Code with the Marp extension for live preview. Open the .md file, hit Ctrl+Shift+V, and you get the source on the left with rendered slides on the right. Claude Code edits the file, VS Code detects the change, and the preview updates automatically. (I keep both open side by side and it just works.)

Export

When you’re done, you get three files:

.md – the source (version-controlled, diffable)
.html – open in any browser, share via Slack
.pptx – open in PowerPoint, present anywhere

$ marp --no-stdin deck.md -o deck.html
[  INFO ] Converting 1 markdown...
[  INFO ] deck.md => deck.html

$ marp --no-stdin --pptx deck.md -o deck.pptx
[  INFO ] Converting 1 markdown...
[  INFO ] deck.md => deck.pptx

$ ls presentations/
api-rate-limiting.md
api-rate-limiting.html   ✓ open in browser, share via Slack
api-rate-limiting.pptx   ✓ open in PowerPoint, present anywhere

The skill runs the export commands automatically after generating the deck. A 15-slide deck converts in about 2 seconds.

Editable PPTX

The standard PPTX export renders each slide as an image – pixel-perfect, but you can’t edit the text in PowerPoint or Google Slides. If you need editable text, Marp has a --pptx-editable flag that uses LibreOffice under the hood to produce real text boxes.

The catch: LibreOffice creates text boxes that are too narrow, so text wraps and overlaps. The skill includes a python-pptx post-processing script that automatically widens the text boxes to fix this. Just ask for “editable PPTX” and the skill handles the rest – the LibreOffice conversion, the text box fix, everything.

Get Started in 5 Minutes

OK, are you ready? Here’s everything you need:

1. Install Marp CLI:

npm install -g @marp-team/marp-cli

2. Install the skill (via skills.sh):

npx skills add Omerr/claude-skills

This works with Claude Code, Cursor, GitHub Copilot, and other AI agents. You can also install manually ( see the repo for details).

3. Run it:

/create-marp-deck your topic here

4. Iterate:

React to the draft, refine through conversation or VS Code, and export.

That’s it. Four steps. Fork the repo and customize the conventions to match your style.

Use Case: This Very Article

Want to see this workflow in practice? You’re looking at it.

I wrote this article by first creating a slide deck using exactly the process I described above. I ran /create-marp-deck, answered the interview questions, got a first draft, and iterated until the story felt right. You can see the deck here.

Why start with slides? Because a deck forces you to be concise and to go through the story. If the story doesn’t flow across 15 slides, it won’t flow across 1,500 words either. The deck became my outline, and once I had a coherent structure there, writing the article was much easier.

So if you’re ever staring at a blank doc thinking “I should write a blog post about X,” try making a deck first. You might be surprised how much faster the writing goes when the story is already figured out. 😎

Under the Hood

If you’re curious about what makes this work, read on. If not, you’re all set. 🙌🏻

Marp: Markdown to Slides

Marp (Markdown Presentation Ecosystem) converts .md files into slides. Your deck starts with frontmatter:

---
marp: true
theme: default
paginate: true
size: 16:9
---

Four lines and you have widescreen, paginated slides. Slide breaks are just --- in the Markdown. Your presentation is a text file: version-controlled, diffable, and AI-editable.

The Skill File

You could just ask Claude Code to “make me a Marp presentation” every time. But you’d spend half the conversation explaining your preferred format, color palette, and slide structure.

Instead, I created a Claude Code skill (see it here), a reusable set of instructions that Claude follows whenever you invoke it. It has two parts:

An interview phase that gathers context before generating anything (the 5 questions from the brainstorm step)
A generation phase with the full Marp conventions: CSS palette, slide structure, breadcrumb pattern, formatting rules, and export commands

The full skill is about 200 lines. That sounds like a lot, but you write it once and then every deck you create follows the same polished conventions automatically.

Section Dividers

Each section of the deck gets its own gradient background. So when you’re presenting, the audience intuitively knows when you’ve moved to a new topic:

Applied via CSS classes in the skill:


# Part 1: The Problem

This is my favorite part of the whole setup.

Every content slide has a breadcrumb header at the top that shows where you are in the deck:

Algorithms > Implementation” with the current section highlighted in blue" style="display:block;margin:0 auto" width="600" height="400" loading="lazy">

See that header? “The Problem > Algorithms > Implementation”, with “Algorithms” highlighted in blue.

In Marp, this is done with a simple HTML comment:

The **bold** text renders in blue (via CSS header strong { color: #2563eb; }), while the rest stays gray. You set it once per section and it persists until you change it.

How often have you sat through a presentation wondering “wait, where are we?” 🤔

Wrapping Up

The hard part of presentations is telling a coherent story. Get yourself a first draft to react to, iterate until it flows, and export. That’s it.

If you want to try it: npm install -g @marp-team/marp-cli, run npx skills add Omerr/claude-skills, and then /create-marp-deck. You'll have a deck in minutes and a workflow you can reuse for every presentation after that.

About the Author

Omer Rosenbaum is the author of the Brief YouTube Channel. He’s also a cyber training expert and founder of Checkpoint Security Academy. He’s the author of Product-Led Research, Gitting Things Done (in English) and Computer Networks (in Hebrew). You can find him on Twitter.

Product-Led Research: A Practical Guide for R&D Leaders [Full Book]

Omer Rosenbaum — Wed, 18 Feb 2026 22:39:21 +0000

Your team needs to solve a problem, and there's no clear solution path. Multiple approaches might work, but you're not sure which. Success isn't guaranteed.

This is Research, not Development. And if you manage it like Development, things aren't going to go well for you.

Maybe you've seen this: A talented engineer spends three weeks trying different approaches, switching between them until the original problem is no longer relevant. Or a researcher declares after two days "it's impossible" and gives up, though later it turns out the problem was actually solvable. Or worse and most frustrating of all: Research succeeds technically but doesn't impact the product, and you realize you've been solving the wrong problem.

Here's the thing: managing Research is fundamentally different from managing Development.

Development has known solution paths, relatively predictable timelines, and measurable progress. You probably learned to manage Development, and know how to do it well. Managing Development is challenging, yet there are established best practices that are well known in the industry.

But Research? Research is inherently uncertain. The techniques that work for Development fail spectacularly for Research. Most organizations either approach Research by deploying techniques devised for managing Development, or entirely give up on managing Research and treat it as mystical work by brilliant people, where the best you can do is not interrupt.

It doesn't have to be this way.

What You'll Learn

By reading this book, you’ll gain practical tools for managing Research that create real product impact.

You’ll learn your two critical roles as a Research leader:

Ensure Research connects to product impact.
Ensure Research is done effectively.

You’ll understand why Research is different from Development. You’ll get concrete tools for managing Research execution, and learn proven heuristics for ensuring product impact.

Throughout the book, you'll see the mentioned methods applied to real challenges.

You’ll feel confident managing Research. You’ll understand when to use which tools. And you’ll know how to keep Research connected to product value while managing its inherent uncertainty.

Who Is This Book For?

Any engineering leader who manages Research, or has researchers on their team.

If you're a CTO, VP of Engineering, R&D Director, or Engineering Manager responsible for Research initiatives – this book is for you.

This book is for product-focused leaders who need their Research to ship value, not just produce interesting findings.

You’ll also notice that I use a casual style throughout the book. I believe that learning Research management should be insightful and practical. These are hard problems, and writing in an overly academic style wouldn't serve you well. This book is for you, written to help you succeed.

Who Am I?

This book is about you, and your journey managing Research. But let me tell you why I think I can contribute to that journey.

I am the CTO and co-founder of Swimm.io, a knowledge management platform for code. At Swimm, I've led multiple Research initiatives — from automatically keeping documentation in sync with code changes, to extracting business rules from legacy COBOL systems. We've faced genuine Research challenges where the path to success wasn't clear, where we didn't know if solutions were even possible.

I've managed Research in product environments where "interesting findings" aren't enough — where Research must ship and create measurable value. I've experienced the failure modes firsthand: Research that succeeded technically but didn't impact the product, teams that got stuck exploring endlessly, and the challenge of managing uncertainty while maintaining velocity.

I've also managed teams of researchers — brilliant people who needed guidance not on technical capability, but on connecting their work to product value and working systematically through uncertainty.

This book combines my experience leading product-focused Research with my background in teaching and making complex ideas practical and actionable.

The Approach of This Book

This is not an academic piece on Research methodology. When writing it, I had three principles in mind:

1. Practical: In this book, you will learn how to accomplish things in Research management. You will understand frameworks not just for the sake of understanding, but with a practical mindset. I sometimes refer to this as the "practicality principle" – which guides me in deciding whether to include certain topics, and to what extent.

2. Based on proven theory: While practical, the methods are grounded in Alan Schoenfeld's problem-solving research — a framework developed by studying how people actually solve uncertain problems. You'll see how Schoenfeld's components (knowledge, heuristics, control, beliefs) map directly to practical Research management tools.

3. Real examples: You will see these methods applied to actual Research challenges. Not toy problems, but real initiatives involving complex problems with genuine uncertainty. These examples show the methods in action, including when they're messy, when approaches fail, and how to adapt.

Structure of This Book

The book is organized in three parts:

Part 1: Foundations: Read this to understand what makes Research different and why it needs specialized management. This part is short — just enough to establish the framework that organizes everything else.

Part 2: Research Management Methods: These are tools that work for any Research.

Part 3: Ensuring Product Impact: Methods specifically for connecting Research to product value.

Why Is This Book Publicly Available?

In short, I'd like this book to get to as many people as possible.

If you would like to support this book, you are welcome to buy E-Book version, Paperback, Hardc over , or buy me a coffee. Thank you!

Feedback Is Welcome

I created this book to help you and people like you manage Research effectively and ensure it creates product impact.

From the beginning, I asked for feedback from experienced leaders and researchers to make sure the book achieves these goals. If you found something valuable, felt something was missing, or thought something needed improvement — I would love to hear from you.

Your feedback helps make this book better for everyone. Please reach out at: gitting.things@gmail.com.

Now, let's begin. In Chapter 1, you'll learn exactly what makes Research different from Development, and why managing them differently matters.

Note

This book is provided for free on freeCodeCamp as described above and according to Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International.

If you would like to support this book, you are welcome to buy E-Book version, Paperback, Hardc over , or buy me a coffee. Thank you!

Part 1: Foundations

Chapter 1 - What is Research?

To manage research effectively, you first need to understand what research is, and what it is not.

From now on, I will use Research (capital R) when referring to our specific concept of research in this book, to distinguish it from general uses of the word.

Consider this scenario: Your team needs to optimize a critical API endpoint. It's slow, users complain, and you know exactly what to do: profile the code, identify bottlenecks, apply standard optimization techniques. This is challenging work, but it's not Research.

Now consider this: Your team needs to automatically extract business rules from 40-year-old COBOL codebases, consisting of millions of lines of code where the original developers are long retired. You're not even sure if extracting these rules automatically is possible. Multiple approaches might work. Or none might. This is Research.

The Difference

The distinction isn't about difficulty or technical sophistication. It's about uncertainty of approach.

Throughout this book, I will adopt the following definition: Research is confronting problems where you don't know if solutions exist or which approaches will work.

Research confronts problems where:

You don't know if a solution exists.
Multiple approaches might work, but you don't know which.
The path to success is not immediately clear.
You may need to invent new techniques.

Development involves:

Applying known techniques to build specific features.
Following established approaches, even if complex.
Clear success criteria tied to working software.

I've found Alan Schoenfeld's model of problem solving [1] to be a useful framework for defining and analyzing research. Schoenfeld, studying how people solve mathematical problems, identified four** components that determine success when facing genuinely uncertain problems. His framework applies directly to software Research:

Schoenfeld's Framework for Problem Solving

1. Knowledge Base — What you know

Are you familiar with relevant tools, algorithms, and techniques?

For COBOL business rule extraction: Do you understand COBOL syntax? Static analysis? Program comprehension techniques?

Without the right knowledge, you will have to spend time acquiring it before making progress, and might miss options that could be obvious to someone with more background.

2. Heuristics — Strategies for approaching problems

We’ll cover heuristics in much more detail later. For now, here are some examples of effective heuristics:

"Work backwards from the desired output".
"Break the problem into smaller pieces".
"Try a simpler version first".
"List all assumptions and test each one".

For our COBOL business rule extraction case: "Start by manually extracting rules from one small program to understand what 'success' looks like".

3. Control — Monitoring and adjusting your approach

Recognizing when your current strategy isn't working. Deciding when to pivot to a different approach. Managing your time and resources effectively.

This is what separates experienced researchers from novices: it's not just what you know, and the heuristics that you may deploy, but when and how you use them. If you choose one approach, reflect on its effectiveness, and decide to try something different when needed, that's an example of control.

4. Beliefs and Attitudes — Your mindset toward the problem

Schoenfeld found that successful problem solvers held certain beliefs that helped them persist through challenges. Examples include:

"I can figure this out" vs. "I'm not good at this kind of thing".
"Problems have multiple solutions" vs. "There's one right answer".
"I should write things down and work systematically" vs. "I should solve this in my head".

These beliefs profoundly affect your ability to persist and succeed.

Why This Matters for R&D Leaders

You've likely seen this: A capable engineer spends three weeks on a Research task, trying approach after approach, until the original problem is no longer relevant. Or they declare after two days "it's impossible" and give up, though it turns out later that the problem was actually solvable.

The issue isn't (necessarily) capability. It's that Research requires different management, and different skills, than Development:

Knowledge base: Did they have the right background, or access to people who did?
Heuristics: Were they using effective problem-solving strategies, or just "trying things"?
Control: Did they know when to pivot? When to ask for help? When to break the problem down differently?
Beliefs: Did they believe the problem was solvable? That systematic approaches work better than random attempts?

The good news: All four components can be improved. People get better at Research through practice, exposure to effective heuristics, and environments that support good Control and healthy Beliefs.

The rest of this book provides concrete tools, like using a Research Tree, drawing backwards, and time-boxing methods, that put Schoenfeld's framework into action in a Product-led environment. These tools help you and your team apply better heuristics, maintain effective control, and build the beliefs that sustain successful Research.

But first, let's make sure we're clear on when you actually need these tools. The next chapter dives deeper into distinguishing Research from Development work.

Notes

** Actually, Schoenfeld (1992) described five components (which he terms "categories"), but I focused on four of them. For the curious reader – the one I skipped is called "Practices" – the habits and cultural norms of the mathematical environment that shape how a student approaches a problem. I chose to skip it as applying it to Research felt artificial.

References

[1] Schoenfeld, A. H. (1992). Learning to think mathematically: Problem solving, metacognition, and sense-making in mathematics. In D. Grouws (Ed.), Handbook for Research on Mathematics Teaching and Learning (pp. 334-370). New York: MacMillan.

Chapter 2 - Research and Development

Most R&D departments have plenty of Development work. Everyone agrees on what Development is. But Research? That's murkier.

Some claim every development task involves "research" – you have to test your code, try different things, read documentation. Is this Research?

Let's be precise.

Defining Research vs. Development

We established in chapter 1 the core distinction: Research involves fundamental uncertainty about whether solutions exist and which approaches will work, while Development applies known techniques to build specific features.

With this foundation, let's explore how to identify Research in practice.

A Quick Test

You're asked to reverse-engineer a specific compiled function: disassemble it and provide the equivalent code in C language. You know assembly, you know C, you have a disassembler. Is this Research?

No. You know how to proceed. It might take three days of careful work, especially if the function is complex, but it's not Research. You're applying known techniques, and know how to progress to a solution.

But if you need to understand how an entire program operates, and one possible approach is reverse engineering its compiled form, and you're not sure if that approach is even feasible time-wise or whether it will yield the insights you need? That's Research.

Key Indicators You're Doing Research

1. Fundamental Uncertainty About Solution Viability

You're asking "Can this even be done?" rather than "How should we do this?" This isn't about implementation details – rather, it's about whether the approach itself is viable.

2. Multiple Competing Approaches Without Clear Superiority

Research often means exploring several paths simultaneously, knowing that many will fail, to discover which approach (if any) can solve the problem.

3. Need for New Fundamental Techniques

You may need to invent new methods rather than adapting existing ones. Note: Not all Research creates new techniques, but the possibility exists.

Common Misconceptions

Misconception 1: Technical Complexity = Research

Many challenging Development tasks involve sophisticated algorithms, large-scale systems, or cutting-edge technologies without requiring Research approaches.

Building a distributed system with complex consensus algorithms? Challenging Development. Figuring out whether a distributed system can meet your latency requirements given your unusual constraints? Might be Research.

Technical complexity is not the same as Research.

Misconception 2: Using Advanced Algorithms = Research

Implementing machine learning pipelines with random forests or neural networks isn't Research – even though the underlying algorithms are sophisticated. The Research happened when those algorithms were first developed. However, if you are using those algorithms when trying to solve a problem where it's unclear if they will work at all, that could be Research.

Misconception 3: Research Can Be Managed Like Development

Perhaps the most damaging misconception. This leads to:

Demanding precise time estimates for uncertain work.
Expecting steady, measurable progress on fixed schedules.
Evaluating Research with Development metrics.

Research requires different approaches. This is exactly what the rest of this book addresses.

Misconception 4: Research Cannot Be Managed

The opposite extreme: treating Research as mystical work by brilliant people. The best a manager can do is not interrupt.

I've had the pleasure and privilege to work with many extremely skilled researchers. I can confidently say that this is simply not the case, as even the most talented researcher can benefit from skillful guidance.

Specifically, even the most talented researcher benefits from:

Clear connections between their work and product goals.
Structured approaches to exploring alternatives.
Regular checkpoints to assess direction.
Team collaboration and brainstorming.

Research is not magic, it can be managed effectively.

Your Role as Research Leader

When leading Product-led Research, your job has two parts:

1. Ensure the Research makes the biggest possible impact on the product

This is your most important responsibility. "Successful" Research that doesn't impact the product is a failed project. Your job is to maintain the connection between Research work and product value – not just at the start, but continuously.

This means:

Starting with clear product needs, not interesting technical questions.
Regularly validating that the Research still serves the product goal.
Making trade-offs between thorough exploration and shipping impact.

We'll cover this in detail in Part 3.

2. Ensure the Research is done in the most effective manner

Even brilliant researchers benefit from structured approaches. Your role is to help the team work systematically rather than randomly.

This means:

Helping identify which questions are worth answering.
Introducing better heuristics when the team is stuck ("Let's work backwards," "Let's time-box this investigation").
Preventing common failure modes like endless exploration or premature commitment.

(We'll cover this in detail in Part 2.)

Responsibility (1) is defining the right goals. Responsibility (2) is reaching these goals effectively.

The rest of this book provides concrete tools for both responsibilities.

Part 1 - Summary

Research is work where the path to success is not immediately obvious. It requires:

Knowledge of relevant domains.
Effective problem-solving strategies (heuristics) .
Monitoring and adjusting approaches (control).
Healthy beliefs and persistence.

Your role as a Research leader is to:

Ensure Research connects to product impact.
Ensure Research is done effectively.

You need different tools to manage Research versus Development – in order to make sure Research is done effectively. Part 2 provides those tools.

Part 3 will focus on connecting Research to product impact.

Part 2: Research Management Methods

Chapter 3 - Why Methodology Matters: A True Story

It was a late evening, and the classroom was filled with three dozen students. They were all sitting in front of their computers, working in silence. I was leading a cybersecurity training focused on reverse engineering. Each exercise included a single compiled program without its source code, with one goal: "understand how this program works." The output would be either a detailed document, an equivalent program implemented in a high-level language, or both.

This particular evening, the students were furiously working on reverse-engineering a game. The instruction was: "understand the game's rules, and document them thoroughly." The game had a user interface with two dimensions and could be played against the computer. Moving behind the students, I could see their screens with various reverse engineering tools open.

At some point we asked them to stop, turn around and look at the instructor. The instructor then provided a guided solution – this was a technique we used quite frequently, showing the students the "right" way to approach a problem they had spent some time tackling. The instructor opened the game, looked at the screen, opened the "File" menu, clicked on "Help" – and there it was, the entire description of the game rules.

The Lesson

The room erupted in nervous laughter. Some students looked embarrassed. Others seemed frustrated. But everyone understood the point.

These were capable people. They had the relevant knowledge base, in Schoenfeld's terms, as presented in chapter 1. That is, they had the relevant technical skills – they knew both assembly and C, they knew how to use disassemblers, debuggers, and all the sophisticated tools of reverse engineering. Yet they had missed something fundamental: checking if there was a simpler solution first.

This happens all the time in Research work.

A team spends weeks diving deep into a complex technical approach, when a simpler path existed that they never explored. Or they try one thing, then another, then another – never systematically evaluating which approaches make sense before starting.

The problem isn't capability. It's approach.

This is exactly why you need structured methods for Research management. Without them, even your most talented people will waste time, miss obvious solutions, and burn out trying random approaches.

As a reminder, in chapter 2, we discussed your role as a Research leader:

Focusing on responsibility (2), the illustration here shows how the same peak can be reached via difficult climbing (reverse engineering) or by using a hot air balloon (reading the Help menu). Part of your job as a Research leader is to help your team find the easiest path to the goal.

What This Part Covers

The rest of Part 2 provides concrete tools to prevent these problems. You'll learn:

The Research Tree – A visual framework for systematically exploring solution paths.
Time-boxing methods – How to limit exploration without killing creativity.

These aren't abstract concepts. They're battle-tested techniques that directly address the failure modes you've probably seen: teams spinning their wheels, giving up too early, or getting stuck on the wrong approach.

Let's get started.

Chapter 4 - The Research Tree

You've probably seen this: An engineer/researcher starts down one path, hits an obstacle, tries something else, hits another obstacle, then tries a third approach. Three weeks later, they're still stuck – or worse, they've built something that technically works but doesn't solve the actual problem.

The issue isn't persistence. It's that they never mapped out the solution space. They never visualized which approaches might work, which questions need answering, and how everything relates to each other.

The Research Tree solves this problem.

It’s a way to visualize and manage the Control component of Schoenfeld's framework from chapter 1 – monitoring and adjusting your approach. This is where most Research efforts struggle.

What Is a Research Tree?

A Research Tree is a living visual representation of your Research journey. It captures three things:

Possible solution paths – different approaches you might take.
Open questions – what you need to learn.
Closed questions – what you've already discovered.

Unlike a static plan, the Research Tree grows and changes as you learn (which is one reason I like the name "tree" 😇). You start with what you know, then update it continuously as you investigate. Dead ends get marked. New branches appear. Questions get answered and new questions emerge.

Think of it like this: You're exploring a cave system. You don't have a map – you're creating the map as you explore. You mark passages you've tried. You note which ones are dead ends. You write down questions: "Does this passage connect to the main chamber?" "Is there water down this route?" As you explore, you answer some questions and discover new ones you hadn't considered. You write down the answers you found, and the experiments you conducted to find them ("I tried going left, hit a wall after 50 feet").

Research works the same way. The Research Tree is both your map and your log.

Your First Research Tree

Let's build one together. Consider a common engineering challenge:

Goal: Reduce API response time from 800ms to under 100ms

(Note: despite not being a real Research task, I chose this example for its simplicity to illustrate the process of creating a Research Tree. It can also show you how Research Trees can be useful in a variety of scenarios.)

You start with a fundamental question that needs answering. What's the first thing you need to know?

Take a moment to think about this before reading on.

The first question is usually: Where is the bottleneck?

Until you answer this, you don't know which approaches make sense. Let's draw this:

You can use a simple pen and paper, a whiteboard, or a digital tool to draw this out. When creating your very first tree, I highly recommend doing it by hand – the physical act of drawing will help you feel comfortable with the process.

Now, how can you answer this question? What approaches might tell you where the bottleneck is?

You might identify:

Profile the application with a performance monitoring tool.
Add detailed logging to measure each operation.
Use database query analysis tools.

Let's add these as branches:

(Note: the Brown status means "uncertain, needs investigation" – more on this later.)

Each approach is an investigation you could run to answer the question.

The Power of Stopping to Think

Before we go further, notice what just happened. You stopped.

Instead of immediately jumping into "Let's add more logging!" or "I bet it's the database, let's check queries," you identified multiple possible approaches. You're looking at three different ways to answer the same question.

This is already valuable. Most engineers would have jumped straight into whichever approach came to mind first. Maybe you've done this yourself: spent two days adding detailed logging, only to discover later that a profiler would have given you the answer in 30 minutes.

By creating this tree, you've avoided that trap. You can see all the approaches before committing to any of them. It doesn't guarantee you will choose the "right" path - you can't always do that in advance – but it will minimize the chances of you omitting it completely.

Remember the reverse engineering students from chapter 3? They never created this tree. They jumped straight to the first approach they knew: disassemblers and debuggers. They didn't stop to think: "What are all the ways we could understand this game's rules?" If they had, they would have listed approaches like: reverse engineer the binary, check the Help menu, just play the game, examine config files, watch network traffic. And if they'd evaluated those approaches using the framework you're about to learn, "check the Help menu" would have scored perfectly: fastest feedback (30 seconds), lowest cost (zero), best coverage (complete rules). Instead, they spent hours on complex reverse engineering when a simple menu click would have worked. Don't be those students.

Now comes the critical question: Which branch do you try first?

Consider our tree again:

Choosing Your First Path

You're looking at three approaches: Profile, Logging, DB Analysis. Each is a heuristic (in Schoenfeld's terms, as presented in chapter 1) – a problem-solving strategy that might work. How do you decide which one to try?

Sometimes, the answer is obvious (I wouldn't really use a framework to check if there's a "Help" menu). Let me show you the framework I use when it's not so clear which path to choose. Ask yourself these questions:

1. Which gives the fastest feedback?

How long until you get an answer?

Profile: Can run in 10 minutes. Setup is probably 5 minutes.
Logging: Need to add logging code, deploy, wait for traffic – maybe 4 hours.
DB Analysis: Need to enable slow query log, wait for data – maybe 2 hours.

Fastest feedback wins. You want to learn quickly.

2. Which has the lowest cost?

What do you need to set up or change?

Profile: Just attach a profiler – no code changes.
Logging: Need to modify code, test, deploy.
DB Analysis: Need database permissions, might need config changes.

Lower cost wins. Why spend hours adding logging if you can get the answer without changing any code?

(In your environment, it may be different. Perhaps logging is really easy, while profiling is super hard to set up. I am not claiming that profiling is a better heuristic than logging – it depends on your circumstances.)

3. Which answers the most questions?

Some approaches answer not just your immediate question, but related questions, too.

Profile: Shows you CPU, memory, database, network – a complete picture.
Logging: Only shows what you logged.
DB Analysis: Only shows database queries.

Broader coverage wins. A profiler might show you that 70% of time is database queries and that 20% is network latency – information you wouldn't get from narrow approaches.

So this is an easy one. Profiling wins on all three criteria. That's your first approach to try.

Update your tree:

This doesn't mean the other approaches are bad, or that this one will necessarily turn out to be the best. It means that profiling is the best starting point given what you know right now.

What If Your First Choice Doesn't Work?

Let's say you try profiling and hit a problem: Your profiler can't attach to the production environment due to security restrictions.

This is valuable information. Mark Profile as Red and move to your next best option:

Now you try Logging. But notice: You didn't waste days trying Profile in production. You tried it, hit a blocker, immediately pivoted to Logging. The tree helped you move quickly.

Choosing When Everything Seems Equal

Sometimes multiple approaches look equally good. Profile and DB Analysis might both be fast, low-cost, and have decent coverage. How do you choose?

Pick one and move on. Don't spend an hour analyzing which approach to try for 30 minutes. The meta-work (deciding) shouldn't take longer than the actual work (trying it).

When approaches seem equal, ask: "Which one am I more familiar with?" or "Which one does the team have experience with?" Use your judgment, make a choice, and start learning.

The worst decision is no decision.

In general, this approach might feel like overkill. Should you really sketch out trees and compare branches before actually doing something?

The surprising answer is that while almost always it feels like overkill, almost every single time, it turns out to be worth it. Try it a few times and you will see for yourself.

Heuristics Can Be Combined

Sometimes you don't have to choose just one. You might run Profile and enable DB Analysis simultaneously. If they don't conflict and you have the time, parallel investigations can be powerful.

But be careful: Don't try to do everything at once. Start with your best option. If that doesn't fully answer your question, then add another approach.

How Answers Lead to New Questions

After marking "Profile" as red, you moved on to "Logging". You added a few indicative log messages and let the system run for a day. You discover: 70% of response time is database queries.

This answer eliminates the need for other approaches (you don't need DB Analysis now – you found the answer). But more importantly, it reveals new questions:

See how the tree grows? One answered question spawns two new questions. Each of these new questions will have its own approaches for answering them.

And you'll apply the same framework to choose which question to answer first: Which gives fastest feedback? Which has lowest cost? Which answers the most questions?

Let's expand "Which queries are slowest?":

Again, you'd evaluate: Which approach gives fastest feedback? Enabling slow query log is probably fastest, as you’d just flip a config flag and wait a few minutes.

You decide to enable the slow query log. After investigating, you discover: User profile queries are slowest: they make 15 separate database calls (N+1 problem).

(What is the N+1 problem in this context? It means that when fetching user profiles, the code first queries for a list of users (1 query), then for each user, it makes an additional query to fetch related data (N queries). If there are 15 users, that's 16 queries total. This is inefficient and slows down response time.)

This answer leads to a new question:

Now you have three solution approaches. Again, evaluate them:

Rewrite queries with JOINs: Fast to implement, proven pattern.
Add Eager Loading: Depends on your ORM, might be quick.
DataLoader Pattern: Requires learning new pattern, takes longer.

Rewrite queries with JOINs probably gives the fastest feedback if your team knows SQL well.

The Complete Picture

Let's see how the full tree looks after a few days of investigation:

(Note: for the "How many queries per request?" node – I skipped the approaches for brevity.)

Reading this tree:

We started with "Where is the bottleneck?" and chose profiling (fastest feedback).
We found out the profiler can't attach to the production environment due to security restrictions, so we pivoted to logging.
Logging revealed that 70% of response time is database queries.
That answer led to two new questions about specific queries.
For "Which queries are slowest?", we chose slow query log (fastest to enable).
Answering it led to another question about fixing the N+1 problem.
For "How can we fix N+1?", we evaluated the approaches and chose "Rewrite queries with JOINs" (team knows SQL, fastest to implement).
Meanwhile, "Can we reduce query count?" is still open and has its own approaches to investigate.

Color-Coding Status

When you're creating your Research Tree, you'll mark both questions and approaches with a particular status. Of course, the following specific colors are just suggestions – the important thing is to keep something consistently so you can quickly see the status at a glance.

For Questions:

Open: Not yet answered.
Closed: Answered (show the answer).
Blocking: Must answer before proceeding with an approach.

For Approaches:

Green: Viable, worth pursuing.
Brown: Uncertain, needs investigation.
Red: Dead end or not viable.

In our example above:

"Rewrite with Joins" is Green because we've identified that it addresses the specific N+1 problem and that the team is confident in implementing it.
"Redesign API" is Red because it would take too long for this project.
Other approaches are Brown because we haven't investigated them yet.

Additional Tips

Keeping the tree clean and simple is important, and obsessing over its looks and details really misses the point. That said, some readers will find benefits by adding a few more details to the tree, specifically:

Order: add a number next to a specific branch when tackling it, so it is easy to track which direction you tried first, which one followed and so on.
Pivot Explanations: if you chose to pivot from one branch to another, write why. This might help when you revise your decisions later, or when reviewing with your team (as described later in this chapter).

The Research Tree Prevents Common Pitfalls

The Research Tree with this decision-making framework addresses five critical failure modes:

1. Jumping on First Idea: Without a tree, people implement the first approach they think of. The tree forces you to identify alternatives and evaluate them systematically before starting.

2. Tunnel Vision: Even when considering alternatives in the beginning, people tend to lock onto one approach and not pivot from it even when it turns out to be the wrong choice. The tree makes alternatives visible and helps you not only choose the best starting point, but also reevaluate continuously.

3. Inefficient Learning: Teams might try expensive, slow approaches first when faster, cheaper ones exist. The decision framework helps you learn quickly.

4. Answering Questions You Don't Need To: Teams waste time investigating interesting but irrelevant questions. The tree shows how questions connect – you only need to answer questions that lead to your goal.

Time to Practice

Open your favorite drawing tool, or just grab a piece of paper. Think of a Research problem you're currently facing or recently faced.

Now answer these questions:

What's your goal? Write it at the top.
What's the first question you need to answer? Write it below the goal.
What are 2-4 approaches to answer that question? Draw them as branches.
For each approach, evaluate:
- How fast is the feedback?
- What's the cost?
- How much does it answer?
Which approach scores best? Mark it "TRY FIRST".

Actually do this. Don't just read and think "I understand." Drawing the tree and evaluating approaches forces you to be explicit, and you'll immediately see gaps in your thinking.

I'll wait.

...

Don't worry about me, I'm enjoying some really great coffee in the meanwhile.

...

Done? Good. You now have your first Research Tree with a clear starting point.

Using the Tree with Your Team

Research Trees become even more powerful when shared with a team. It actually provides you, the Lead, with a way to see what directions the team is executing upon, and why. Your job here is to make sure the framework is used. Help your team stop and ask: are we asking the right questions? Are there approaches that we missed? Are we choosing the right approach?

During Planning:

Draw the tree together as a group.
Brainstorm questions and approaches.
Evaluate approaches using the framework (fastest feedback, lowest cost, best coverage).
Everyone sees why you're trying a particular approach first.

During Execution:

Update the tree as you learn.
When stuck, revisit the tree to identify alternative approaches.
Make sure you consider whether you are asking all of the important questions, and whether you are considering all relevant approaches.
If you pivoted from a branch, explain your reason and ask if someone can challenge your logic.
Conduct regular tree reviews (weekly or bi-weekly).

Note that the Research Tree is also useful for one-on-one sessions: you can review the tree with individual team members to understand their progress and help them choose next steps. It actually makes the Control component of Schoenfeld's framework much easier to manage - as you see the variolus questions and approaches laid out visually.

Tools

Pen and paper (seriously, this works great).
Whiteboard (for team sessions).
Miro, Mural, or similar digital whiteboards.
Mind mapping software (XMind, MindNode, and so on).
Even a simple text file with indentation.

The tool doesn't matter. What matters is that the tree exists, is visible, and gets updated.

Pro Tips

Start with the most important question

Don't try to list all possible questions upfront. Start with the one question that, if answered, would most clarify your path forward. Answer it, then see what new questions emerge. More on finding the questions to start from in chapter 7.

Show how answers lead to new questions

When you close a question, immediately ask: "What new questions does this answer reveal?" Draw those as branches from the answer.

Update questions weekly

In your weekly check-ins, explicitly review: Which questions did we close? What did we learn? Which new questions emerged? Which questions are blocking progress?

Re-evaluate when context changes

If you learn something new that changes the evaluation (maybe a tool you thought was fast turns out to be slow), re-evaluate your approaches. The tree is living – update it.

Recap - The Research Tree

The Research Tree is a living visual framework that:

Shows questions you need to answer to reach your goal.
Maps approaches for answering each question.
Helps you choose the best starting point for each question.
Prevents jumping on the first idea without considering alternatives.
Captures how answers lead to new questions.
Tracks status of questions (open/closed/blocking) and approaches (green/brown/red).
Documents the investigation path so the team understands why decisions were made.
Evolves as you learn – questions get answered, new questions emerge.

Key structure:

Goal → Question → Approaches to answer it → Answer → New questions

Decision framework for choosing approaches:

Which gives fastest feedback?
Which has lowest cost?
Which answers the most questions?

In the next chapter, you'll learn how to manage execution using time-boxing and decision points to keep your Research moving forward without getting stuck.

Chapter 5 - Time-Boxing Research Explorations

In the previous chapter, you learned about the Research Tree, a powerful tool for visualizing and managing Research efforts. The tree helps you systematically explore different solution paths and keep track of open questions. It also helps you decide which path to try first.

But once you've chosen a branch, how long should you pursue it before stepping back to reconsider?

The Problem: Research Without Time Limits

A researcher investigates whether a machine learning model can predict code complexity. Day one goes well. Day two, they need more features. Day three, a different architecture might work better. They switch. Day four, the new architecture needs different preprocessing.

Two weeks later, they're still on this path. When you ask about trying alternatives, they say "I'm close. Just need a few more days."

Three weeks in, they admit this approach isn't viable. Meanwhile, a simpler approach sat unexplored on the Research Tree.

Without a defined checkpoint, there's no natural moment to ask: "Given what I've learned, is this still the best path?" The sunk cost fallacy (continuing an approach because you've already invested time, rather than because it's still the best option) takes over. This is extremely common by Researchers, who tend to be dedicated, brilliant people who get fixated on problems they try to solve.

Time-Boxing: Creating Decision Points

Let's face it: it's hard, even impossible, to estimate how long a Research task will take. But you should still provide time limits based on how long you're willing to invest before reconsidering.

Time-boxing provides mandatory decision points.

Note that we are not talking about deadlines, but decision points. These are moments where you stop and evaluate: What did I learn? Is this still the most promising path?

As a rule, for every task longer than a day, define a time limit. For example: "I'll spend three days investigating whether we can cluster files into logical folders based on their I/O operations."

Three things can happen:

Early success: The researcher figures it out in a few hours. Done early, move to the next question.
Early blocker: After one hour, they discover they don't have access to filenames. They can immediately reconsider: "Without filenames, is this viable?"
Time box expires: Three days pass with partial progress. Now comes the mandatory decision point.

The Decision Point

When your time box expires, stop. Pull out your Research Tree (yes, you already love that Tree). Ask three questions:

1. What was your goal in pursuing this direction?

Which question were you trying to answer? Why did you choose this approach over alternatives?

2. What did you learn?

Document your discoveries, even if incomplete:

"The approach works but needs more sophisticated algorithms than we thought."
"We need data we don't currently have."
"This is harder than expected – would take at least 2 more weeks."
"We're 70% there – just need to handle edge cases."

3. Given what you learned, is this still the most promising path?

Look at your Research Tree. You have other branches. Given what you now know, is continuing the best use of time?

You have three options:

You can continue with a new time box: "I've solved the core challenge. One more day for edge cases." Define the new time box. The key: you consciously decided to continue based on what you learned, not inertia.

You can pivot: "This would take two more weeks, and I'm not confident it'll work. There's a simpler approach on my tree." Mark this branch Red. Move to a different branch.

Or you can reconsider the question: "Files in this codebase don't have clear I/O patterns. Maybe I should try clustering by function dependencies instead." Go back to your tree and identify a different question.

Example: Detecting God Objects

You're detecting God Objects (classes that do too much) using static analysis. Your Research Tree shows three approaches: complexity metrics, method naming patterns, or machine learning on AST features.

You choose complexity metrics (fastest feedback, lowest cost) and set a 2-day time box.

Day 1-2: You implement cyclomatic complexity and SLOC metrics. Results: 65% accuracy, but 40% false positives.

Decision point: You stop. Looking at your tree, method naming patterns might provide complementary information. You decide to spend 1 day exploring whether combining both approaches improves accuracy.

Day 3: Combined approach: 78% accuracy, 25% false positives. Promising.

New decision: Time-box 3 more days to refine and test on larger codebases.

Notice what happened: The 2-day time box forced assessment before perfecting the first approach. You learned combining approaches was better. Without time-boxing, you might have spent a week perfecting complexity metrics alone.

How to Set Time Boxes

When choosing a branch on your Research Tree, ask: Is this shorter than a day? A few days? Longer than a week?

Shorter than a day: Just do it. Don't overthink time-boxing for tasks this short.

A few days (2-7 days): Time-box for 2-5 days. I usually time-box for slightly less than my estimate – if I think 4 days, I set 3 days. This forces reflection based on learning, not arbitrary completion.

Longer than a week: Time-box for one week maximum. Even if you're making progress, a week is long enough that stepping back is valuable.

Don't overthink the exact duration. The goal is creating natural stopping points. The specific number matters less than having some checkpoint.

Critical: Time-boxing is not about pressure. You're not failing if the time box expires without solving the problem. That's expected in Research. What time-boxing does is force moments of reflection that prevent weeks spent on directions you'd have abandoned if you'd stopped to reconsider.

Using Time-Boxing with Your Team

When managing researchers, set time boxes together during planning. Make sure they understand it's a decision point, not a deadline. Schedule the review explicitly.

At the review, look at the Research Tree together. Ask the three questions. Make the decision collaboratively. This prevents researchers from getting stuck without asking for help, and makes pivoting feel like a positive decision rather than failure.

Integration with the Research Tree

Time-boxing combines naturally with the Research Tree (chapter 4): each branch gets a time box, and when it expires, the tree shows your alternatives.

Time-boxing creates moments to ask "given what I've learned, what should I do next?" The Research Tree helps you answer that question.

Recap

For tasks longer than a day, set a time limit (2-5 days for medium tasks, max 1 week). When time expires, stop and evaluate using your Research Tree: What was my goal? What did I learn? Is this still the best path?

Time boxes acknowledge that research estimation is hard. They prevent sunk cost fallacy and endless exploration. They're not about pressure or finishing "on time" – they're about forcing reflection instead of momentum-driven continuation.

Combined with the Research Tree, time-boxing gives you control over research execution while respecting its inherent uncertainty.

Part 2 Summary

Effective Research requires structured approaches, not just technical skill:

The Research Tree visualizes solution paths, open questions, and closed questions.
Time-boxing prevents endless exploration while preserving flexibility.
Systematic evaluation helps you choose the best path forward.

In chapter 2, I argued that your role as a Research leader is to:

Ensure Research connects to product impact.
Ensure Research is done effectively.

This part handled the latter part of your role: how to ensure Research is done effectively.

Within this part, your role is to:

Ensure the team uses these methods consistently.
Help identify when to pivot, when to persist, and which questions matter.

These tools prevent common failure modes: jumping on the first idea, tunnel vision, inefficient learning, and wasting time answering irrelevant questions.

Part 3 shows how to connect Research to product impact.

Part 3: Ensuring Product Impact

In chapter 2, I argued that your role as a Research leader is to:

Ensure Research connects to product impact.
Ensure Research is done effectively.

Part 2 addressed (2) above – ensuring Research is done effectively – with tools that work in ANY research context. This part provides the complete answer to (1) – ensuring Research connects to product impact:

First, choose research that matters (chapter 6).
Then, work backwards from product value (chapter 7).
Continuously validate with end-to-end iterations (chapter 8).

Chapter 6 - How to Choose Research Initiatives

The very first step in making sure your Research impacts the product is choosing the right thing to research. And, just as important, avoiding Research that won't impact the product.

Research initiatives can start from two different places:

From a problem the product is facing.
From a technological opportunity.

1. Starting From a Concrete Problem

The most promising way to find research initiatives that will have a big impact on the product is to start from an acute problem that the product is facing.

At Swimm, we allowed users to write documents about their code, but inevitably, the code changed, and the documentation became outdated. This made writing documentation not worth the effort in the first place. We needed to find a way to make sure the documentation stayed up to date automatically, with a good user experience. This was a clear problem we faced, and we didn't know if it was even possible to solve.

Consider a different example: a medical company that wants to diagnose a disease based on a few blood samples. Currently, they have an algorithm in place, but it's not very accurate. Specifically, it yields too many false positives. They need to find a way to improve their prediction accuracy. This is a clear problem the product is facing, and it doesn't have a clear technological solution.

In both cases, the problem is clear, and its impact on the product or company is clear. At the same time, the solution is not clear, and it's not certain that a solution will be technologically feasible.

2. Starting From a Technological Opportunity

When generative AI became popular, many companies started exploring how to leverage it to improve their products. This is an example of an emerging technology that can enable new product features.

The same can happen with smaller, more specific technologies, and not necessarily new technologies – sometimes technologies that the relevant teams just familiarized themselves with. For example, if a researcher reads a paper about a new way to parse source code, that researcher might have an idea for a new product feature that can leverage this technology.

While many good ideas come from technological opportunities, it's important to remember that the real impact of Research is determined by the product, not the technology. It's far more risky to pursue a technological opportunity than a concrete problem. If you do start a Research based on a technological opportunity, your responsibility is to make sure that the technological opportunity, if pursued successfully, will indeed have a big impact on the product.

Problem-Driven vs. Opportunity-Driven: A Comparison

Aspect	Problem-Driven Research	Opportunity-Driven Research
Starting Point	Customer pain point or product limitation	New technology or technique becomes available
Product Impact Clarity	High – you know exactly what problem you're solving	Low to Medium – you're searching for problems this technology can solve
Risk Level	Lower – you know there's demand if you succeed	Higher – solution might not match any important problem
Validation	Problem already validated through user feedback	Needs validation that the solution matters to users
Examples	Swimm's auto-updating docs; Reducing false positives in medical diagnosis	"How can we use LLMs in our product?"; "This new parsing technique could enable..."
Success Criteria	Did we solve the problem?	Did we find a valuable use case AND solve the problem?

Problem-driven research starts with validated demand: you know that the goal is worth pursuing. Opportunity-driven research starts with a hammer looking for nails. Sometimes you find valuable nails, but it's riskier.

Should You Pursue a Research Initiative?

Say you have a research initiative – an idea you'd like to research. To know whether you should pursue it, you should be able to answer a simple set of questions:

1. Product Impact – If the Research succeeds, how big will the impact be?

This is the most crucial question.

For problem-driven research, this is usually straightforward: "If we solve automatic doc updates, we'll retain 40% more customers who currently churn due to outdated docs."

For opportunity-driven research, you need to work harder: "If we use LLMs for code analysis, we could enable X feature, which would help Y users save Z hours per week, translating to $W in additional revenue."

If you're not convinced a successful Research result would make a huge impact on the product, it's probably not worth pursuing at the moment (until you are convinced). "Huge impact" means:

Solving a top-3 customer pain point, OR
Enabling a new product capability that significantly expands your market, OR
Reducing a major cost or risk factor

Anything less, and your resources are better spent on Development work with clearer ROI.

2. Time to Impact – How long until we see product value?

Time estimation is always hard in software. This is true for Development tasks, and even more so for Research. You learned ways to manage this uncertainty during Research in chapter 5, but even at this early pre-Research stage, you should consider the timeline.

Ask yourself two related questions:

How long for the Research itself? (days? weeks? months?)
How long from successful Research to product impact? (immediate integration? requires significant Development? needs market validation?)

The total timeline matters because:

Research that takes 6 months but delivers immediate product value might be worthwhile.
Research that takes 2 months but requires 8 more months of Development might not be worth it if your product roadmap can't accommodate that.
Research that takes 1 year with uncertain outcomes probably isn't worth pursuing unless the potential impact is transformational.

Of course, the actual timespans vary greatly by context (and specifically by the company you work for).

Consider also whether you can achieve incremental value. Can you get some product impact in 3 months even if the full solution takes 9 months? This significantly de-risks longer Research initiatives.

3. Resources – Do you have what you need?

Research requires specific resources beyond just "engineering time":

Knowledge: Do you have team members familiar with the relevant:

Technical domain (for example, NLP, compiler design, distributed systems)?
Business domain (for example, medical diagnostics, financial regulations)?
Similar problems solved elsewhere?

If not, can you acquire this knowledge in reasonable time? (Reading papers for a week: reasonable. Earning a PhD: not reasonable.)

Capacity: Can you dedicate someone (or multiple people) for the expected duration? Research requires sustained focus – splitting someone 10% on Research and 90% on urgent product work rarely succeeds.

Dependencies: Do you need:

Access to specific data or systems?
Collaboration from other teams?
External expertise or consulting?
Budget for tools, cloud resources, or datasets?

If critical resources are unavailable or expensive to obtain, the initiative may not be viable.

Pre-Research Checks

You might not have clear answers to all three questions above. In that case, it makes sense to spend time answering them before actually pursuing the Research. This phase should be limited in time – ideally a few days, at most a week or two.

In this stage you might:

Interview customers and business stakeholders to understand the real impact of solving the problem.
Read about the technological aspects and previous research done in this field to assess feasibility and timeline.
Consult with people who have faced similar challenges (internal experts, academic contacts, or practitioners in your network).
Run quick feasibility tests – not full Research, but simple checks like "Can we even access the data we'd need?" or "Does this library exist in our language?"

After pre-research checks, you should have a clear answer: "Yes, we should pursue this because the impact is X, the timeline is roughly Y, and we have (or can get) the resources we need."

If you're not confident about substantial product impact, don't start the Research. This might sound harsh, but it's crucial: unfocused Research that doesn't connect to product needs wastes your most valuable resource – talented engineers' time and attention. One way to enhance your certainty about product impact is described in chapter 7.

The next chapters assume you understand the product impact of successful Research outcomes, and will help you ensure you actually achieve this impact as quickly as possible.

How to Choose Research Initiatives - Summary

Research initiatives start from either:

Problem-driven: A clear product need (strongly preferred)
Opportunity-driven: A new technology (higher risk)

Before pursuing Research, answer three questions:

Product Impact: Will success create huge value?
Time to Impact: How long until we see product results?
Resources: Do we have the knowledge, capacity, and dependencies?

Run pre-research checks (days, not weeks) to answer these questions if unclear.

Only pursue Research when you're confident about substantial product impact.

The next chapter shows how to maintain that product connection throughout the Research process.

Chapter 7 - Drawing Backwards

So you've chosen a research initiative, and done so correctly (following chapter 6). Now, how do you start working on it?

Most teams start by diving into technical challenges: parsing COBOL, building callgraphs, implementing algorithms. But there's a more powerful approach that ensures your Research actually impacts the product: start from the end and work backwards.

This heuristic – working backwards from your goal – is one of the most valuable problem-solving strategies you can use. Let me show you why with a simple game.

The Spiral Game

Consider this game:

The rules are simple:

The pawn starts on spot 41.
On each turn, a player moves the pawn backward between 1 to 6 spots. That is, if the pawn is on spot 41, you can move it to any spot from 35 to 40.
The player who moves the pawn to spot 1 wins.

Take a moment: If you go first, how would you play to guarantee a win?

(I do encourage you to take a moment and try this for yourself first.)

Most people start thinking from the current position (spot 41) and try to calculate forward: "If I move 3 spaces, they can move 2, then I can move 4..." This quickly becomes overwhelming – too many possible moves to track.

But if you work backwards, the solution becomes clear:

Starting from the end (spot 1):

To win, you need the pawn on spot 1 on your turn.
Your opponent just moved, so the pawn could be on spots 2-7 (since they moved 1-6 backward from wherever it was).
For any spot from 2-7, you can move directly to spot 1.
Conclusion: If the pawn is on spots 2-7 at the start of your turn, you win.

Continue working backwards (from spots 2-7):

You want your opponent to land on spots 2-7.
From spot 8, no matter what they do (move 1-6), they land on spots 2-7.
Conclusion: If you get the pawn to spot 8, you guarantee a win.

Continue working backwards from spot 8:

Notice that you've just created a "new" game: you no longer need to land on spot 1 in order to win. It's enough that you land on spot 8 – as from there, you can win no matter what your opponent does.

So, how do you ensure you can land on spot 8?

Spots 9-14 all allow moving to spot 8.
So if your opponent starts their turn with pawn on a spot between 9-14, you can force it to 8.
Which means you do not want to land on spots 9-14, but you do want to land on 15 (which will force your opponent, in turn, to land somewhere between 9-14).

Again, you've just created a "new" game: where your goal is to land on spot 15. From there, you already know how to win.

You can keep going like this, drawing backwards – would you want to land on spot 16 or not? How about 17? Until, at some point...

The pattern emerges:

Safe spots for you: 1, 8, 15, 22, 29, 36
From any of these, your opponent cannot avoid giving you another safe spot
These are all numbers of the form: 1 + 7n

The winning strategy:

From spot 41, move 5 spots backward to reach spot 36 (a "safe spot").
No matter what your opponent does, you can always move to the next "safe spot".
Eventually, you reach spot 1 and win.

Notice what happened: By working backwards from the goal, you discovered the systematic solution. Working forward from the start position would have been much, much harder.

You actually deployed a second, powerful heuristic here: considering specific cases (1, 8, 15, 22, 29, 36) – and generalizing (1+7n). This heuristic is also very common in research, though not specifically when aiming to ensure product impact.

How to Apply Drawing Backwards to Product-led Research

Let's connect this to Product-led Research. When you face a complex Research challenge, the question isn't "What technical problem should I solve first?" but rather:

"If the Research succeeds, what would the result look like?"

This forces you to:

Connect to product impact: You must envision the end state that creates value.
Work systematically: Like the spiral game, you identify the chain of dependencies backward.
Validate assumptions: Before solving sub-problems, ensure they lead to your goal.

Let me show you how this worked in practice at Swimm.

Case Study: Extracting Business Rules from COBOL

At Swimm, we wanted to automatically generate documents from COBOL codebases that included all the extracted business rules.

Quick context on business rules: Business rules are the constraints, conditions, and actions embedded within software that reflect organizational policies. For example, in money transfer logic:

A customer cannot transfer more than their available balance (overdraft limits notwithstanding).
High-value transfers require additional verification.
Cross-currency transfers must apply current exchange rates.

Some sources define business rules with three elements: Event, Condition, Action:

ON 
IF 
THEN 
ELSE

Our goal was to extract all business rules from a COBOL codebase. While challenging in any codebase, it's particularly acute in legacy COBOL code. (If you're interested in the technical details, see this post.) For this book, it's sufficient to know that many research attempts over the last few decades have tried different approaches to face this challenge.

Where Do You Even Start?

Faced with this problem, you might think:

"Should I build a callgraph of all functions? That means I need to parse COBOL code..."
"Should I create a COBOL parser first?"
"Maybe I should read academic papers about program comprehension?" (Good practice during the pre-Research checks phase from chapter 6)
"Perhaps I should track COBOL variables through the codebase?"
"Should I distinguish business conditions ('if requested transfer amount > available balance') from technical conditions ('if variable not initialized, show error')?"

Each of these might require its own research effort. Where do you start?

Drawing Backwards: Start with the End Result

Drawing backwards made us ask:

"If the Research succeeds, what would the result look like?"

When we first asked ourselves this question, we weren't sure. We knew from our clients that they wanted extracted business rules, but we couldn't know what the "ideal" output would look like.

So we decided, before tackling any technical challenges, to manually create documents showing extracted business rules from sample programs. We did this completely manually: no parsing, no algorithms, just understanding COBOL code ourselves and writing documentation.

We did this for various types of applications from different codebases, and learned:

There's no single "right" way to construct such a document.
The output structure differs from one program to another.
Certain patterns appear consistently across business logic.

By creating these documents manually, we formed a hypothesis: "This is what the output should look like, which means this output would make the biggest impact on the product. This is our north star."

But was it actually the north star?

Validating the Hypothesis

Once we manually wrote the documents, it was time to verify our hypothesis. With concrete output in hand, we could:

Discuss internally within the team – get feedback from engineers who understand both COBOL and our product.
Reach out to clients – show them the actual output and ask: "Does this solve your problem?"

We deliberately refrained from solving hard technological challenges before knowing where we were aiming.

Working Backwards Through Sub-Problems

The manual documents also gave us something crucial: a concrete example to analyze backwards.

For instance, we saw that our documents listed many conditions. This made us realize we would (most probably) need to:

Find conditions in the code.
Filter out business-related conditions (vs. technical conditions).
Explain the condition in the document.

Here's where drawing backwards becomes powerful: We tackled (3) before (2), and (2) before (1).

Why? Because we needed to solve (3) to reach our goal – creating impactful documents.

This means we mocked the first two steps – we assumed we already had a way to find conditions and filter business-related ones. So we already had our list of business conditions, and now the challenge was: for each condition, explain it clearly in the output document.

This approach prevented us from spending weeks on tasks (1) and (2), only to discover that our explanation approach (3) didn't actually work. If we can't accomplish (3) assuming (1) and (2) are solved, then perhaps this entire approach isn't viable, and we need to reconsider alternatives.

The logic is clear: While solving (3) ultimately requires solving (1) and (2), if we fail at (3) even with (1) and (2) accomplished, then pursuing (1) and (2) might not be worth the effort at all.

What does mocking a dependency look like in practice? In our case, mocking steps (1) and (2) meant manually identifying a handful of business conditions from the COBOL code ourselves, rather than building automated systems to find them. We created a small, hand-crafted list of conditions that we knew were business-relevant and existed in our sample programs, and used that as input for testing our explanation approach in step (3).

Why Drawing Backwards Is So Powerful

The advantages of drawing backwards become clear from both the game examples and the COBOL case study:

1. Forces Connection to Product Impact

Like working backwards from spot 1 in the spiral game, starting with "what does successful output look like?" forces you to think about end-user value. You can't start drawing backwards without a clear goal. This prevents the common failure mode of technically interesting Research that doesn't impact the product.

2. Provides a System for Progressing

When Research seems like a huge, daunting task with endless options, working backwards gives you a systematic approach. Just as the spiral game became trivial once you worked backwards to identify the safe spots (1, 8, 15, 22...), Research becomes more manageable when you work backwards from the desired output to identify the dependencies.

3. Validates Each Step Before Investment

Working backwards lets you validate that each sub-problem actually contributes to your goal before you invest significant effort. In the COBOL example, we could verify that our explanation approach (step 3) worked before spending weeks on finding and filtering conditions (steps 1 and 2).

Practical Application: Your Research Tree

Drawing backwards integrates naturally with the Research Tree from chapter 4. When you create your tree:

Start with the end:

Root of tree: "Generate impactful business rule documentation".
First question: "What should successful output look like?"
Approach: Create manual examples.

Then work backwards:

Once you have output, ask: "What do we need to produce this?".
This reveals the actual sub-questions and their dependencies.
Each branch represents a prerequisite you need to solve.

Validate before going deeper:

Before pursuing any branch deeply, ask: "If I solve this, does it actually get me closer to the goal?"
Mock out dependencies to test approaches cheaply.
Use time-boxing (from chapter 5) to limit exploration of any branch.

Summary: Drawing Backwards

Drawing backwards is the heuristic of starting from your desired end state and working systematically toward your current position.

In Product-led Research, drawing backwards means:

Start by defining what successful output looks like (often manually or semi-manually).
Validate the output with stakeholders before technical work.
Work backwards through dependencies, solving them in reverse order.
Validate that each step contributes to the goal before major investment.

This heuristic ensures that Research connects to product impact, since you start with the product goal. It provides systematic progress even when problems seem overwhelming, and makes you validate each step before you invest heavily.

Integration with other tools:

Use with Research Tree (chapter 4) to map backwards dependencies.
Apply time-boxing (chapter 5) to limit exploration of each branch.
Combine with pre-Research checks (chapter 6) to validate product impact.

In the next chapter, you’ll learn about two limitations of drawing backwards and how to address them with continuous end-to-end iterations.

Chapter 8 - End-to-End Iterations

Your most important role is ensuring that Research impacts the product. One major risk: spending weeks on research questions that seem vital, only to discover they don't actually impact the product.

In chapter 7, I advocated for drawing backwards – starting from product impact and working your way back to research questions. This is indeed powerful, especially because it forces you to focus on the end result and validate it with users.

But drawing backwards alone has limitations. Two risks emerge:

Risk 1: Infeasibility in practice Your manually-created "ideal output" might be technically infeasible or impossibly expensive to generate. You won't discover this until you try to build it.

Risk 2: Lack of real-world validation Since you haven't completed an end-to-end process, you probably haven't run your solution on clients' actual data (assuming you can't access it during the manual phase). Continuing our COBOL example from the previous chapter, what works on carefully-chosen examples might fail on real codebases.

These two risks are why I advocate for continuous end-to-end iterations.

Drawing Backwards + End-to-End: A Combined Approach

These aren't competing approaches – they're complementary:

Heuristic	Purpose	What It Gives You	Strength	Limitation
Drawing Backwards (chapter 7)	Define target and path	1. Target output\ 2. Hypothesized chain of steps to get there\ 3. Order of dependencies	Ensures product focus; reveals what you need to build	Hypotheses may be wrong; doesn't validate feasibility on real data
End-to-End Iterations (this chapter)	Validate and build incrementally	1. Proof the chain works\ 2. Learning from real data\ 3. Prioritized improvements	Validates feasibility; discovers what actually works	Can lose direction without clear target and chain

The recommended flow:

Use drawing backwards to:
- Define your target output (manually create examples, validate with users).
- Identify the chain of intermediary steps needed to produce that output.
- Understand the order of dependencies.
Switch to end-to-end iterations to:
- Test whether your hypothesized chain actually works.
- Validate on real data (not just your manual examples).
- Incrementally build toward the target.
Throughout iterations, keep both the target AND the chain from step 1 as your guide.

Drawing backwards already outlines your end-to-end process (Principle 1 below). End-to-end iterations validate and build that process incrementally, learning what works and what doesn't.

The Five Principles of End-to-End Iterations

The end-to-end approach relies on five principles:

Outline the end-to-end process.
Get to end-to-end by simplifying.
Ship it as fast as you can.
Gradually replace steps.
Get frequent feedback on results.

Let me explain each principle using the COBOL business rules example from chapter 7. Since this book isn't about COBOL, I'll keep explanations short while focusing on the methodology.

Principle 1: Outline the End-to-End Process

Start by outlining the entire process from input to output. More accurately, outline the hypothesized end-to-end process – you can't know for certain until you test with users.

If you followed drawing backwards from chapter 7, you already have this. Drawing backwards naturally produces this chain: you started with the target output, asked "what do I need to produce this?", then "what do I need for that?", working your way back to the starting input. That chain IS your end-to-end process outline.

For the COBOL business rules example: We used drawing backwards to identify that our document needed business rule sections, which meant we needed to explain conditions, which meant we needed to filter business conditions, which meant we needed to find conditions, which meant we needed to parse COBOL. Working backwards gave us this chain:

Start with a COBOL program.
Parse the COBOL program into an Abstract Syntax Tree (AST).
Traverse the AST to find conditions.
Filter out business-related conditions (vs. technical conditions).
For every business rule, create a document section explaining the condition.
Sort document sections according to business rule dependencies.

This is your first draft: the hypothesized process that drawing backwards revealed.

How to outline:

Draw boxes on a whiteboard.
Use a flowchart if the process isn't linear.
Keep it visible throughout the research.
Update it as you learn.

The outline serves as your map – it shows you where you are and what you're building toward.

Principle 2: Get to End-to-End by Simplifying

Your goal: make the process work end-to-end. Start with input (a COBOL program), reach the output (a document with business rules), while passing through the intermediate stages.

This sounds like too much. Don't you need to complete the whole research to achieve this?

Definitely not. The trick is to find the easiest way to get end-to-end by taking shortcuts and making assumptions that are definitely too generous for production. You should fight your inner engineer who wants to "do it right" from the start. Your goal is to get an end-to-end process working, as this heuristic is far more valuable than perfecting one step.

This means that some of the steps can be completed manually, or with very simple implementations that you know won't work in production. Remember it is an intermediate milestone, not the final product.

For our COBOL example:

Start with a single, known COBOL program.
Skip parsing – just manually write a list of conditions for the next stage.
The filtering function returns true if business-related, false otherwise.
- First implementation: a simple mapping between input conditions (that you know of) and whether to return true or false.
- Alternative: always return true – yes, you'll get non-relevant rules, but that's a problem for later.
The document generation might also be manual for the first pass.

End result: A rough-looking document, generated by a combination of manual work and code that works solely on a single program. This is far from shippable, but it's an extremely important milestone for ensuring research impacts the product.

You might argue that it's overkill, and why waste this time on manual steps when you'll need to automate them eventually? Even if you don't, I promise from experience that many researchers and engineers feel that way.

From my own experience, I learned (the hard way) that this pays off. Getting to a working end-to-end makes sure you:

Validate that the entire flow works.
Identify bottlenecks and blockers early.
Have something concrete to show and get feedback on.
Prevent spending months on one component that turns out to be unnecessary.

Principle 3: Ship It as Fast as You Can

By the end of Principle 2, you have a working end-to-end process for a single case. You definitely can't ship it yet – it only works on one specific program, certainly not the client's program.

Next milestone: make it shippable.

Does "shippable" mean it works on any program? That's too hard and takes too long. You need shortcuts.

This is where creativity matters. For our COBOL example:

Information gathering:

If running on a client's program, get as much information as possible.
Perhaps assume it's less than 1,000 lines of code.
Perhaps you know which COBOL dialect the client uses, so you don't need to support others (fun fact: COBOL has more than 300 dialects. Well, it's fun for you, not for those who need to support them).

Algorithmic shortcuts:

Use regular expressions to find conditions instead of a full parser.
Yes, you'll miss some conditions – that's a problem for later.

UX shortcuts:

Skip the document generation. Just print rules to the console.
Run from the command line without a GUI.
Manual configuration file instead of user interface.

Your goal is clear: ship it.

It doesn't need to be perfect. It needs to be enough to learn from this iteration. If you don't ship, you can only learn from your intuition – a very bad idea.

Unlike in Principle 2, note that here you don't generate an output (partially) manually - you need a working software that can run on real data.

What "shippable" means:

Works on the client's actual data (even if imperfectly).
Produces output you can get real feedback on.
Doesn't require your manual intervention for each run.

What "shippable" doesn't mean:

Perfect accuracy.
Handles all edge cases.
Production-quality code.
Beautiful user interface.

You can't ship just anything, though. In our example, if you only have a solution that works on one specific test program you created, generating an irrelevant document for the client wastes time and teaches you nothing.

You can't learn from iteration without shipping. And you can't ship without a working end-to-end process. I know it sounds obvious, but many teams miss this in practice as they get into the rabbit hole of solving one step "the right way" before validating the entire flow.

Principle 4: Gradually Replace Steps, While Carefully Prioritizing

Now you have a working end-to-end process. You can start replacing various steps' implementations with better ones:

Replace manual steps with automated ones.
Remove shortcuts and add more robust implementations.
Improve accuracy and coverage

After shipping, you'll have many things you think you must replace immediately. But remember: the goal is making research impact the product. To do that, you need careful prioritization.

The Prioritization Framework

Prioritize changes based on three criteria:

1. Learned Necessity Did you learn that something doesn't work in the current implementation and must be fixed for the product to be viable?

Example: "Regular expressions miss nested conditions, and 60% of the client's business rules are in nested conditions. We must use a real parser."

2. Learning Potential Will changing this implementation help you learn more about product impact in the next iteration?

Example: "If we improve the filtering accuracy from 40% to 80%, we'll learn whether the document format is actually useful when it contains mostly-correct content."

3. Effort Estimation How much time will this change take?

Example: "Building a full parser: 3 weeks. Improving regex to handle nested conditions: 2 days. The latter gives us 80% of the value for 10% of the effort."

Continuing with our COBOL example, you may consider and prioritize these changes following the first iteration:

Changes after first iteration:

- Fix parser to handle nested conditions [Learned: Critical, 2 days] → DO FIRST
- Add GUI for document generation [Nice-to-have, 1 week] → DEFER
- Improve filtering accuracy [Learning: Critical, 3 days] → DO SECOND  
- Support additional COBOL dialects [No evidence needed, very long time...] → DEFER
- Better document formatting [Client mentioned it, 1 week] → DEFER (validate content first)

Iterate fast: change something, ship again, get feedback. Don't solve every issue you find, even issues clients mention. Ask: "What's the most important thing to change to learn something in the very next iteration?"

Principle 5: Get Frequent Feedback on Results

The trick is to be obsessed with getting as much feedback as you can, on each and every iteration.

On each iteration:

Get feedback on the end result
- Show the actual output to users.
- When applicable, don't just ask "does this work?" – also watch them try to use it.
- Identify what works, what doesn't, what's missing.
Understand the next questions to answer
- What did you learn about product impact?
- What assumptions were validated or invalidated?
- What new questions emerged?
Plan the next iteration accordingly
- Use the prioritization framework above.
- Focus on learning, not building.
Implement minimal changes to answer questions
- Don't fix everything.
- Make the smallest changes that will answer your next most important question.
- Keep the cycle fast (days to weeks, not months).

Example iteration cycle:

Iteration 1:
- Shipped: Regex-based condition finder, always-true filter, console output.
- Learned: Document structure works, but too many false positives (noise).
- Question: Is filtering accuracy the blocker to usefulness?
- Next: Improve filtering to 80% accuracy, ship again.

Iteration 2:
- Shipped: Same regex finder, smarter filtering (80% accurate), console output.
- Learned: With better filtering, users found the output useful!
- Question: Do we need a parser, or is regex sufficient?
- Next: Test on larger programs to see where regex breaks down.

Iteration 3:
- Shipped: Same pipeline, tested on 10 real programs.
- Learned: Regex fails on 4/10 programs (nested conditions).
- Question: Parser worth the investment now?
- Next: Build parser for nested conditions, ship again.

Notice: Each cycle is fast, focused on one question, and based on real learning.

In real life, you may want to tackle a few questions per iteration. If two things are clear, fix them before shipping again so you can actually gain valuable feedback rather than hearing the same complaints.

Also, when working with real clients, they might not be as receptive to trying things so many times – so you’ll need to consider that aspect as well. Regardless, the key remains the same: keep cycles short and focused on learning.

Integration with Other Tools

End-to-end iterations work best when combined with other research management tools:

Research Tree (chapter 4):

The outlined process becomes a branch in your tree.
Each iteration tests different approaches on branches.
Failed iterations mark branches red, successful ones mark green.

Time-boxing (chapter 5):

Time-box each iteration.
If you can't ship in the time box, you're building too much.

Drawing Backwards (chapter 7):

Drawing backwards (chapter 7) defines both:
- The target output.
- The hypothesized chain of intermediary steps to reach it.
End-to-end iterations test whether that chain actually works on real data.
The target from drawing backwards acts as your north star throughout iterations.
Each iteration validates or refines the steps that drawing backwards identified.
Use both: drawing backwards reveals what to build, while end-to-end iterations prove it works and let you test it incrementally.

Summary: End-to-End Iterations

End-to-end iterations ensure Research impacts the product by continuously validating feasibility and learning from real data.

The five principles:

Outline the process: Draw backwards already gives you this: the chain from input to output.
Simplify to get end-to-end: Use shortcuts and manual steps to make the whole chain work.
Ship fast: Real data teaches what theory can't.
Prioritize carefully: Use the three-criteria framework:
- Learned necessity (is it broken?)
- Learning potential (what will you learn?)
- Effort estimation (how long will it take?)
Get frequent feedback: Fast cycles (days to weeks) focused on learning.

Part 3 Summary

Back in chapter 2, you learned that your role as a Research leader is to:

Ensure Research connects to product impact.
Ensure Research is done effectively.

Part 2 handled ensuring Research is done effectively.

Part 3 handled ensuring Research connects to product impact – your most important responsibility. This part provided a complete methodology for ensuring product impact through three complementary stages:

The Three Stages of Product-Led Research

Stage 1: Choose research that matters (chapter 6)

Before starting any Research, answer three critical questions:

Product impact: Will success create huge value?
Time to impact: How long until we see product results?
Resources: Do we have the knowledge, capacity, and dependencies?

You learned to distinguish between problem-driven research (starting from validated customer pain points, which is strongly preferred) and opportunity-driven research (starting from new technologies).

Run focused pre-research checks to answer these questions. Only pursue Research when confident about substantial product impact.

Stage 2: Start from product value and work backwards (chapter 7)

Once you've chosen what to research, the drawing backwards heuristic ensures that you start right. Instead of diving into technical challenges, start from the end:

Manually create the desired output: What should successful Research produce? Create it by hand before solving any technical problems.
Validate with stakeholders or customers: Show them the output and confirm it solves their problem.
Work backwards through dependencies: From that validated output, identify what you need to produce it, then what you need for that, working your way back to your starting point.
Solve in reverse order: Tackle the final step first (with earlier steps mocked), validating that each step contributes to the goal before investing heavily in earlier dependencies.

The spiral game example showed why this works: working backwards from the goal reveals systematic solutions that working forward obscures.

Drawing backwards forces product connection because you literally start with product output. It integrates with the Research Tree (chapter 4): While drawing backwards identifies what questions matter, the Tree helps you explore approaches for answering them.

Stage 3: Validate and build iteratively (chapter 8)

Drawing backwards alone has two limitations:

Your manually-created "ideal output" might be technically infeasible to generate.
You haven't validated on real user data.

End-to-end Iterations address both limitations. These two tools aren't competing approaches – rather, they complement one another:

How the Three Stages Work Together

These chapters form a complete methodology for product-led Research:

Choose → Start from the end → Validate iteratively

Chapter 6 ensures you choose research that COULD have huge impact (strategic decision).
Chapter 7 ensures you start from product value (planning backward from validated output).
Chapter 8 ensures you continuously validate with real users.

Each stage prevents different undesired, but painfully common outcomes:

Chapter 6 prevents pursuing research that won't matter (even if successful).
Chapter 7 prevents building technically correct solutions that don't create product value.
Chapter 8 prevents building infeasible solutions or solutions that fail on real data.

You now have the complete answer to "How do I ensure Research impacts the product?":

Choose wisely: Only pursue research with clear, huge product impact.
Start from product value: Manually create and validate desired output before technical work.
Work backwards: Identify dependencies from output back to starting point.
Build end-to-end fast: Get entire chain working with shortcuts and manual steps.
Ship to real users: Validate on actual data, not just examples.
Iterate based on learning: Improve the chain incrementally, prioritizing what teaches you most.

This methodology keeps product impact central at every stage: choosing, planning, and executing. It prevents the most expensive failure: "successful" Research that doesn't affect the product.

Book Summary

You picked up this book because managing Research is different from managing Development – and you needed concrete tools to handle that difference.

What You Learned About Research

In chapter 1, you learned that Research isn't about difficulty or technical sophistication. It's about uncertainty of approach – that is, confronting problems where you don't know if a solution exists, where multiple approaches might work but you're not sure which, and where the path to success isn't immediately clear.

You saw how Alan Schoenfeld's problem-solving framework breaks down the Research process into four components:

Knowledge base – what you know.
Heuristics – strategies for approaching problems.
Control – monitoring and adjusting your approach.
Beliefs – your mindset toward the problem.

The good news: all four can be improved with the right management and methods.

In chapter 2, you explored the distinction between Research and Development more deeply. You learned that your role as a Research leader has two parts:

Ensure Research connects to product impact: "Successful" Research that doesn't affect the product is a failed project. This is the most important part in Product-led companies.
Ensure Research is done effectively: Even brilliant researchers benefit from structured approaches.

This two-part framework organized everything that followed.

How to Do Research Effectively

Part 2 gave you concrete methods for effective Research execution – tools that work in any research context (whether it's Product-led or not).

In chapter 3, I shared with you a personal reverse engineering classroom story. Students with sophisticated technical skills missed the obvious solution (checking the Help menu) because they lacked structured methodology. The story clearly showed that the problem isn't capability, it's often approach. This illustrated why you need the methods that follow.

Chapter 4 introduced the Research Tree method – a living visual framework for systematically exploring solution paths. You learned:

How to map questions you need to answer and approaches for answering them.
A decision framework for choosing which approach to try first: fastest feedback, lowest cost, best coverage.
How using the tree helps avoid common failure modes: jumping on the first idea, tunnel vision, inefficient learning, answering questions you don't need to, and lost context.

The Research Tree helps you implement Schoenfeld's "control" component – helping you monitor and adjust your approach systematically rather than randomly trying things. It is helpful for a researcher, but as a Research leader, it allows you to guide your team effectively.

In chapter 5, you learned how to manage exploration without killing creativity. Since Research estimation is inherently difficult, time-boxing provides structure by setting time limits for specific research directions. After the allocated time, you stop to reconsider: What did you learn? Is this still the most promising path? This tool acknowledges uncertainty while preventing endless exploration, or diving too deep into rabbit holes that don't necessarily help your Product goals.

How to Ensure Product Impact

Part 3 focused on your most important responsibility: ensuring Research creates product value.

Chapter 6 showed you how to choose what directions to research – and more importantly, what not to pursue. You learned the distinction between:

Problem-driven research – starting from customer pain points (strongly preferred, lower risk).
Opportunity-driven research – starting from new technologies (higher risk, needs validation).

Before pursuing any Research initiative, you need to answer three questions:

Product impact: Will success create huge value?
Time to impact: How long until we see product results?
Resources: Do you and your team have the knowledge, capacity, and dependencies needed?

You saw how to run focused pre-research checks to answer these questions.

Chapter 7 introduced a powerful heuristic for ensuring product connection: start from the end and work backwards. Through the spiral game example, you saw how working backwards reveals systematic solutions that working forward obscures.

In the COBOL business rules case study, you saw a practical application:

Start by manually creating the desired output (before solving any technical challenges).
Validate that output with stakeholders.
Work backwards through dependencies, solving them in reverse order.
Validate that each step contributes to the goal before major investment.

Drawing backwards forces connection to product impact because you must start with the product goal.

Chapter 8 showed you how to address two limitations of the drawing backwards heuristic through continuous end-to-end iterations. Your manually-created "ideal output" might be infeasible to generate, and you haven't validated it on real data. End-to-end iterations solve both problems.

You learned five principles to run effective end-to-end iterations:

Outline the end-to-end process: Drawing backwards already gives you this chain.
Get to end-to-end by simplifying: Use shortcuts and manual steps to make the whole chain work.
Ship it as fast as you can: Real data teaches what theory can't.
Gradually replace steps: Prioritize based on learned necessity, learning potential, and effort.
Get frequent feedback: Fast cycles focused on learning.

Drawing backwards reveals what to build and in what order, while end-to-end iterations prove it works and build it incrementally.

Your Toolkit for Research Management

You now have a complete toolkit:

From Part 2 (works for any research):

Research Tree – maps solution space, chooses approaches systematically.
Time-boxing – manages exploration with structure.

From Part 3 (specifically for product impact):

Choosing frameworks – decides what deserves Research effort.
Drawing backwards – forces product connection from the start.
End-to-end iterations – validates feasibility and learns from real users.

These methods work together. Drawing backwards identifies your goal and the chain of steps. The Research Tree maps approaches for each step. Time-boxing prevents getting deep into a rabbit hole when you should reconsider based on what you've learned, while acknowledging the inability to provide exact time estimates on Research tasks. End-to-end iterations validate and build incrementally.

My Message To You

Research is fundamentally uncertain work. Applying traditional Development management practices fails because it assumes known solution paths, predictable timelines, and steady progress.

But Research doesn't have to be mystical or random. With the right frameworks, you can manage it systematically while maintaining focus on what matters: creating product value. You learned to ensure that Research is done effectively (part 2) and that it connects to product impact (part 3). You have concrete tools, real examples, and a clear framework for both responsibilities.

Now go turn your team's uncertain Research into systematic progress toward measurable product impact. I am confident that you can lead Research teams to success, and I would be happy to hear about your experiences applying these methods.

If you liked this book, please share it with more people.

Acknowledgements

I am extremely lucky to have such wonderful people supporting me in this journey.

Abbey Rennemeyer has been a wonderful editor. Abbey had edited my posts for freeCodeCamp over the past few years, as well as my previous book Gitting Things Done, so I knew she was the perfect fit for this book as well. Her insights both on the content and the writing style have greatly improved this book.

Quincy Larson founded the amazing community at freeCodeCamp. I thank him for starting this incredible community, his ongoing support, and for his friendship.

Estefania Cassingena Navone designed the cover of this book. I am grateful for her professional work and her patience with my perfectionism and requests.

Beta readers who contributed their time and mind to read through an unfinished version of this book to improve it for all of you have helped me get it to its current shape. Specifically, I would like to thank Jason S. Shapiro and Omer Gull for their insights.

Dr. David Ginat introduced me first to Alan Schoenfeld's problem-solving research during my time at Tel Aviv University. His teaching inspired me to apply these ideas in practical contexts, including Research management.

I was privileged to work with many brilliant researchers and engineering leaders over the years, too many to name here.

To readers of my previous book, Gitting Things Done, who were kind enough to provide feedback and support – you are awesome. Receiving your emails and comments made me feel like there is a real reason to keep writing.

If You Wish to Support This Book

If you would like to support this book, you are welcome to buy E-Book version, Paperback, Hardc over , or buy me a coffee. Thank you!

Contact Me

If you liked something about this book, or felt that something was missing or needed improvement – I would love to hear from you. Please reach out at: gitting.things@gmail.com.

Thank you for learning and allowing me to be a part of your journey.

- Omer Rosenbaum

About the Author

Omer Rosenbaum is an established technologist and writer. He's the author of the Brief YouTube Channel, and the books Gitting Things Done and Computer Networks (in Hebrew). He's also a cyber training expert and founder of Checkpoint Security Academy.

How to Run LaTeX Projects Locally (for Free) On Windows

Md. Fahim Bin Amin — Tue, 25 Feb 2025 15:44:04 +0000

LaTeX is a high-quality typesetting system that is widely used in technical, academic, and scientific writing. It’s very popular in academia, especially in fields like mathematics, physics, computer science, and engineering.

LaTeX is not a word processor like Microsoft Word – rather, it’s a document preparation system that allows you to focus on the content of your writing while it handles the formatting. If you use LaTeX to write your formal documents (like a CV, résumé, or research paper), then you don’t need to worry about the formatting and structure, as everything can be done using LaTeX scripts.

If you use LaTeX to write your academic or research papers, you might be familiar with website-based applications like Overleaf. Overleaf is a website that allows anyone to read, write, and compile LaTeX scripts online.

These sites are okay for small tasks or compilations, or if you need only a little bit of free collaboration. But if you need to work on bigger projects or need to conduct many collaborative tasks, then the free tier may be insufficient. And in my opinion, the paid subscription costs too much.

But don’t worry: running LaTeX locally may be the perfect solution for you. I know this because I also faced a similar situation, and this simply changed my life! I also keep all of the tracks in Git (GitHub, GitLab, and so on) along with unlimited collaboration opportunities and compilation. And the great thing is, all of these are completely free as it’s all happening on my local machine.

So in this article, I am going to discuss the methods in detail. I have also created an in-depth video for you to understand how this works.

Video Tutorial

Resources You’ll Need:

1. GitHub Repository

This entire guide is available in one of my GitHub projects named Install-LaTeX. The live website is available here (fahimfba.github.io/Install-LaTeX) as well. I would highly appreciate it if you star (⭐) the repository. Also, you can create issues there if you face any problems. Any kind of good contribution is also welcome here.

2. Operating System

You can install LaTeX on any major operating system (Windows, MacOS, and Linux-based OSes). But in this article, I am only going to talk about the Windows operating system.

Here, I’m using the latest Windows 11 operating system, but the same procedure should be applicable to all of the Windows-based operating systems that are going to come out in the future. Windows 10 should also be okay too.

3. Editor

I am going to use the popular Visual Studio Code as my editor. It is a 100% free and robust editor that’s very popular among devs all over the world. If you don’t already have it, go ahead and install it before proceeding further.

4. LaTeX Compiler/IDE

To work on LaTeX files, you’ll need a specific compiler. I am going to use MikTeX. There are other tools out there, but this is the best tool right now (according to me!). It is completely free and supports all major operating systems as well. It also has a built-in IDE, but we are going to use VS Code as our main editor.

Download the Windows executable file from the Download section.

After the download is finished, install the executable. At the end of the installation, keep the tick in “Check for updates now”.

You will find the MikTeX console in your taskbar. Open that.

Go to the “Updates” tab and click “Update now”. It will install all of those packages.

At the end, it will prompt you to close the console. Click “OK”. Open MiKTeX again.

That’s it for this tool.

5. Perl

The commands we are going to execute for building the LaTeX files are dependent on Perl. As the Windows operating system doesn’t come with a built-in Perl compiler, we are going to install the Strawberry Perl.

Download the latest MSI package from it.

Install the executable after it gets finished downloading the application.

We need to add Perl’s path to the system environment. To do that, go into the location where it has been installed. By default, it gets installed inside C:\Strawberry\perl\bin directory. Copy the path.

Now search for “env” in the Windows search bar until you find something called “Edit the system environment variable”.

Now click on “Environment Variables…”.

Now select “Path” from “System variables” and click “Edit”.

Click “New”. Paste the path. Now exit every windows sequentially by clicking on “OK” in each window.

Visual Code Studio Extensions

We need some extensions in VS Code to streamline our workflow.

First, let’s get LaTeX Workshop. It is the core extension for working with LaTeX files inside VS Code Studio.

Next, you’ll need Rewrap. It is an amazing tool that lets you wrap longer lines. It helps you work in a long line in separate lines without breaking any structure or sentence.

Build the LaTeX File

Whenever you want to build any LaTeX file inside VS Code studio, simply open that file in it. Then open the command palette using Ctrl + Shift + P.

Search for “LaTeX Workshop: Build with recipe” and go there. It will start building the file. Whenever it prompts you to install any missing package, untick the box that says “Always show this dialog” and press “Install”. I do this because clicking on “Install” on hundreds of prompt windows for building a LaTeX file is very difficult for me.

After it finishes building the LaTeX file, you will get the output PDF file inside VS Code. You can open the PDF file directly in VS Code.

If you want to go into any specific line in the code from the output PDF file like Overleaf, simply click on that specific portion in the PDF by pressing the Ctrl key. It will immediately take you to the code part where it belongs.

That’s it! It’s now running on your local machine and there are no restrictions or limitation to it, literally! Also, for collaboration and keeping track of the history, using Git is the best option, like I do.

Conclusion

Thanks for reading this short tutorial. I hope it helped you interact more easily with LaTeX.

You can follow me on GitHub, LinkedIn, and You Tube to get more content like this. Also, my website is always available for you!

The Business of Technology Research – Who Funds It, And What's Being Done

David Clinton — Tue, 07 Mar 2023 17:07:38 +0000

Getting a new technology out to consumers will usually require good people and boat loads of resources – including money. Generally, lots of money.

A lot of that money will be spent on research. And, more often than not, the hard research needed to translate a great idea into a usable product will be performed by someone whose job title isn't "entrepreneur." Sometimes, in fact, the research will be done by individuals who are barely aware that their innovations have any commercial value at all.

If you're here because you want to get the jump on cutting edge technologies, then you may want to keep an eye on the organizations that are known to produce practical research.

Knowing who's big in research, who's funding it, and where the big bucks are being spent can give you useful insights into what might be coming next. From there, you're just a step away from, say, spending time learning the tools that'll come with the new tech or positioning yourself to profit when it finally shows up.

This chapter was taken from the book, Keeping Up: Backgrounders to All the Big Technology Trends You Can't Afford to Ignore. If you'd prefer to watch this chapter as a video, feel free to follow along here:

Who Funds Commercial Science and Why?

Once upon a time, major breakthroughs in serious scientific research were the products of private patronages. The Italian Medici family, for instance, famously supported many individuals whose work would prove to be pivotal, including Leonardo da Vinci and Galileo.

But the years leading up to the Second World War saw the scope and complexity of research projects growing far beyond the capacity of private support. The war's dependence on unprecedented technological complexity – exemplified by the work of the Manhattan Project building the atom bomb – pushed more and more research under government charge.

Government involvement in research has continued in the generations since the war. Still, it's been estimated that universities and governments are responsible for only 30% of research funding between them, with most of the rest provided by private industry (see this article for more info).

Let's see how that breaks down.

Funding from Taxpayers

Democratic governments, of course, don't spend their own money, of which they traditionally have none. Their many programs and services are funded by revenues raised, one way or another, from their capital assets and from their populations. In modern nation states, "populations" would mean those individuals and corporations who pay taxes.

Public research and development can be performed within government agencies. According to the terms of some agency mandates, research results must immediately enter the public domain.

But even those who retain rights to their research will often point their work towards businesses and institutions that can use it productively. The US National Science Foundation (NSF), for instance, uses its $8 billion annual budget to fund "approximately 25 percent of all federally supported basic research conducted by America's colleges and universities" (Source).

Other American agencies do much or all of their research in-house. Here are some examples:

The National Institute of Standards and Technology (NIST) has a mandate to "promote innovation and industrial competitiveness." One very important part of that mission is maintaining the National Vulnerability Database (NVD) which plays a foundational role in the management of the vulnerability assessment and detection systems protecting our IT infrastructure.
The US military's Defense Advanced Research Projects Agency (DARPA) collaborates with private and public sector partners to aid in the development of emerging technologies. Work in recent years has included research into robotics and autonomous vehicles, but you might be more familiar with a DARPA innovation from a few decades ago: the internet.
The National Institutes of Health (NIH) employs 6,000 research scientists across 27 research institutes and centers. Their "mission is to seek fundamental knowledge about the nature and behavior of living systems and the application of that knowledge to enhance health, lengthen life, and reduce illness and disability."

The complete list of US government research agencies makes for quite a read. Take a look for yourself.

Naturally, governments of other countries have their own research agencies. One example is Canada's National Research Council (NRC), which has evolved from its military technology origins through the two world wars, to its current focus on partnerships with private and public-sector technology companies.

The NRC now divides its work into four "business lines:"

Strategic research and development
Technical services
Management of science and technology infrastructure
NRC-Industrial Research Assistance Program (IRAP)

As I mentioned when discussing the NSF, a significant proportion of taxpayer funds directed towards research and development are granted to public and private colleges and universities.

But, from the college perspective, how much academic R&D funding comes from government sources?

A 2016 review of the 20 US colleges that spent the most on R&D found that they each spent between $837,000 and $2.4 million. It also found that between 47-87% of their total spending came from government sources of one sort or another (see this article for more info). By contrast, businesses only provided between 2 and 22% of that funding.

Private Charitable Funding

While we're on the subject of academic research, we shouldn't ignore a third source of funding: private endowments. Some – although not all – permanent endowments were targeted by their donors at research activities. Although the fund capital can't be spent each year, the income that capital generates can.

Harvard university famously – or perhaps infamously – has a total endowment greater than 40 billion dollars. Some of that undoubtedly finds its way to R&D.

Curiously, according to that 2016 study, Harvard's total R&D spending that year – including activities funded by governments (52.1%), businesses (4.7%), and endowments – was just over one million dollars.

Of course, donations support plenty of research outside of academic settings, too. Most serious diseases have associated charitable foundations that exist to raise money for both victim care and medical research.

Also, many thousands of registered non-profits exist throughout the world supporting non-medical causes, including many that involving technology-related research. The Bill & Melinda Gates Foundation is a particularly well-known example.

Funding from Corporations

Technology-oriented companies have a strong interest in getting their hands on innovations before their competition. To improve their chances, many will run their own research labs in-house.

The Bell Telephone Company, for instance – and its successors including American Telephone & Telegraph Company (AT&T) – maintained the active and enormously creative Bell Labs. Bell Labs, under various names, was responsible for many innovations, including the transistor, lasers, and the Unix operating system.

Individual technologists at some companies are often sources of innovation. 3M, for instance, has what they call a "15% Culture," where employees are allowed to use company time and space to pursue research based on their own ideas and interests. Over the years, the program has generated successful products for the company, including their sticky paper Post-its.

In another example, Percy Spencer, working on radar for US defense contractor Raytheon, accidentally discovered that microwaves could cook food.

It should be noted that not all corporate innovation is truly home-grown. A lot of it is actually funded indirectly through government money in the form of tax incentives or credits. Under such programs, companies may be permitted to use research-related spending (including salary expenses) to reduce the income taxes they would otherwise pay.

Major Fields of Commercial Technology Research

Trying to grasp the full scope of technology development at this point in history is an unforgivable waste of time. There's serious innovation going on every minute of the day, in every time zone, through countless labs, office towers, warehouses, garages, basements, bedrooms and, of course, invisibly within creative people's minds.

No one's keeping track of it all because it's not possible. Not to mention the fact that much of that innovation happens under a thick shroud of secrecy.

But it's probably worth offering just a couple of examples to give you a feel for where to look.

Quantum Computing (and Why We Should Care)

A cousin of mine with an advanced physics degree from Cambridge University once tried to explain quantum mechanics to me. He failed. Miserably. My poor old brain just couldn't absorb it. So don't expect any full, measured descriptions of the underlying science here.

Instead, I'll try to show you how experimental compute technologies that depend on the physics might work, and what can be done with them.

The super-quick executive summary version of this is that computers powered by one quantum technology or other will work a lot faster than any of the super-est of super computers we have now.

So much faster, in fact, that they may be able to solve problems that would be simply unfeasible using traditional computers (an achievement known as quantum supremacy). This would mean that some long-held assumptions about the way software works will no longer apply.

For instance, the reason the best encryption tools we currently use to protect sensitive data work, is because it would take hundreds or even thousands of hours of high-performance compute time to successfully break the encryption key. In most cases, it's just not worth the effort and expense.

But if you could easily buy time on a computer that processed operations exponentially faster, then two things would immediately happen:

Cracking encryption algorithms would become trivial
Honest folk would have to seriously look for a new way to protect their data

Currently, Google and IBM are among the major companies that have invested heavily in quantum compute research projects.

As well as I can understand it, quantum computers would measure the state of subatomic particles and use that binary measurement to represent a computational value. The description of that state is known as a qubit, which is effectively the quantum equivalent of traditional computing's bit.

But because a qubit can also exist within what's known as coherent superposition – meaning that its value can exist in a superposition of two possible states – it can be used to represent a more complex range of values.

And that, I'm given to believe, means that such computers will be able to do stuff much, much faster than they can now. If this actually happens, it'll be big.

Energy Technologies

The modern world consumes an awful lot of energy. We're constantly moving about, controlling our indoor (and in-transit) climate conditions, exchanging information, and expecting that all the world's riches be delivered to our doorsteps. By tomorrow.

But those energy-thirsty activities come with costs, not the least of which from the emissions they leave behind. The search for reliable, steady, and affordable energy sources that can help us find a healthy balance between consumption and emissions is ongoing – and unimaginably expensive.

Small modular nuclear reactors (SMRs) have been the focus of some serious developments in recent years. They appear to promise reliable, steady, and affordable energy in ways that their expensive and complex nuclear predecessors couldn't.

First and second generation reactors were, overall, reliable and steady – and they were clean – but their massive capital costs and large physical footprints made them more than a bit inflexible.

The idea behind SMRs is that highly efficient reactors can be manufactured off-site and delivered on trucks one module at a time for on-site assembly. The design makes the per megawatt generation of power far cheaper and project completion much faster. And it allows the deployment of nuclear power to service smaller markets that previously couldn't consider it as a realistic option.

As the name implies, SMRs are smaller than traditional reactors. They're designed to deliver between 50 and 300 MW of electricity each, compared with the 800 to 1,200 MW outputs that were previously common.

Companies heavily involved in this research include Britain's Rolls-Royce and an American company with historical connections to the US Department of Energy called NuScale Power. Various governments around the world have also invested in the technology one way or another.

Medical Technology Research

If you think we're spending a lot of money on energy, wait 'till you see how much health care costs.

Across the 37 Organisation for Economic Co-operation and Development (OECD) nations, health care industry spending accounts for around 10% of the total gross domestic product. That's more than $3,000 a year for every single living person.

On the one hand, with all that money being thrown around, there are undoubtedly many business opportunities waiting to be discovered. But there's also a lot of room for new and innovative technologies that can improve the delivery of health care while reducing the costs. Here are two excellent candidates:

Telehealth involves the provision of health services (like patient-doctor consultations) through a telecommunication medium. This might mean having a simple telephone conversation rather than a visit to the office, but it could also incorporate video conferencing tools or even the use of remote diagnostic equipment.

For example, small, remote communities could maintain imaging facilities and technicians even many hundreds of miles away from the nearest medical labs and radiologist specialists. Digital connections can permit distant doctors to view, say, ultrasound results, speak directly with patients, and confidently reach diagnoses. And all without the need for anyone to undertake exhausting and expensive travel.

Telehealth also allows for meaningful patient-doctor contact without the risk of spreading disease.

Telesurgery is an extension of telehealth which can allow some surgical procedures even when doctors are many miles away from their patients. The technology makes use of high-definition video feeds and purpose-built robotic arms that can be controlled by doctors remotely.

Telesurgery tools have the potential to save money for cash-strapped health systems but, more importantly, they can improve health care and save lives.

So whether you're thinking about building the tech behind the next mega business model, learning a new technology, or looking for a good investment, having a good sense of where the serious tech research happens and who pays for it can be useful.

YouTube videos of all ten chapters from this book are available here. Lots more tech goodness - in the form of books, courses, and articles - can be had here. And consider taking my AWS, security, and container technology courses here.

What is the Difference Between an Independent Variable and a Dependent Variable?

Kolade Chris — Thu, 15 Dec 2022 17:32:36 +0000

The meaning of the word "variable" depends on the field where it's being used.

In programming, a variable is a particular piece of data that holds a value. Depending on the configuration, that value can change or remain fixed.

For instance, in JavaScript, you can implement a variable to change over time with the "var" and "let" keywords. But if you want, you can implement a variable so it doesn't change with the "const" keyword.

In this article, the kind of variable we’ll be looking at is not the one in programming but the one you'll deal with in research. Precisely, we’ll look at the differences between the two main types of variables in research – dependent and independent variables.

But before we look at the differences between dependent and independent variables, we need to understand what a variable is first.

What We'll Cover

What is a Variable in Research?
What are Dependent and Independent Variables?
What are the Differences between Dependent and Independent Variables?
How to Identify a Dependent Variable from an Independent Variable
Conclusion
More Readings

What is a Variable in Research?

If you’re conducting research, you’ll be measuring a lot of values. So, in research, a variable is anything you’re trying to measure. It could be age, temperature, length, height, mass, weight, or any other thing that can have a value.

In addition, you’ll be measuring those variables in different units – centimeters (cm), meters (m), grams (g), kilograms (kg), and many more.

These units can’t be neglected, but as far as variables are concerned, whether dependent or independent, the values are what really matter.

What are Dependent and Independent Variables?

Dependent and independent variables depend on whether one variable determines the outcome of the other or not.

A dependent variable is a variable whose changes and its outcome depend on another variable. On almost all occasions, the variable the dependent variable depends on is an independent variable.

Dependent variables are also called the response or outcome variables because they represent the outcome of the values you're measuring. That is, what you record after manipulating the independent variables.

An independent variable is a variable whose outcome or changes do not depend on another variable. It is the exact opposite of the dependent variable, at least according to what the name implies.

Independent variables are also called predictor variables because you can use them to predict the outcome of a dependent variable. That is, when you manipulate independent variables, they can give you the outcome of a dependent variable.

What are the Differences between Dependent and Independent Variables?

Basis	Dependent Variable	Independent Variable
Type	It is the "response" variable	It is the "effect" variable
Outcome	Outcome depends on another variable (usually the independent variable)	Outcome does not depend on another variable
Changes	This variable changes over time. Consider the dependent variable as a variable you declare with the "let" or "var" keyword in JavaScript. You can later change it.	This variable never changes. Consider the independent variable as the variable you declare with the "const" keyword in JavaScript. It is fixed unless you explicitly change the value.
Manipulation	Dependent variables cannot be manipulated because their value depends on the independent variable.	Independent variables can be manipulated to determine the outcome of a dependent variable.
Position on a Graph	Dependent variables are placed on the y-axis (vertical axis) on a graph	Independent variables are placed on the x-axis (horizontal axis) on a graph

How to Identify a Dependent Variable from an Independent Variable

We've taken a look at what variables are, what dependent and independent variables are, and the exact differences between dependent and independent variables.

But how exactly would you differentiate a dependent variable from an independent variable? We are going to look at two experiments or examples:

how Vitamin A helps mothers produce milk
the level of light nocturnal insects (insects active at night) are attracted to

For the first experiment, the sources of vitamin A the mother takes in from foods like fish oil and green vegetables are the independent variable. That's because the researcher can change [decrease or increase] the amount given to the mothers. How the body of the mother responds in producing more milk is the dependent variable.

For the second experiment, the level of light is the independent variable because the researcher can change it. How nocturnal insects react to that level of light is the dependent variable.

Conclusion

In this article, you learned about the differences between dependent and independent variables. We looked at:

what a variable is in research
the two main types of variables (dependent and independent variables)
what dependent and independent variables are
and how to differentiate a dependent variable from an independent variable.

I hope this article gives you a knowledge of what research variables are and how to differentiate a dependent variable from an independent variable.

Thank you for reading.

How to Google like a Pro – 10 Tips for More Effective Googling

Soham De Roy — Fri, 12 Aug 2022 18:19:11 +0000

Pretty much everything is available on the internet, and Google is one of the main ways people search for information.

And while you might think it's obvious, many internet users don't know how to use Google search efficiently and effectively.

In truth, Googling is an art.

To get the right answers, you need to ask the right questions. And to get the right answers quickly, you must know how to ask the right questions.

Everyone should learn the how part, and that's what this tutorial is all about. Below are some helpful tips and tricks you can use to find correct answers to your questions effectively and efficiently.

How to Search Google Effectively

1. Use quotes to get an "EXACT" match

Generally, when we do a regular search on Google, it makes a shallow search. This may result in pages that may or may not contain all the words you mentioned in your search query.

But if you wrap your search query or question (either a term or a phrase) in quotes like this: "your question goes here" then Google will do a deep search. This means that all the results will include pages that contain all the terms in your question that you put within quotes.

This is helpful when you absolutely want a particular term or phrase to be there in your search results.

2. Search within a specific site with `site:`

If you want Google to return results from within a particular website, just add site: before the question you are searching.

This is also helpful when you want to search for something within a website that doesn't have an internal search option or perhaps doesn't have a very effective one.

For example, searching for site:freecodecamp.org react will populate search results only from the www.freecodecamp.org website on the topic of React.

3. Exclude a term from search results with `-`

If you don't want a term or phrase to appear in your search results, then just add - in front of that word.

For e.g. a search on => how to write components in React -class will return all the search results that don't have the term "class" in them. So, (if you know a bit of React) it'll give you only ways to write functional components in React.

4. Search images of a particular size with `imagesize:`

If you want to search images of a particular size, then use the tag imagesize: in your search query along with the width and height in pixels.

Dimension should be in pixels only – for example, imagesize:500x500 will populate image results which have a dimension of 500px x 500px. So a search on => "cute dog images imagesize:500x600" (coz why not) will result in pictures of cute dogs with an image size of 500px x 600px.

5. Search for a particular filetype with `filetype:`

If you want to get search results that contain a particular file type such as PDF or PPT, then add filetype: (without the angular brackets). For example, react tutorial filetype:pdf will generate results that contains PDF in them, as seen in the following image:

6. Use wildcard `*` to make searches

If you are unsure about or have forgotten any term in your search query/ question, then use the wildcard * charachter. Google will replace it for you with relevant terms.

For example, a search on => the of money will populate the following results. It will be a bunch of pages that include the phrase such as "the exchange of money", "the use of money", "the role of money", "the psychology* of money", and so on.

7. Combine searches with `OR`, `AND` logic

If you want your search results to contain two terms, then put the AND keyword in between them. For example, the search on => React AND Angular will fetch results that have both the terms react and angular in it.

Similarly, if you want either of the terms in your search results, then use the OR keyword in between them. For example, the search on => React OR Angular will fetch results that have either of the terms or even both.

8. Filter out searches with `AFTER:`, `BEFORE:` or `..` between two numbers

If you want Google to populate search results that were published after a particular year, then use the tag AFTER:. For example, the search on => React tutorials AFTER:2020 will populate search results published after 2020.

Similarly adding the tag BEFORE: will return results published before a particular year.

You can also search for results published in a certain year range, or for that matter between any numbers. Just add .. between the two numbers you want to search between along with the units, if any.

9. Check out related websites using `related:`

If you want to know what all other websites are available on the internet that are similar to a particular website, then use the related: tag.

For example, a search on related:google.com will fetch all the websites that are similar to Google like Bing, Yahoo, DuckDuckGo, and so on.

10. Use `cache:` to see Google's cached version of a website

Google stores a cached version of websites to provide search results quickly. To see if Google has cached any site or not, just use the tag cache: in front of the website's URL.

This is especially helpful for web developers if they want to check if they are currently viewing a cached site or the latest version of a website after they have pushed some changes to the website. For example, a search for => cache:www.sohamderoy.dev on the day of writing this tutorial (that is, 4th Aug 2022) returns the following result:

As you see, even though I am searching for the site on 4th August 2022, I still see a copy of the site from 3rd August 2022, as that is when Google has cached the website.

It is important to note that "Not all search operators return exhaustive results." as mentioned by Google itself.

Still, I believe that these are some really handy tips that help you make an effective and efficient Google search. I hope I've explained them properly and have encouraged you to use them in your regular Google search.

Wrapping Up

Thanks for reading! I really hope you enjoyed reading this short article about how to google efficeiently and effectively and found this tutorial useful.

Do consider sharing it with your friends – I'd really appreciate that. Follow me on LinkedIn and Twitter and stay tuned for more amazing content! Peace out! 🖖

How Search Google Like a Pro – Tips to Get Better Search Results

Hillary Nyakundi — Wed, 10 Mar 2021 16:53:16 +0000

If you want to find the right answer to a problem you have, you need to know how to ask the right question. For most people this is the most difficult thing to do – asking the right question without beating around the bush.

To find your answer, you have to be able to point straight at what you want. You will also need a good search engine to help you out.

What is a Search Engine?

According to the OED,

"A search engine is a program that searches for and identifies items in a database that correspond to keywords or characters specified by the user, used especially for finding particular sites on the World Wide Web."

Yep you read that right – items that correspond to keywords we search for. Now there are different types of search engine like:

Google
DuckDuckGo
Yahoo
and many more...

Our main focus for this article is GOOGLE and how to learn how to use it like the pro you are.

So, what is Google? It's one of the most used and popular internet search engines. And it uses its own secret algorithm to fetch and organize search results in order to give you the most relevant and reliable sources of data possible.

Let's have a look at some tricks to change how you use it.

How to Search Google Like a Pro

Searching Google is a skill, believe it or not. It takes practice, but if you employ these techniques you'll soon be finding answers to even your trickiest questions.

1) Use the Keyword related: to find similar websites.
If you were to search anything related to freeCodeCamp it would look something like the below, showing all related sites to freeCodeCamp.

2) Use the Keyword search: to search within a website.
Let's say you wanted to find something related to web development on freeCodeCamp this is how to use the search: keyword to get right answers:

3) Use the Keyword ... to search within a time frame.
For example say you wanted to find an article about developers or anything to do with developers between 1992 and 1999, this is what it would look like : Developer 1992...1999

4) Use the Keyword - to exclude a word.
If I wanted to know more about Android development but didn't want to include flutter related skills in my search results, I would search it like this: android development -flutter

5) Use the keyword to replace missing words. Trying to search for something but you happen to have forgotten some words? Don't worry, try using the `` to replace the missing word that you have forgotten. Like this:

6) If you're looking for a definition of a word, use the keyword define: to find a definition.
If I wanted to find the definition of object oriented programming, this would give results of all articles with definitions of object oriented programming. This is how I would do it:

7) You can search for an exact phrase using double quotation marks " " If I wanted to find free tutorial on Python I would do it like this:

8) Use site: to search for a particular website or content. For example if I wanted to find a site with blogs related to web development this how we would do the search:

And here's some extra tips for you:

Want to go Incognito without having to use your mouse to find the option? Try pressing Ctrl + Shift + N.
Are you in an open tab and you want to easily close it without using the mouse to click the x? Try pressing Ctrl + W.

Conclusion

There you have it – 8 Google search tricks to make you a Google search wizard.

Now get out there and start using your newfound magical search powers to conquer the world! Or at least, to find better search results. Either way, I hope you found this article helpful. If you did, don’t forget to share it!

Connect With me at Twitter | Insta | YouTube | LinkedIn | GitHub

Do share your valuable opinion, I appreciate your honest feedback!

Enjoy Coding ❤

How to Read a Research Paper – A Guide to Setting Research Goals, Finding Papers to Read, and More

Harshit Tyagi — Thu, 04 Mar 2021 19:01:49 +0000

If you work in a scientific field, you should try to build a deep and unbiased understanding of that field. This not only educates you in the best possible way but also helps you envision the opportunities in your space.

A research paper is often the culmination of a wide range of deep and authentic practices surrounding a topic. When writing a research paper, the author thinks critically about the problem, performs rigorous research, evaluates their processes and sources, organizes their thoughts, and then writes. These genuinely-executed practices make for a good research paper.

If you’re struggling to build a habit of reading papers (like I am) on a regular basis, I’ve tried to break down the whole process. I've talked to researchers in the field, read a bunch of papers and blogs from distinguished researchers, and jotted down some techniques that you can follow.

Let’s start off by understanding what a research paper is and what it is NOT!

What is a Research Paper?

A research paper is a dense and detailed manuscript that compiles a thorough understanding of a problem or topic. It offers a proposed solution and further research along with the conditions under which it was deduced and carried out, the efficacy of the solution and the research performed, and potential loopholes in the study.

A research paper is written not only to provide an exceptional learning opportunity but also to pave the way for further advancements in the field. These papers help other scholars germinate the thought seed that can either lead to a new world of ideas or an innovative method of solving a longstanding problem.

What Research Papers are NOT

There is a common notion that a research paper is a well-informed summary of a problem or topic written by means of other sources.

But you shouldn't mistake it for a book or an opinionated account of an individual’s interpretation of a particular topic.

Why Should You Read Research Papers?

What I find fascinating about reading a good research paper is that you can draw on a profound study of a topic and engage with the community on a new perspective to understand what can be achieved in and around that topic.

I work at the intersection of instructional design and data science. Learning is part of my day-to-day responsibilities. If the source of my education is flawed or inefficient, I’d fail at my job in the long term. This applies to many other jobs in Science with a special focus on research.

There are three important reasons to read a research paper:

Knowledge — Understanding the problem from the eyes of someone who has probably spent years solving it and has taken care of all the edge cases that you might not think of at the beginning.
Exploration — Whether you have a pinpointed agenda or not, there is a very high chance that you will stumble upon an edge case or a shortcoming that is worth following up. With persistent efforts over a considerable amount of time, you can learn to use that knowledge to make a living.
Research and review — One of the main reasons for writing a research paper is to further the development in the field. Researchers read papers to review them for conferences or to do a literature survey of a new field. For example, Yann LeCun’s paper on integrating domain constraints into backpropagation set the foundation of modern computer vision back in 1989. After decades of research and development work, we have come so far that we're now perfecting problems like object detection and optimizing autonomous vehicles.

Not only that, with the help of the internet, you can extrapolate all of these reasons or benefits onto multiple business models. It can be an innovative state-of-the-art product, an efficient service model, a content creator, or a dream job where you are solving problems that matter to you.

Goals for Reading a Research Paper — What Should You Read About?

The first thing to do is to figure out your motivation for reading the paper. There are two main scenarios that might lead you to read a paper:

Scenario 1 — You have a well-defined agenda/goal and you are deeply invested in a particular field. For example, you’re an NLP practitioner and you want to learn how GPT-4 has given us a breakthrough in NLP. This is always a nice scenario to be in as it offers clarity.
Scenario 2 — You want to keep abreast of the developments in a host of areas, say how a new deep learning architecture has helped us solve a 50-year old biological problem of understanding protein structures. This is often the case for beginners or for people who consume their daily dose of news from research papers (yes, they exist!).

If you’re an inquisitive beginner with no starting point in mind, start with scenario 2. Shortlist a few topics you want to read about until you find an area that you find intriguing. This will eventually lead you to scenario 1.

ML Reproducibility Challenge

In addition to these generic goals, if you need an end goal for your habit-building exercise of reading research papers, you should check out the ML reproducibility challenge.

[https://openreview.net/group?id=ML_Reproducibility_Challenge/2020](https://openreview.net/group?id=ML_Reproducibility_Challenge/2020" rel="nofollow noopener)

You’ll find top-class papers from world-class conferences that are worth diving deep into and reproducing the results.

They conduct this challenge twice a year and they have one coming up in Spring 2021. You should study the past three versions of the challenge, and I’ll write a detailed post on what to expect, how to prepare, and so on.

Now you must be wondering – how can you find the right paper to read?

How to Find the Right Paper to Read

In order to get some ideas around this, I reached out to my friend, Anurag Ghosh who is a researcher at Microsoft. Anurag has been working at the crossover of computer vision, machine learning, and systems engineering.

https://anuragxel.github.io/

Here are a few of his tips for getting started:

Always pick an area you're interested in.
Read a few good books or detailed blog posts on that topic and start diving deep by reading the papers referenced in those resources.
Look for seminal papers around that topic. These are papers that report a major breakthrough in the field and offer a new method perspective with a huge potential for subsequent research in that field. Check out papers from the morning paper or CVF - test of time award/Helmholtz prize (if you're interested in computer vision).
Check out books like Computer Vision: Algorithms and Applications by Richard Szeliski and look for the papers referenced there.
Have and build a sense of community. Find people who share similar interests, and join groups/subreddits/discord channels where such activities are promoted.

In addition to these invaluable tips, there are a number of web applications that I’ve shortlisted that help me narrow my search for the right papers to read:

r/MachineLearning — there are many researchers, practitioners, and engineers who share their work along with the papers they've found useful in achieving those results.

[https://www.reddit.com/r/MachineLearning/](https://www.reddit.com/r/MachineLearning/" rel="nofollow noopener)

Arxiv Sanity Preserver — built by Andrej Karpathy to accelerate research. It is a repository of 142,846 papers from computer science, machine learning, systems, AI, Stats, CV, and so on. It also offers a bunch of filters, powerful search functionality, and a discussion forum to make for a super useful research platform.

Google Research — the research teams at Google are working on problems that have an impact on our everyday lives. They share their publications for individuals and teams to learn from, contribute to, and expedite research. They also have a Google AI blog that you can check out.

How to Read a Research Paper

After you have stocked your to-read list, then comes the process of reading these papers. Remember that NOT every paper is useful to read and we need a mechanism that can help us quickly screen papers that are worth reading.

To tackle this challenge, you can use this Three-Pass Approach by S. Keshav. This approach proposes that you read the paper in three passes instead of starting from the beginning and diving in deep until the end.

The three pass approach

The first pass — is a quick scan to capture a high-level view of the paper. Read the title, abstract, and introduction carefully followed by the headings of the sections and subsections and lastly the conclusion. It should take you no more than 5–10 mins to figure out if you want to move to the second pass.
The second pass — is a more focused read without checking for the technical proofs. You take down all the crucial notes, underline the key points in the margins. Carefully study the figures, diagrams, and illustrations. Review the graphs, mark relevant unread references for further reading. This helps you understand the background of the paper.
The third pass — reaching this pass denotes that you’ve found a paper that you want to deeply understand or review. The key to the third pass is to reproduce the results of the paper. Check it for all the assumptions and jot down all the variations in your re-implementation and the original results. Make a note of all the ideas for future analysis. It should take 5–6 hours for beginners and 1–2 hours for experienced readers.

Tools and Software to Keep Track of Your Pipeline of Papers

If you’re sincere about reading research papers, your list of papers will soon grow into an overwhelming stack that is hard to keep track of. Fortunately, we have software that can help us set up a mechanism to manage our research.

Here are a bunch of them that you can use:

Mendeley [not free] — you can add papers directly to your library from your browser, import documents, generate references and citations, collaborate with fellow researchers, and access your library from anywhere. This is mostly used by experienced researchers.

https://www.mendeley.com/?interaction_required=true

Zotero [free & open source] — Along the same lines as Mendeley but free of cost. You can make use of all the features but with limited storage space.

https://www.zotero.org/

Notion — this is great if you are just starting out and want to use something lightweight with the option to organize your papers, jot down notes, and manage everything in one workspace. It might not stand anywhere in comparison with the above tools but I personally feel comfortable using Notion and I have created this board to keep track of my progress for now that you can duplicate:

⚠️ Symptoms of Reading a Research Paper

Reading a research paper can turn out to be frustrating, challenging, and time-consuming especially when you’re a beginner. You might face the following common symptoms:

You might start feeling dumb for not understanding a thing a paper says.
Finding yourself pushing too hard to understand the math behind those proofs.
Beating your head against the wall to wrap it around the number of acronyms used in the paper. Just kidding, you’ll have to look up those acronyms every now and then.
Being stuck on one paragraph for more than an hour.

Here’s a complete list of emotions that you might undergo as explained by Adam Ruben in this article.

Key Takeaways

We should be all set to dive right in. Here’s a quick summary of what we have covered here:

A research paper is an in-depth study that offers an detailed explanation of a topic or problem along with the research process, proofs, explained results, and ideas for future work.
Read research papers to develop a deep understanding of a topic/problem. Then you can either review papers as part of being a researcher, explore the domain and the kind of problems to build a solution or startup around it, or you can simply read them to keep abreast of the developments in your domain of interest.
If you’re a beginner, start with exploration to soon find your path to goal-oriented research.
In order to find good papers to read, you can use websites like arxiv-sanity, google research, and subreddits like r/MachineLearning.
Reading approach — Use the 3-pass method to find a paper.
Keep track of your research, notes, developments by using tools like Zotero/Notion.
This can get overwhelming in no time. Make sure you start off easy and increment your load progressively.

Remember: Art is not a single method or step done over a weekend but a process of accomplishing remarkable results over time.

You can also watch the video on this topic on my YouTube channel:

Feel free to respond to this blog or comment on the video if you have some tips, questions, or thoughts!

If this tutorial was helpful, you should check out my data science and machine learning courses on Wiplane Academy. They are comprehensive yet compact and helps you build a solid foundation of work to showcase.

Inductive VS Deductive Reasoning – The Meaning of Induction and Deduction, with Argument Examples

Abigail Rennemeyer — Sun, 29 Nov 2020 19:05:00 +0000

If you're conducting research on a topic, you'll use various strategies and methods to gather information and come to a conclusion.

Two of those methods are inductive and deductive reasoning.

So what's the difference between inductive and deductive reasoning, when should you use each method, and is one better than the other?

We'll answer those questions and give you some examples of both types of reasoning in this article.

What is Inductive Reasoning?

The method behind inductive reasoning

When you're using inductive reasoning to conduct research, you're basing your conclusions off your observations. You gather information - from talking to people, reading old newspapers, observing people, animals, or objects in their natural habitat, and so on.

Inductive reasoning helps you take these observations and form them into a theory. So you're starting with some more specific information (what you've seen/heard) and you're using it to form a more general theory about the way things are.

What does the inductive reasoning process look like?

You can think of this process as a reverse funnel – starting with more specifics and getting broader as you reach your conclusions (theory).

Some people like to think of it as a "bottom up" approach (meaning you're starting at the bottom with the info and are going up to the top where the theory forms).

Here's an example of an inductive argument:

Observation (premise): My Welsh Corgis were incredibly stubborn and independent (specific observation of behavior).
Observation (premise): My neighbor's Corgis are the same way (another specific observation of behavior).
Theory: All Welsh Corgis are incredibly stubborn and independent (general statement about the behavior of Corgis).

As you can see, I'm basing my theory on my observations of the behavior of a number of Corgis. Since I only have a small amount of data, my conclusion or theory will be quite weak.

If I was able to observe the behavior of 1000 Corgis (omg that would be amazing), my conclusion would be stronger – but still not certain. Because what if 10 of them were extremely well-behaved and obedient? Or what if the 1001st Corgi was?

So, as you can see, I can make a general statement about Corgis being stubborn, but I can't say that ALL of them are.

What can you conclude with inductive reasoning?

As I just discussed, one of the main things to know about inductive reasoning is that any conclusions you make from inductive research will not be 100% certain or confirmed.

Let's talk about the language we use to describe inductive arguments and conclusions. You can have a strong argument (if your premise(s) are true, meaning your conclusion is probably true). And that argument becomes cogent if the conclusion ends up being true.

Still, even if the premises of your argument are true, and that means that your conclusion is probably true, or likely true, or true much of the time – it's not certain.

And – weirdly enough – your conclusion can still be false even if all your premises are true (my Corgis were stubborn, my neighbor's corgis were stubborn, perhaps a friend's Corgis and the Queen of England's Corgis were stubborn...but that doesn't guarantee that all Corgis are stubborn).

How to make your inductive arguments stronger

If you want to make sure your inductive arguments are as strong as possible, there are a couple things you can do.

First of all, make sure you have a large data set to work with. The larger your sample size, the stronger (and more certain/conclusive) your results will be. Again, thousands of Corgis are better than four (I mean, always, amiright?).

Second, make sure you're taking a random and representative sample of the population you're studying. So, for example, don't just study Corgi puppies (cute as they may be). Or show Corgis (theoretically they're better trained). You'd want to make sure you looked at Corgis from all walks of life and of all ages.

If you want to dig deeper into inductive reasoning, look into the three different types – generalization, analogy, and causal inference. You can also look into the two main methods of inductive reasoning, enumerative and eliminative. But those things are a bit out of the scope of this beginner's guide. :)

What is Deductive Reasoning?

The method behind deductive reasoning

In order to use deductive reasoning, you have to have a theory to begin with. So inductive reasoning usually comes before deductive in your research process.

Once you have a theory, you'll want to test it to see if it's valid and your conclusions are sound. You do this by performing experiments and testing your theory, narrowing down your ideas as the results come in. You perform these tests until only valid conclusions remain.

What does the deductive reasoning process look like?

You can think of this as a proper funnel – you start with the broad open top end of the funnel and get more specific and narrower as you conduct your deductive research.

Some people like to think of this as a "top down" approach (meaning you're starting at the top with your theory, and are working your way down to the bottom/specifics). I think it helps to think of this as "reductive" reasoning – you're reducing your theories and hypotheses down into certain conclusions.

Here's an example of a deductive argument:

We'll use a classic example of deductive reasoning here – because I used to study Greek Archaeology, history, and language:

Theory: All men are mortal
Premise: Socrates is a man
Conclusion: Therefore, Socrates is mortal

As you can see here, we start off with a general theory – that all men are mortal. (This is assuming you don't believe in elves, fairies, and other beings...)

Then we make an observation (develop a premise) about a particular example of our data set (Socrates). That is, we say that he is a man, which we can establish as a fact.

Finally, because Socrates is a man, and based on our theory, we conclude that Socrates is therefore mortal (since all men are mortal, and he's a man).

You'll notice that deductive reasoning relies less on information that could be biased or uncertain. It uses facts to prove the theory you're trying to prove. If any of your facts lead to false premises, then the conclusion is invalid. And you start the process over.

What can you conclude with deductive reasoning?

Deductive reasoning gives you a certain and conclusive answer to your original question or theory. A deductive argument is only valid if the premises are true. And the arguments are sound when the conclusion, following those valid arguments, is true.

To me, this sounds a bit more like the scientific method. You have a theory, test that theory, and then confirm it with conclusive/valid results.

To boil it all down, in deductive reasoning:

"If all premises are true, the terms are clear, and the rules of deductive logic are followed, then the conclusion reached is necessarily true." (Source)

So Does Sherlock Holmes Use Inductive or Deductive Reasoning?

Sherlock Holmes is famous for using his deductive reasoning to solve crimes. But really, he mostly uses inductive reasoning. Now that we've gone through what inductive and deductive reasoning are, we can see why this is the case.

Let's say Sherlock Holmes is called in to work a case where a woman was found dead in her bed, under the covers, and appeared to be sleeping peacefully. There are no footprints in the carpet, no obvious forced entry, and no immediately apparent signs of struggle, injury, and so on.

Sherlock observes all this as he looks in, and then enters the room. He walks around the crime scene making observations and taking notes. He might talk to anyone who lives with her, her neighbors, or others who might have information that could help him out.

Then, once he has all the info he needs, he'll come to a conclusion about how the woman died.

That pretty clearly sounds like an inductive reasoning process to me.

Now you might say - what if Sherlock found the "smoking gun" so to speak? Perhaps this makes his arguments and process seem more deductive.

But still, remember how he gets to his conclusions: starting with observations and evidence, processing that evidence to come up with a hypothesis, and then forming a theory (however strong/true-seeming) about what happened.

How to Use Inductive and Deductive Reasoning Together

As you might be able to tell, researchers rarely just use one of these methods in isolation. So it's not that deductive reasoning is better than inductive reasoning, or vice versa – they work best when used in tandem.

Often times, research will begin inductively. The researcher will make their observations, take notes, and come up with a theory that they want to test.

Then, they'll come up with ways to definitively test that theory. They'll perform their tests, sort through the results, and deductively come to a sure conclusion.

So if you ever hear someone say "I deduce that x happened", they better make sure they're working from facts and not just observations. :)

TL;DR: Inductive vs Deductive Reasoning – What are the Main Differences?

Inductive reasoning:

Based on observations, conversations, stuff you've read
Starts with information/evidence and works towards a broader theory
Arguments can be strong and cogent, but never valid or sound (that is, certain)
Premises can all be true, but conclusion doesn't have to be true

Deductive reasoning:

Based on testing a theory, narrowing down the results, and ending with a conclusion
Starts with a broader theory and works towards certain conclusion
Arguments can be valid/invalid or sound/unsound, because they're based on facts
If premises are true, conclusion has to be true

And here's a cool and helpful chart if you're a visual learner:

Image source

That's about it!

Now, if you need to conduct some research, you should have a better idea of where to start – and where to go from there.

Just remember that induction is all about observing, hypothesizing, and forming a theory. Deducing is all about taking that (or any) theory, boiling it down, and testing until a certain conclusion(s) is all that remains.

Happy reasoning!

Qualitative VS Quantitative Definition – Research Methods and Data

Abigail Rennemeyer — Tue, 03 Nov 2020 04:04:18 +0000

When you’re conducting research, your data will fall into two categories: qualitative or quantitative. So what’s the difference between these two data types?

Well, here’s a quick and easy way to remember at least the basic difference: quantitative data deals with quantities of things – numbers and measurable information, like how many people visit a website each day. That’s all about quantity (sounds like quantitative, right?).

On the other hand, qualitative data gives you more insight into what people think, feel, and believe – the quality of a thing, person, or situation. Alright that one’s a bit more of a stretch, but it works.

Now let’s get more into the details of qualitative and quantitative research so you know how to conduct each.

What is Qualitative research?

Qualitative research focuses on the human perspective, and usually answers the question “why?” If you want to learn how people perceive their environment, why they hold certain beliefs, or how they understand their problems, you’ll conduct qualitative research.

It’s also all about context. When you’re researching a group, you want to study them in their natural environment. This gives you insights into their behavior, beliefs, opinions, and so on.

How do you conduct qualitative research?

You can conduct qualitative research in a few different ways. Doing interviews, setting up focus groups, giving people open-ended questionnaires, studying photo collections, and observing people in their daily routines are all forms of qualitative data collection.

When you engage with people in these ways, you’re giving the opportunity to give more in-depth, elaborate responses. They’re not just responding “yes” or “no” – they’re telling you what they think.

You can also make observations from photographs or from watching people – things like the way people are looking at each other lovingly, or how two old people might hold hands while they watch TV.

From these observations, you can theorize that those people love each other, are close to each other, know each other well and are comfortable around each other, and so on. Things that are hard to quantify with numbers or measure with figures.

What is Quantitative research?

Quantitative research, on the other hand, involves collecting facts and figures and often results in numerical, structured data. Think data you can put in a spreadsheet and analyze.

Instead of talking to people and getting their opinions, you’re asking them yes or no questions. Instead of asking someone why they do something, you’re finding out what they do, or how many people do that thing, or how often – and so on.

Real quick - what is structured data?

Let's say you're looking at a recipe on your favorite online cooking blog. The structured data are things like the ingredients, the oven temperature, how many calories a serving has, and how long you cook the food. These are all quantifiable (and measurable with numbers/facts) things.

Unstructured data, on the other hand, would include the food blogger's little story about how they discovered or created the recipe, what people have said about how delicious it is, and how much they love the texture of those soft, gooey cookies. You can't measure these data – they're opinion and experience-based.

How do you conduct quantitative research?

You can conduct quantitative research by looking at statistical data (how many people did x), giving people multiple choice or true/false tests, asking them yes/no questions on a survey, and so on.

All in all, you’re trying to answer the question “what” or “how” – what something is, what’s the number of people who order from Amazon every day, how many cars are in that parking lot.

Because of the nature of the data and collection methods, context isn’t a factor in this type of research.

With quantitative research, you’re interested in gathering data that support and prove or disprove a hypothesis or theory you already have.

So instead of observing and talking to people and then forming a theory about what’s going on, you collect your data, and then make conclusions about the validity of your hypothesis based on that data.

Is Qualitative or Quantitative research better?

Alright, so you have these two methods of research – which is better?

Well, most people would argue that they’re better when used together. They’re complementary. Each has its pros and cons (which we’ll discuss), but each method definitely brings important information to the table.

Before we discuss just how they can work together, let’s look at the good and the bad of each.

Pros and Cons of Qualitative research

Let's start with the good. Qualitative research lets you dig deeper into a problem, situation, or context and see why things are happening. You get personal insights from your subjects that can't necessarily come from numbers and figures.

You also have the benefit of context, which can shed light on why a person said certain things or was feeling a certain way (for example if they live in a war zone or in a small village in the middle of nowhere or in the largest city in the world).

On the other hand, qualitative research is more time-consuming and therefore expensive. It takes a lot more time to interview people or set up focus groups than it does to send someone a simple yes/no survey.

It can also be harder to get people to participate in qualitative research. They might not have the time or energy (or desire) to share extensively.

Finally, qualitative research is never really definitive. People are always changing, as are their perceptions of the world around them. So while qualitative data can help inform your hypothesis and fill in gaps in your research, it should usually be supported by quantitative data.

Pros and Cons of Quantitative research

Quantitative research produces hard facts, numbers, and other measurable things. Which can be very useful when you're trying to prove a theory or understand what you're dealing with.

It's also independent of changeable things, like researcher bias or people's current opinions or moods. So quantitative research is repeatable and can be tested and re-tested again and again.

And, practically speaking, quantitative data analysis can be performed much more quickly than qualitative research. You can simply send someone a survey, collect the response data, and dump that data into a spreadsheet or database. From there, running various queries and analyses is easy (assuming you know what you want to ask).

Still, quantitative research is limiting in certain ways. People can't explain their answers to a multiple choice test or yes/no survey (again, lack of context). This means you can't take human factors into account.

So while you have the facts and numbers, you have to decide how to interpret them and use them in your research. (This can be both good and bad.)

How to use Qualitative and Quantitative research together

Sometimes it’s best to start with qualitative research – gather information, talk to people, try to understand their problems/perceptions/opinions, then form a hypothesis.

Then, once you have your hypothesis, use quantitative methods to confirm (or disprove) it with data analysis. This will show you whether the issue/problem/situation exists in general, or was just part of someone’s perception.

But qualitative research/insights can also help round out your structured data/conclusions – if you’ve learned that x people use your site every day, quotes from people about why they use it (as opposed to another company) can teach you more about what’s working (or not) and why.

Examples of Qualitative and Quantitative research

First example

Say you want to learn more about people who visit Paris on vacation. You could look at flight data, museum admission numbers, tourist info to figure out how many people visit Paris each year. But that won’t tell you why they’re visiting.

To learn why, you have to ask people why they wanted to visit Paris, what was their favorite part of the city, what was their experience like as a tourist in Paris, and so on. This will give you insights into what motivates people to travel there in the first place.

Another example

Let’s say you run an e-commerce site that helps people resell their gently used clothing.

You can gather information about how many people sell clothes on your site, how many items the average person has sold, how many people visit the site to buy those clothes, and so on. All that’s right there in the analytics.

But if you want to know why people choose to use your site – either to sell or buy clothes – you’d want to start by conducting an open-ended questionnaire or ask for feedback on a survey.

Also, if you want to know what they like about your site, and how that influences their decision to use it, you could ask them to describe their experience using the site, and so on.

Ultimately, you’ll want to use both qualitative and quantitative research to get the whole picture. And you won’t just use one, and then just use the other. You can go back and forth between the two methods as your research evolves and you gather more information.

This will help you get a more complete picture, form a stronger and deeper hypothesis, and establish both facts about and insights into the situation.

How to Ask Effective Questions: A Practical Guide for Developers

Bolaji Ayodeji — Tue, 12 May 2020 08:20:49 +0000

Learning is a journey that never ends. At every point in your career, you will keep learning, re-learning, and un-learning.

The ability to find answers to problems is key, and these answers can be gotten by either debugging, googling or asking questions. These are skills everyone should learn.

While asking questions is great, doing it the wrong or less effective way might lead to many hours of unproductive work. In this article show you how to ask effective questions, become a better googler, and hone your problem-solving skills.

The Read Search Ask Method

freeCodeCamp came up with this amazing method for their students called the Read Search Ask method where you can solve your problems efficiently while respecting other people’s time, too.

It is expected that you follow these steps in order before you ask a question.

Read the error or documentation
Search Google or Stackoverflow
Ask for help

Read your errors

The first step to finding a solution(s) to your problem is to understand the problem itself. Many developers ask questions without understanding the problem. Remember you want to ask questions efficiently and also respect the time of the individual you will be asking.

Most often, the answers to your bugs are in the errors itself. It's not considerate to ask questions that you could have solved yourself if you had read the error itself.

One great way to debug your errors is to log the error on your console or terminal and read the errors. Five things can happen at this stage:

You identify the bug
You determine the location of the bug (possible a file or section of your code)
You understand the cause(s) of the bug
You outline possible fixes of the bug
You implement the possible fixes

Most often, just this step alone can help you solve your problem. The more you practice this, the better you become at debugging and fixing issues. Some bugs might be more complex than your capacity, and then you need to seek external help or resources.

Read the Documentation

Let's assume you couldn't fix your issues from the first step (but you were able to identify the bug at least). Then you need to read some documentation.

This might not apply to all bugs, but make sure to run your issue through this step before moving to the next. After identifying your bug, it's time to determine the cause and outline possible fixes.

Documentation is a set of documents provided on paper, or online, or on digital or analogue media, such as audio tape or CDs. Examples are user guides, white papers, on-line help, quick-reference guides. Paper or hard-copy documentation has become less common. ~ Wikipedia

Documentation serves as the official guide of a particular programming language, framework, library or technology. The best place to get the first-hand resource on a specific issue is to check the official help guide of the technology.

You might have misconfigured something, installed some wrong package or forgot to import something. But when you read the docs of the tech responsible for the bug it can help you find a fix quickly.

Recently I was working on a Gatsby project, and I ran into some console errors and blank white page in production. But this didn't occur in development.

TypeError: t.test is not a function

So vague right? After identifying the bug location, reading the docs and trying out several fixes, I discovered something: in the Gatsby Google analytics plugin configuration, I left the array meant to avoid sending pageview hits from custom paths empty.

plugins: [
    {
      resolve: `gatsby-plugin-google-analytics`,
      options: {
        trackingId: 'UA-XXXXXXXX-1',
        head: false,
        anonymize: true,
        respectDNT: true,
        exclude: [''],
        pageTransitionDelay: 0,
      }
]

After 3 hours of reading the error, trying to identify the bug and reading Gatsby docs, I figured out that the exclude[] option had no values in it. Removing this fixed my issue since I didn't need to exclude any page after all.

This shows why you should spend some time debugging. It might take some time, and it will be frustrating. The bug might look stupid, the fix might be simple, but it's worth it. The more time you spend debugging, the better you become and the faster you will fix even more complex bugs next time.

Search

Now if you have read your errors and several docs, but your bug is becoming more complex, it's time to search the web "effectively".

One of the best places to search for solutions to your issues is Google since it already crawls content from numerous webpages, dev communities, and docs like StackOverflow, Hashnode, freeCodeCamp News, MDN, CSS tricks, Hackernoon, and many others.

Here are some helpful Google search tips you can try to get answers more efficiently:

Add a specific domain to channel your search to a particular website and save some time. You can append the site: sitename to your search, and it'll return only results from that website.

Search for specific title and text in specific sites. This will allow your results to include resources related to the title or text keyword you specified.

intitle:axe

intitle:axe intext:web ui testing

intitle:axe intext:web ui testing site:bolajiayodeji.com

Add specific keywords to your search and pull out individual details like file paths. If your bug is related to Python, add it to your search query. If it is related to objects, add that too. This will streamline the results to a wider range of possible resources related to your bug, since there are objects in just about every programming language.
Google automatically omits most non-alphanumeric characters from your queries. Symbols like `!@#$%^&*(){}[]<> are not included in your search. So you can focus on more specific keywords, as discussed above.
Copy and paste your errors (removing specific information); there is a 99.9% chance that someone else has encountered the same error and has maybe opened some issue or written an article about it. Never underestimate the power of StackOverflow and GitHub issues.
Search within the past few years since programming changes pretty quickly. This will ensure you don't hit redundant resources.
Reiterate, try different keywords and more general keywords multiple times, and open multiple tabs until you find an answer. This might be awkward at first, but as you consistently do this, you'll get better, and with time it will take you less time and fewer tabs to find an answer.

Ask

Now comes the sought-after diamond. I would advise, before you ask ANY questions, that you make sure you have completed steps 1 and 2.

This might be hard, but the earlier you get used to this, the better you grow. I've had people ask me questions, and when I copy and paste their question into Google, the first result answers the question 100%. This doesn't feel great.

Most often, I'll have to send the link to them. Meanwhile, they could have done this themselves and saved me some time while increasing their problem-solving skills.

Even if your next-door neighbour works at Google, if your dad is a Software Engineer, if your close friend has two more years experience beyond you, try not to ask them questions until you have invested time into finding solutions yourself. This will help you:

Build more self-confidence, as you won't need to rely so much on others. (What if that close friend is unavailable to respond to you someday, what will you do?!)
Save the time of whomever you are planning to ask questions. There are tons of developers around the world, and you might not be the only one asking that same person questions. By completing steps 1 and 2 first, you are also helping others with less experience or more complex problems get answers quickly.
Build your problem-solving, debugging, research and googling skills. These are essential skills that will help you become a better developer. You only master a skill when you consistently practice it, so try to practice this regularly and watch yourself grow and master the art.

But let's say that you have completed both steps, and you still need external help from a mentor, senior developer, team lead, the local community, group, or friend. How then do you ask questions effectively?

Complete step 1 and make sure you understand the problem already
Complete step 2 so you might at least have gotten some insights into possible causes and solutions of the problem
Now do some research: you are either asking the question in dev communities like Stackoverflow or Hashnode or requesting help from an individual you feel has more experience than you.
Developers answering questions on dev communities are sacrificing their time to help you. Help them also by asking "clear and concise questions". You should choose your words deliberately and precisely, constructing your sentences carefully to avoid confusion. From step 1 and step 2, properly draft your questions based on:
- The "problem" you have encountered
- Your thoughts on what you think might be wrong "based on your research."
- Proof of research efforts (Devs will be happier to help you once they see you have made some effort already). Add links to resources you have checked and solutions you have tried already.
- Be conversant with tools like CodePen, CodeSandBox, GitHub Gist, Repl.it, and GitHub so you can easily provide a link to a snippet of your code, a working project or mimimal reproduction without having to type so much text. Please try as much as possible not to paste code that is more than one line in chat.
- Take clean and clear screenshots where necessary. Most often, you can take a snapshot of your console errors for smaller issues. Try not to strain your helper's eyes. Take specific and clear screenshots related to your error.
Be sure you're asking the right person or community. Don't ask a python related question to someone who knows just Java. Your research from the previous step will help you here. It's also great to bookmark specific communities, individuals or groups where you will regularly ask questions based on how helpful their responses have been to you.
Now don't be afraid to ask. I've had my share of toxic responses from people who respond with comments like "this is simple", "come on, google it", "you should know this", and you might get that too :(. The internet is a combination of good and toxic individuals who might be or not welcoming. Be confident in yourself, you have done your part by completing steps 1 and 2, so ask the question freely.

Conclusion

You don't need to "know it all", you just need to "know it and know how and where to find more knowledge."

Dear 10x, Senior, "It's Simple", Engineer: please note that you're discouraging developers who are asking these questions. I had to battle so much with my self-confidence some years ago, and it took me a while to get over it.

There is so much for learners to battle with already (learning, imposter syndrome, and personal issues), so please don't add to it. Be welcoming, make them feel like they belong here, and support them.

Your response can intentionally or unintentionally influence someone's decision to ask questions freely next time. If you can't or won't be available, you can either not respond at all or respond in a friendly way with feedback.

Refer people to your friends if you don't have the time or capabilities to attend to them, give constructive criticism, and try not to leave a question without feedback or impact.

Beginner developers need support. Learning to code is hard, especially if you're doing it alone. If you're learning in a traditional classroom, be sure to network with the other students. Do homework together. Discuss the problems. If you're learned online, it's very easy to get discouraged. Find chat spaces with others who are on a similar journey and use that space to encourage one another. Twitter also makes the experts accessible to you (reason #49948 why I love Twitter). Find the experts in whatever it is you're learning and consume their content. Ask them questions if you need to. The key is not to do it alone. ~ Angie Jones

The industry is growing daily. We are all trying to learn and get better at different levels. A lot of developer communities are supportive and make their resources available for consumption. Make sure you are a part of this movement, consume those available resources, contribute to these resources, and be a helper in any way you can.

Thank you to all the amazing developers across the globe, creating and sharing contents/ projects. Your contributions are the pillars holding up the tech ecosystem today. ??

Quick note, I interview awesome women in tech on the She Inspires at Hashnode where I understand the current health of the tech industry, get insights from their personal and career growth, and inspire other women to become better. Read all past interviews here, trust me it's worth your time. :)

Don't spend your entire life waiting for the best time to make that move – write that article, build that side-project, apply for that job or ask that question today. The best time is always now!

There is a real connection between technical interview performance and salary. Here’s the data.

freeCodeCamp — Tue, 12 Mar 2019 15:43:04 +0000

By Cat Hicks

At the end of the day, money is a huge driver for the decisions we make about what jobs to go after. In the past, we’ve written about how to negotiate your salary, and there are a lot of labor statistics and reports out there looking at salaries in the tech industry as a whole. But as with many things in eng hiring, there’s very little concrete data on whether technical interview performance plays a role in compensation offers.

So we set out to gather the data and asked our users who had gone on to successfully get jobs after using our platform to share their salary info. With our unique dataset of real coding interviews, we could ask questions like:

Does interview performance matter when it comes to compensation packages?
Do engineers who prioritize other parts of a role over compensation (e.g. values alignment) end up with lower salaries?
What else seems to matter in getting a higher salary?

To be clear, this is an exploration of past average interview performance and its connection with current salary, versus looking at how someone did in an interview and then what salary they got when they took that specific job. In other words, we haven’t paired job interviews with the salary for that same job. We believe that looking at these more general measures is more informative than trying to match single interviews and job offers, given how volatile individual interview performance can be. But our interviewing platform allowed us to look at performance across multiple interviews for respondents, which gave us more stability and more data.

The setup

On the interviewing.io platform, people can practice technical interviews online and anonymously, with real engineers on the other side.

When an interviewer and an interviewee match on our platform, they meet in a collaborative coding environment with voice, text chat, and a whiteboard and jump right into a technical question. Check out our recordings page to see this process in action.

Interview questions on the platform tend to fall into the category of what you’d encounter at a phone screen for a back-end software engineering role, and interviewers typically come from top companies like Google, Facebook, Dropbox, Airbnb, and more.

After every interview, interviewers rate interviewees on a few different dimensions: technical skills, communication skills, and problem solving skills. These each get rated on a scale of 1 to 4, where 1 is “poor” and 4 is “amazing!”. On our platform, a score of 3 or above has generally meant that the person was good enough to move forward. You can see what our feedback form looks like below:

With this in mind, we surveyed interviewing.io users about their current roles, including salary, bonuses, and how satisfied they felt in their job. We then tied their comp back to how they did in interviews on our platform.

We ended up with responses from 494 engineers, and because compensation packages are so complex and vary from company to company, we analyzed the data in several different ways, looking at annual salary numbers, bonuses, and equity. Then we tied compensation data to performance in technical interviews to see whether it matters, and if so, how much.

The results

We looked at the relationships between interview performance (technical skills, communication ability, and problem solving ability) and the following: base salary, bonuses, and equity. In all cases, we corrected for location (being in the Bay Area means a higher salary) and experience (senior engineers make senior salaries), and where we could, we corrected for company size (bigger companies can generally pay bigger salaries).

The mean yearly salary for all survey participants was around $130k, and 57% of them reported a yearly bonus. For that group, the average yearly bonus was $20k. For people who reported a dollar amount for equity, the average was $54k. Below is the distribution of experience level/seniority of survey respondents.

Here’s what we found.

Better technical skills correlate with higher compensation

As probably comes as no surprise, people who score higher on technical skills during interviews do make more money. First, let’s look at base salary. [2]

Bonuses, too, correlate with technical skills, with an additional point in performance potentially worth about 10k. [3]

The relationship between compensation and other interviewing skills

We also looked at the two other ratings that interviewers give after interviews: communication and problem solving. Better communication scores had a small but statistically significant correlation with salaries (r = .15, p < .01), but we found no significant relationships for problem solving scores.

We also didn’t see a relationship between either communication ability or problem solving ability when it came to bonuses.

The non-relationships didn’t surprise us too much, to be honest, because with a relatively small sample size it’s notoriously difficult to get subcomponents of ratings to show a relationship to something distal and complicated like salaries. It’s very possible these relationships do exist, and with many determiners besides actual interview performance, like seniority and market salary norms, we’d like to repeat this survey at a bigger scale to inform this question.

What else?

We asked engineers whether they felt satisfied with their role, and found that engineers who felt satisfied earned an average $14k more than engineers who felt dissatisfied. [4]

We also looked at people’s perceptions of their own performance. In a previous post, we explored how people rated their own technical performance after an interview compared to how the interviewer rated them, and found that even experienced engineers aren’t great at guessing how they did.

For this project, we were curious about whether overconfident engineers might net higher salaries (perhaps they negotiate harder!). So we also looked at people who rated their performance higher than their actual interview score — but found no difference in their compensation packages.

Another thing we were curious about was whether people who valued money over other factors while making a job decision would have higher salaries. So we asked people to rank the most important variables in their job decisions. 32% of respondents said that a compensation package was the most important part of their decision; the next highest response was “matches my interests and values.”

But these questions didn’t have any predictive value for the actual salary amount: people who said money matters the most didn’t have significantly different salaries from people who said money matters the least.

It’s possible that with salaries being impacted by so many outside factors, like location and role type, candidates don’t truly have a lot of negotiating power over that salary number.

We looked at equity as well, and the average reported equity package size was 54k. We did not find any significant association between interview performance and reported equity packages. That said, enormous amounts of research have documented various salary gaps based on gender, race, and other important sociocultural and demographic factors, and we hope to repeat this analysis when we have more data.

What do these findings mean for you?

Interview performance doesn’t just get you in the door or not: it can have a demonstrable connection to your eventual compensation. For instance, doing just a point better in your technical interview could be worth 10k or more, and with bonus, it could add 20k to your annual comp.

Given how much technical interview performance matters, we’d be remiss if we didn’t suggest signing up for free, anonymous mock interviews on our platform. So, please go do that.

And, if you’re curious about what our salary survey looked like or want to participate and contribute to v2 of this post, please do so too!

Want to become awesome at technical interviews and land your next job in the process? Join interviewing.io!

Cat Hicks, the author of this guest post, is a researcher and data scientist with a focus on learning. She consults in research, has published empirical research on learning environments, and has led research on the cognitive work of engineering teams at Google and Travr.se. She holds a PhD in Psychology from UC San Diego.

Robustness in Complex Systems: an academic article summary

freeCodeCamp — Thu, 22 Mar 2018 15:43:29 +0000

By Shubheksha Jalan

Today, we’re going to look at the paper titled “Robustness in Complex Systems” published in 2001 by Steven D. Gribble. All pull quotes and figures are from the paper.

This paper argues that a common design paradigm for systems is fundamentally flawed, resulting in unstable, unpredictable behavior as the complexity of the system grows.

The “common design paradigm” refers to the practice of predicting the environment the system will operate in and its failure modes. The paper states that a system will deal with conditions that weren’t predicted as it becomes more complex, so it should be designed to cope with failure gracefully. The paper explores these ideas with the help of “distributed data structures (DDD), a scalable, cluster-based storage server.”

By their very nature, large systems operate through the complex interaction of many components. This interaction leads to a pervasive coupling of the elements of the system; this coupling may be strong (e.g., packets sent between adjacent routers in a network) or subtle (e.g., synchronization of routing advertisements across a wide area network).

A common characteristic that such large systems exhibit is something known as the Butterfly Effect. This refers to a small unexpected disturbance in the system resulting from the intricate interaction of various components that causes a widespread change.

A common goal for system design is robustness: the ability of a system to operate correctly in various conditions and fail gracefully in an unexpected situation. The paper argues against the common pattern of trying to predict a certain set of operation conditions for the system and architecting it to work well in only those conditions.

It is also effectively impossible to predict all of the perturbations that a system will experience as a result of changes in environmental conditions, such as hardware failures, load bursts, or the introduction of misbehaving software. Given this, we believe that any system that attempts to gain robustness solely through precognition is prone to fragility.

DDS: A Case Study

The hypothesis stated above is explored using a scalable, cluster-based storage system, Distributed Data Structures (DDD) — “a high-capacity, high-throughput virtual hash table that is partitioned and replicated across many individual storage nodes called bricks.”

This system was built using a predictive design philosophy as the one described above.

Based on extensive experience with such systems, we attempted to reason about the behavior of the software components, algorithms, protocols, and hardware elements of the system, as well as the workloads it would receive.

When the system operated within the scope of the assumptions made by the designers, it worked fine. They were able to scale it and improve performance. However, in the case when one or more of the assumptions about the operating conditions were violated, the system behaved in unexpected ways resulting in data loss or inconsistencies.

Next, we talk about several such anomalies.

Garbage Collection Thrashing and Bounded Synchrony

The system designers used timeouts to detect failure of components in the system. If a particular component didn’t respond within the specified time, it was considered dead. They assumed bounded synchrony in the system.

The DDS was implemented in Java, and therefore made use of garbage collection. The garbage collector in our JVM was a mark-and-sweep collector; as a result, as more active objects were resident in the JVM heap, the duration that the garbage collector would run in order to reclaim a fixed amount of memory would increase.

When the system was at saturation, even slight variations in load on the bricks would increase the time taken by the garbage collector in turn dropping the throughput of the brick. This is called GC thrashing. The affected bricks would lag behind their counterparts leading to a further degradation in performance of the system.

Hence, garbage collection violated the assumption of bounded synchrony when it was nearing or beyond the saturation point.

Another assumption made while designing the system was that the failures are independent. DDS used replication to make the system fault-tolerant. The probability of multiple replicas failing simultaneously was very small.

However, this assumption was violated when they encountered a race condition in their code that caused a memory leak without affecting correctness.

Whenever we launched our system, we would tend to launch all bricks at the same time. Given roughly balanced load across the system, all bricks therefore would run out of heap space at nearly the same time, several days after they were launched. We also speculated that our automatic failover mechanisms exacerbated this situation by increasing the load on a replica after a peer had failed, increase the rate at which the replica leaked memory.

Since all the replicas were subjected to a uniform load without taking performance degradation and other issues into consideration, this created a coupling between the replicas and…

…when combined with a slow memory leak, lead to the violation of our assumption of independent failures, which in turn caused our system to experience unavailability and partial data loss

Unchecked Dependencies and Fail-stop

Based on the assumption that if a component timed out, it has failed, the designers also assumed “fail-stop” failures, that is a component that has failed will not resume functioning after a while. The bricks in the system performed all long-latency work (disk I/O) in an asynchronous way.

However, they failed to notice that some parts of their code made use of blocking function calls. This caused the main event-handling thread to be randomly borrowed leading to bricks seizing inexplicably for a couple of minutes and resuming post.

While this error was due to our own failure to verify the behavior of code we were using, it serves to demonstrate that the low-level interaction between independently built components can have profound implications on the overall behavior of the system. A very subtle change in behavior resulted in the violation of our fail-stop assumption across the entire cluster, which eventually lead to the corruption of data in our system.

Towards Robust Systems

..small changes to a complex, coupled system can result in large, unexpected changes in behavior, possibly taking the system outside of its designers’ expected operating regime.

A few solutions which can help us make more robust systems:

Systematic Over-provisioning

When approaching the saturation point, systems tend to become fragile when trying to accommodate unexpected behavior. One way to combat this is to deliberately over-provision the system.

However, this has its own set of issues: it leads to the under-utilization of resources. It also requires predicting the expected operating environment and hence the saturation point of the system. This can’t be done in an accurate manner in most cases.

Use Admission Control

Another technique is to start rejecting load once the system starts approaching the saturation point. However, this requires predicting the saturation point — something that’s not always possible, especially with large systems which have a lot of contributing variables.

Rejecting requests also consumes some resources from the system. Services designed with admission control in mind usually have two operating modes: normal where the requests are processed and an extremely lightweight mode where they’re rejected.

Build Introspection into the system

an introspective system is one in which the ability to monitor the system is designed in from the beginning.

When a system can be monitored, and designers and operators can derive meaningful measurements about its operation, it’s much more robust than a black-box system. It’s easier to adapt such a system to change in its environment, as well as manage and maintain it.

Introduce adaptivity by closing the control loop

An example of a control loop is human designers and operators adapting the design in response to a change in its operating environment indicated through various measurements. However, the timeline for such a control loop isn’t very predictable. The authors argue that systems should be built with internal control loops.

These systems incorporate the results of introspection, and attempt to adapt control variables dynamically to keep the system operating in a stable or well-performing regime.

All such systems have the property that the component performing the adaptation is able to hypothesize somewhat precisely about the effects of the adaptation; without this ability, the system would be “operating in the dark”, and likely would become unpredictable. A new, interesting approach to hypothesizing about the effects of adaptation is to use statistical machine learning; given this, a system can experiment with changes in order to build up a model of their effects.

Plan for failure

Complex systems must expect failure and plan for it accordingly.

A couple of techniques to do this:

decoupling of components to contain failures locally
minimize damage by using robust abstractions such as transactions
minimize amount of time in failure state (using checkpointing to recover rapidly)

In this paper, the authors argue that designing systems by assuming the constraints and nature of its operation, failures, and behavior often leads to fragile and unpredictable systems. We need a radically different approach to build systems that are more robust in the face of failure.

This different design paradigm is one in which systems are given the best possible chance of stable behavior (through techniques such as over-provisioning, admission control, and introspection), as well as the ability to adapt to unexpected situations (by treating introspection as feedback to a closed control loop). Ultimately, systems must be designed to handle failures gracefully, as complexity seems to lead to an inevitable unpredictability.

Building on Quicksand: A Summary

freeCodeCamp — Mon, 04 Dec 2017 17:11:21 +0000

By Shubheksha Jalan

Let’s try to break down the paper “Building On Quicksand” published by Pat Helland and David Campbell in 2009. All pull quotes are from the paper.

The paper focuses on the design of large, fault-tolerant, replicated distributed systems. It also discusses its evolving based on changing requirements over time. It starts off by stating “Reliable systems have always been built out of unreliable components”.

As the granularity of the unreliable component grows (from a mirrored disk to a system to a data center), the latency to communicate with a backup becomes unpalatable. This leads to a more relaxed model for fault tolerance. The primary system will acknowledge the work request and its actions without waiting to ensure that the backup is notified of the work. This improves the responsiveness of the system because the user is not delayed behind a slow interaction with the backup.

Fault-tolerant systems can be made of many components. Their goal is keep functioning when one of those components fail. We don’t consider Byzantinefailures in this discussion. Instead, the fail fast model where either a component works correctly or it fails.

The paper goes on to compare two versions of the Tandem NonStop system. One that used synchronous checkpointing and one that used asynchronous checkpointing. Refer section 3 of the paper for all the details. I’d like to touch upon the difference between the two checkpointing strategies.

Synchronous checkpointing: in this case, with every write to the primary, state needed to be sent to the backup. Only after the backup acknowledged the write, did the primary send a response to the client who issued the write request. This ensured that when the primary fails, the backup can take over without losing any work.
Asynchronous checkpointing: in this strategy, the primary acknowledges and commits the write. This is done as soon as it processes it without waiting for a reply from the backup. This technique has improved latency but also poses other challenges addressed later.

Log Shipping

A classic database system has a process that reads the log and ships it to a backup data-center. The normal implementation of this mechanism commits transactions at the primary system (acknowledging the user’s commit request) and asynchronously ships the log. The backup database replays the log, constantly playing catch-up.

The mechanism described above is termed as log shipping. The main problem this poses is that when the primary fails and the back up takes over, some recent transactions might be lost.

This inherently opens up a window in which the work is acknowledged to the client but it has not yet been shipped to the backup. A failure of the primary during this window will lock the work inside the primary for an unknown period of time. The backup will move ahead without knowledge of the locked-up work.

The introduction of asynchrony into the system has an advantage in latency, response time and performance. However, it makes the system more prone to the possibility of losing work when the primary fails. There are two ways to deal with this:

Discard the work locked in the primary when it fails. Whether a system can do that or not depends on the requirements and business rules.
Have a recovery mechanism to sync the primary with backups when it comes back up and retries lost work. This is possible only if the operations can be retried in an idempotent way and the out-of-order retries are possible.

The system loses the notion of what the authors call “an authoritative truth”. Nobody knows the accurate state of the system at any given point in time if the work is locked in an unavailable backup or primary.

The authors conclude that business rules in a system with asynchronous checkpointing are probabilistic.

If a primary uses asynchronous checkpointing and applies a business rule on the incoming work, it is necessarily a probabilistic rule. The primary, despite its best intentions, cannot know it will be alive to enforce the business rules.

When the backup system that participates in the enforcement of these business rules is asynchronously tied to the primary, the enforcement of these rules inevitably becomes probabilistic!

The authors state that commutative operations, operations that can be reordered, can be executed independently, as long as the operation preserves business rules. However, this is hard to do with storage systems because the write operation isn’t commutative.

Another consideration is that work of a single operation is idempotent. For example, executing the operation any number of time should result in the same state of the system.

To ensure this, applications typically assign a unique number or ID to the work. This is assigned at the ingress to the system (i.e. whichever replica first handles the work). As the work request rattles around the network, it is easy for a replica to detect that it has already seen that operation and, hence, not do the work twice.

The authors suggest that different operations within a system provide different consistency guarantees. Yet, this depends on the business requirements. Some operations can choose classic consistency over availability and vice versa.

Next, the authors argue that as soon there is no notion of authoritative truth in a system. All computing boils down to three things: memories, guesses, and apologies.

Memories: you can only hope that your replica remembers what it has already seen.
Guesses: Due to only partial knowledge being available, the replicas take actions based on local state and may be wrong. “In any system which allows a degradation of the absolute truth, any action is, at best, a guess.” Any action in such a system has a high probability of being successful, but it’s still a guess.
Apologies: Mistakes are inevitable. Hence, every business needs to have an apology mechanism in place either through human intervention or by automating it.

The paper next discusses the topic of eventual consistency. The authors do this by taking the Amazon shopping cart built using Dynamo & a system for clearing checks as examples. A single replica identifies and processes the work coming into these systems. It flows to other replicas as and when connectivity permits. The requests coming into these systems are commutative (reorderable). They can be processed at different replicas in different orders.

Storage systems alone cannot provide the commutativity we need to create robust systems that function with asynchronous checkpointing. We need the business operations to reorder. Amazon’s Dynamo does not do this by itself. The shopping cart application on top of the Dynamo storage system is responsible for the semantics of eventual consistency and commutativity. The authors think it is time for us to move past the examination of eventual consistency in terms of updates and storage systems. The real action comes when examining application based operation semantics.

Next, they discuss two strategies for allocating resources in replicas that might not be able to communicate with each other:

Over-provisioning: the resources are partitioned between replicas. Each has a fixed subset of resources they can allocate. No replica can allocate a resource that’s not actually available.
Over-booking: the resources can be individually allocated without ensuring strict partitioning. This may lead to the replicas allocating a resource that’s not available, promising something they can’t deliver.

The paper talks also about something termed as the “seat reservation pattern”. This is a compromise between over-provisioning and over-booking:

Anyone who has purchased tickets online will recognize the “Seat Reservation” pattern where you can identify potential seats and then you have a bounded period of time, (typically minutes), to complete the transaction. If the transaction is not successfully concluded within the time period, the seats are once again marked as “available”.

ACID 2.0

The classic definition of ACID stands for “Atomic, Consistent, Isolated, and Durable”. Its goal is to make the application think that there is a single computer which isn’t doing anything else when the transaction is being processed. The authors talk about a new definition for ACID. which stands for Associative, Commutative, Idempotent, and Distributed.

The goal for ACID2.0 is to succeed if the pieces of the work happen: At least once, anywhere in the system, in any order. This defines a new KIND of consistency. The individual steps happen at one or more system. The application is explicitly tolerant of work happening out of order. It is tolerant of the work happening more than once per machine, too.

Going by the classic definition of ACID, a linear history is a basis for fault tolerance. If we want to achieve the same guarantees in a distributed system, it’ll require concurrency control mechanisms which “tend to be fragile”.

When the application is constrained to the additional requirements of commutativity and associativity, the world gets a LOT easier. No longer must the state be checkpointed across failure units in a synchronous fashion. Instead, it is possible to be very lazy about the sharing of information. This opens up offline, slow links, low-quality datacenters, and more.

In conclusion:

We have attempted to describe the patterns in use by many applications today as they cope with failures in widely distributed systems. It is the reorderability of work and repeatability of work that is essential to allowing successful application execution on top of the chaos of a distributed world in which systems come and go when they feel like it.

P.S. — If you made it this far and would like to receive a mail whenever I publish one of these posts, sign up here.

Harvest, Yield, and Scalable Tolerant Systems: A Summary

freeCodeCamp — Mon, 30 Oct 2017 17:14:37 +0000

By Shubheksha Jalan

This article presents a summary of the paper “Harvest, Yield, and Scalable Tolerant Systems” published by Eric Brewer & Amando Fox in 1999. All unattributed quotes are from this paper.

The paper deals with the trade-offs between consistency and availability (CAP) for large systems. It’s very easy to point to CAP and assert that no system can have consistency and availability.

But, there is a catch. CAP has been misunderstood in a variety of ways. As Coda Hale explains in his excellent blog post “You Can’t Sacrifice Partition Tolerance”:

Of the CAP theorem’s Consistency, Availability, and Partition Tolerance, Partition Tolerance is mandatory in distributed systems. You cannot not choose it. Instead of CAP, you should think about your availability in terms of yield (percent of requests answered successfully) and harvest (percent of required data actually included in the responses) and which of these two your system will sacrifice when failures happen.

The paper focuses on increasing the availability of large scale systems by fault toleration, containment and isolation:

We assume that clients make queries to servers, in which case there are at least two metrics for correct behavior: yield, which is the probability of completing a request, and harvest, which measures the fraction of the data reflected in the response, i.e. the completeness of the answer to the query.

The two metrics, harvest and yield can be summarized as follows:

Harvest: data in response/total data
For example: If one of the nodes is down in a 100 node cluster, the harvest is 99% for the duration of the fault.
Yield: requests completed with success/total number of requests
Note: Yield is different from uptime. Yield deals with the number of requests, not only the time the system wasn’t able to respond to requests.

The paper argues that there are certain systems which require perfect responses to queries every single time. Also, there are systems that can tolerate imperfect answers once in a while.

To increase the overall availability of our systems, we need to carefully think through the required consistency and availability guarantees it needs to provide.

Trading Harvest for Yield — Probabilistic Availability

Nearly all systems are probabilistic whether they realize it or not. In particular, any system that is 100% available under single faults is probabilistically available overall (since there is a non-zero probability of multiple failures)

The paper talks about understanding the probabilistic nature of availability. This helps in understanding and limiting the impact of faults by making decisions about what needs to be available and what kind of faults the system can deal with.

They outline the linear degradation of harvest in case of multiple node faults. The harvest is directly proportional to the number of nodes that are functioning correctly. Therefore, it decreases/increases linearly.

Two strategies are suggested for increasing the yield:

Random distribution of data on the nodes
If one of the nodes goes down, the average-case and worst-case fault behavior doesn’t change. Yet if the distribution isn’t random, then depending on the type of data, the impact of a fault may vary.
For example, if only one of the nodes stored information related to a user’s account balance goes down, the entire banking system will not be able to work.
Replicating the most important data
This reduces the impact in case one of the nodes containing a subset of high-priority data goes down.
It also improves harvest.

Another notable observation made in the paper is that it is possible to replicate all your data. It doesn’t do a lot to improve your harvest/yield, but it increases the cost of operation substantially. This is because the internet works based on best-in-effort protocols which can never guarantee 100% harvest/yield.

Application Decomposition and Orthogonal Mechanisms

The second strategy focuses on the benefits of orthogonal system design.

It starts out by stating that large systems are composed of subsystems which cannot tolerate failures. But they fail in a way that allows the entire system to continue functioning with some impact on utility.

The actual benefit is the ability to provision each subsystem’s state management separately, providing strong consistency or persistent state only for the subsystems that need it, not for the entire application. The savings can be significant if only a few small subsystems require the extra complexity.

The paper states that orthogonal components are completely independent of each other. They have no run time interface to other components, unless there is a configuration interface. This allows each individual component to fail independently and minimizes its impact on the overall system.

Composition of orthogonal subsystems shifts the burden of checking for possibly harmful interactions from runtime to compile time, and deployment of orthogonal guard mechanisms improves robustness for the runtime interactions that do occur, by providing improved fault containment.

The goal of this paper was to motivate research in the field of designing fault-tolerant and highly available large scale systems.

Also, to think carefully about the consistency and availability guarantees the application needs to provide. As well as the trade offs it is capable of making in terms of harvest against yield.

If you enjoyed this paper, please hit the clap button so more people see it. Thank you.

P.S. — If you made it this far and would like to receive a mail whenever I publish one of these posts, sign up here.

research - freeCodeCamp.org

Stop Staring at a Blank Deck: How I Use Claude Code + Marp to Think Through Presentations

We'll cover:

The Workflow

Brainstorm

React

Iterate

Export

Editable PPTX

Get Started in 5 Minutes

1. Install Marp CLI:

2. Install the skill (via skills.sh):

3. Run it:

4. Iterate:

Use Case: This Very Article

Under the Hood

Marp: Markdown to Slides

The Skill File

Section Dividers

Breadcrumb Navigation

Wrapping Up

About the Author

Product-Led Research: A Practical Guide for R&D Leaders [Full Book]

What You'll Learn

Who Is This Book For?

Who Am I?

The Approach of This Book

Structure of This Book

Why Is This Book Publicly Available?

Feedback Is Welcome

Table of Contents

Part 1: Foundations

Part 2: Research Management Methods

Part 3: Ensuring Product Impact

Book Summary

Note

Part 1: Foundations

Chapter 1 - What is Research?

The Difference

Schoenfeld's Framework for Problem Solving

Why This Matters for R&D Leaders

Notes

References

Chapter 2 - Research and Development

Defining Research vs. Development

A Quick Test

Key Indicators You're Doing Research

Common Misconceptions

Your Role as Research Leader

Part 1 - Summary

Part 2: Research Management Methods

Chapter 3 - Why Methodology Matters: A True Story

The Lesson

What This Part Covers

Chapter 4 - The Research Tree

What Is a Research Tree?

Your First Research Tree

The Power of Stopping to Think

Choosing Your First Path

What If Your First Choice Doesn't Work?

Choosing When Everything Seems Equal

Heuristics Can Be Combined

How Answers Lead to New Questions

The Complete Picture

Color-Coding Status

Additional Tips

The Research Tree Prevents Common Pitfalls

Time to Practice

Using the Tree with Your Team

Tools

Pro Tips

Recap - The Research Tree

Chapter 5 - Time-Boxing Research Explorations

The Problem: Research Without Time Limits

Time-Boxing: Creating Decision Points

The Decision Point

Example: Detecting God Objects

How to Set Time Boxes

Using Time-Boxing with Your Team

Integration with the Research Tree