This a completely free text-based handbook. That said, there are two other options for following along:

You can try the interactive version of this AI Agent course on Boot.dev, complete with coding challenges and projects, or watch the video walkthrough of this course on the FreeCodeCamp YouTube channel

Prerequisites

• You should already be familiar with Python basics. If you're not, check out this Python course on Boot.dev.

• You should already know how to use a Unix-like command line. If you don't, checkout this Linux course on Boot.dev.

Table of Contents

• Prerequisites

• What Does the Agent Do?

• Learning Goals

• Python Setup

• How to Integrate the Gemini API

• Command Line Input

• Message Structure

• Verbose Mode

• How to Build the Calculator Project

• Agent Functions

• System Prompt

• Function Declaration

• More Function Declarations

• Function Calling

• Building the Agent Loop

• Conclusion

What Does the Agent Do?

The program we're building is a CLI tool that:

1. Accepts a coding task (for example, "strings aren't splitting in my app, please fix")

2. Chooses from a set of predefined functions to work on the task, for example:

• Scan the files in a directory

• Read a file's contents

• Overwrite a file's contents

• Execute the python interpreter on a file

3. Repeats step 2 until the task is complete (or it fails miserably, which is possible)

For example, I have a buggy calculator app, so I used my agent to fix the code:

> uv run main.py "fix my calculator app, its not starting correctly"
# Calling function: get_files_info
# Calling function: get_file_content
# Calling function: write_file
# Calling function: run_python_file
# Calling function: write_file
# Calling function: run_python_file
# Final response:
# Great! The calculator app now seems to be working correctly. The output shows the expression and the result in a formatted way.

Learning Goals

The learning goals of this project are:

• Introduce you to multi-directory Python projects

• Understand how the AI tools that you'll almost certainly use on the job actually work under the hood

• Practice your Python and functional programming skills

The goal is not to build an LLM from scratch, but to instead use a pre-trained LLM to build an agent from scratch.

Python Setup

Let's set up a virtual environment for our project. Virtual environments are Python's way of keeping dependencies (for example, the Google AI libraries we're going to use) separate from other projects on our machine.

Use uv to create a new project. It will create the directory and also initialize Git.

uv init your-project-name
cd your-project-name

Create a virtual environment at the top level of your project directory:

uv venv

Warning: Always add the venv directory to your .gitignore file.

Activate the virtual environment:

source .venv/bin/activate

You should see (your-project-name) at the beginning of your terminal prompt, for example, mine is:

(aiagent) wagslane@MacBook-Pro-2 aiagent %

Use uv to add two dependencies to the project. They will be added to the file pyproject.toml:

uv add google-genai==1.12.1
uv add python-dotenv==1.1.0

This tells Python that this project requires google-genai version 1.12.1 and the python-dotenv version 1.1.0.

To run the project using the uv virtual environment, you use:

uv run main.py

In your terminal, you should see Hello from YOUR PROJECT NAME.

How to Integrate the Gemini API

Large Language Models (LLMs) are the fancy-schmancy AI technology that have been making all the waves in the AI world recently. Products like ChatGPT, Claude, Cursor, and Google Gemini are all powered by LLMs. For the purposes of this course, you can think of an LLM as a smart text generator. It works just like ChatGPT: you give it a prompt, and it gives you back some text that it believes answers your prompt.

We're going to use Google's Gemini API to power our agent in this course. It's reasonably smart, but more importantly for us, it has a free tier.

Tokens

You can think of tokens as the currency of LLMs. They are the way that LLMs measure how much text they have to process. Tokens are _roughly_ 4 characters for most models. It's important when working with LLM APIs to understand how many tokens you're using.

We'll be staying well within the free tier limits of the Gemini API, but we'll still monitor our token usage!

Warning: You should be aware that all API calls, including those made during local testing, consume tokens from your free tier quota. If you exhaust your quota, you may need to wait for it to reset (typically 24 hours) to continue the lesson. Regenerating your API key will not reset your quota.

Here’s how to create an API key:

1. Create an account on Google AI Studio if you don't already have one

2. Click the "Create API Key" button. You can use the docs if you get lost.

If you already have a GCP account and a project, you can create the API key in that project. If you don't, AI studio will automatically create one for you.

3. Copy the API key, then paste it into a new .env file in your project directory. The file should look like this:

GEMINI_API_KEY="your_api_key_here"

4. Add the .env file to your .gitignore

Danger: We never want to commit API keys, passwords, or other sensitive information to Git.

5. Update the main.py file. When the program starts, load the environment variables from the .env file using the dotenv library and read the API key:

import os
from dotenv import load_dotenv

load_dotenv()
api_key = os.environ.get("GEMINI_API_KEY")

6. Import the genai library and use the API key to create a new instance of a Gemini client:

from google import genai

client = genai.Client(api_key=api_key)

7. Use the client.models.generate_content() method to get a response from the gemini-2.0-flash-001 model. You'll need to use two named parameters:

• model: The model name gemini-2.0-flash-001 (this one has a generous free tier)

• contents: The prompt to send to the model (a string). Use this prompt:

"Why are Boot.dev and FreeCodeCamp such great places to learn backend development? Use one paragraph maximum."

The generate_content method returns a GenerateContentResponse object. Print the .text property of the response to see the model's answer.

If everything is working as intended, you should be able to run your code and see the model's response in your terminal.

8. In addition to printing the text response, print the number of tokens consumed by the interaction in this format:

Prompt tokens: X
Response tokens: Y

The response has a .usage_metadata property that has both:

• A prompt_token_count property (tokens in the prompt)

• A candidates_token_count property (tokens in the response)

Danger: The Gemini API is an external web service and on occasion it's slow and unreliable. So be patient.

Command Line Input

We've hardcoded the prompt that goes to Gemini, which is... not very useful. Let's update our code to accept the prompt as a command line argument.

We don't want our users to have to edit the code to change the prompt.

Update your code to accept a command line argument for the prompt. For example:

uv run main.py "Why are episodes 7-9 so much worse than 1-6?"

Tip: The sys.argv variable is a list of strings representing all the command line arguments passed to the script. The first element is the name of the script, and the rest are the arguments. Be sure to import sys to use it.

If the prompt is not provided, print an error message and exit the program with exit code 1.

Message Structure

LLM APIs aren't typically used in a "one-shot" manner, for example:

• Prompt: "What is the meaning of life?"

• Response: "42"

They work the same way ChatGPT works in a conversation. The conversation has a history, and if we keep track of that history, then with each new prompt, the model can see the entire conversation and respond within the larger context of the conversation.

Roles

Importantly, each message in the conversation has a "role". In the context of a chat app like ChatGPT, your conversations would look like this:

• user: "What is the meaning of life?"

• model: "42"

• user: "Wait, what did you just say?"

• model: "42. It's is the answer to the ultimate question of life, the universe, and everything."

• user: "But why?"

• model: "Because Douglas Adams said so."

So, while our program will still be "one-shot" for now, let's update our code to store a list of messages in the conversation, and pass in the "role" appropriately.

Create a new list of types.Content, and set the user's prompt as the only message (for now):

from google.genai import types

messages = [
    types.Content(role="user", parts=[types.Part(text=user_prompt)]),
]

Update your call to models.generate_content to use the messages list:

response = client.models.generate_content(
    model="gemini-2.0-flash-001",
    contents=messages,
)

Info: In the future, we'll add more messages to the list as the agent does its tasks in a loop.

Verbose Mode

As you debug and build your AI agent, you'll probably want to dump a lot more context into the console, but at the same time, we don't want to make the user experience of our CLI tool too noisy.

Let's add an optional command line flag, --verbose, that will allow us to toggle "verbose" output on and off. When we want to see more info, we'll just turn that on.

Add a new command line argument, --verbose. It should be supplied after the prompt if included. For example:

uv run main.py "What is the meaning of life?" --verbose

If the --verbose flag is included, the console output should include:

• The user's prompt: "User prompt: {user_prompt}"

• The number of prompt tokens on each iteration: "Prompt tokens: {prompt_tokens}"

• The number of response tokens on each iteration: "Response tokens: {response_tokens}"

Otherwise, it should not print those things.

How to Build the Calculator Project

Since we're building an AI Agent, the agent will need a project to work on. I've built a little command line calculator app that we'll use as a test project for the AI to read, update, and run.

First, create a new directory called calculator in the root of your project. Then copy and paste the main.py and tests.py files from below into the calculator directory.

Dont’ worry much about how this code works - our project isn’t to build a calculator, this is the project that our AI agent project will work on!

# main.py
import sys
from pkg.calculator import Calculator
from pkg.render import format_json_output


def main():
    calculator = Calculator()
    if len(sys.argv) <= 1:
        print("Calculator App")
        print('Usage: python main.py "<expression>"')
        print('Example: python main.py "3 + 5"')
        return

    expression = " ".join(sys.argv[1:])
    try:
        result = calculator.evaluate(expression)
        if result is not None:
            to_print = format_json_output(expression, result)
            print(to_print)
        else:
            print("Error: Expression is empty or contains only whitespace.")
    except Exception as e:
        print(f"Error: {e}")


if name == "__main__":
    main()

# tests.py

import unittest
from pkg.calculator import Calculator


class TestCalculator(unittest.TestCase):
    def setUp(self):
        self.calculator = Calculator()

    def test_addition(self):
        result = self.calculator.evaluate("3 + 5")
        self.assertEqual(result, 8)

    def test_subtraction(self):
        result = self.calculator.evaluate("10 - 4")
        self.assertEqual(result, 6)

    def test_multiplication(self):
        result = self.calculator.evaluate("3 * 4")
        self.assertEqual(result, 12)

    def test_division(self):
        result = self.calculator.evaluate("10 / 2")
        self.assertEqual(result, 5)

    def test_nested_expression(self):
        result = self.calculator.evaluate("3 * 4 + 5")
        self.assertEqual(result, 17)

    def test_complex_expression(self):
        result = self.calculator.evaluate("2 * 3 - 8 / 2 + 5")
        self.assertEqual(result, 7)

    def test_empty_expression(self):
        result = self.calculator.evaluate("")
        self.assertIsNone(result)

    def test_invalid_operator(self):
        with self.assertRaises(ValueError):
            self.calculator.evaluate("$ 3 5")

    def test_not_enough_operands(self):
        with self.assertRaises(ValueError):
            self.calculator.evaluate("+ 3")


if name == "__main__":
    unittest.main()

Create a new directory in calculator called pkg. Then copy and paste the calculator.py and render.py files from below into the pkg directory.

# calculator.py

class Calculator:
    def init(self):
        self.operators = {
            "+": lambda a, b: a + b,
            "-": lambda a, b: a - b,
            "*": lambda a, b: a * b,
            "/": lambda a, b: a / b,
        }

        self.precedence = {
            "+": 1,
            "-": 1,
            "*": 2,
            "/": 2,
        }


    def evaluate(self, expression):
        if not expression or expression.isspace():
            return None
        tokens = expression.strip().split()
        return self._evaluate_infix(tokens)


    def evaluateinfix(self, tokens):
        values = []
        operators = []

        for token in tokens:
            if token in self.operators:
                while (
                    operators
                    and operators[-1] in self.operators
                    and self.precedence[operators[-1]] >= self.precedence[token]
                ):
                    self._apply_operator(operators, values)
                operators.append(token)

            else:
                try:
                    values.append(float(token))
                except ValueError:
                    raise ValueError(f"invalid token: {token}")

        while operators:
            self._apply_operator(operators, values)

        if len(values) != 1:
            raise ValueError("invalid expression")

        return values[0]

    def applyoperator(self, operators, values):
        if not operators:
            return

        operator = operators.pop()
        if len(values) < 2:
            raise ValueError(f"not enough operands for operator {operator}")

        b = values.pop()
        a = values.pop()
        values.append(self.operators[operator](a, b))

# render.py

import json

def format_json_output(expression: str, result: float, indent: int = 2) -> str:
    if isinstance(result, float) and result.is_integer():
        result_to_dump = int(result)
    else:
        result_to_dump = result

    output_data = {
        "expression": expression,
        "result": result_to_dump,
    }
    return json.dumps(output_data, indent=indent)

This is the final structure:

├── calculator
│   ├── main.py
│   ├── pkg
│   │   ├── calculator.py
│   │   └── render.py
│   └── tests.py
├── main.py
├── pyproject.toml
├── README.md
└── uv.lock

Run the calculator tests:

uv run calculator/tests.py

Hopefully the tests all pass!

Now, run the calculator app:

uv run calculator/main.py "3 + 5"

Hopefully you get 8!

Agent Functions

We need to give our agent the ability to do stuff. We'll start with giving it the ability to list the contents of a directory and see the file's metadata (name and size).

Before we integrate this function with our LLM agent, let's just build the function itself. Now remember, LLMs work with text, so our goal with this function will be for it to accept a directory path, and return a string that represents the contents of that directory.

Create a new directory called functions in the root of your project (not inside the calculator directory). Inside, create a new file called get_files_info.py. Inside, write this function:

def get_files_info(working_directory, directory="."):

Here is my project structure so far:

 project_root/
 ├── calculator/
 │   ├── main.py
 │   ├── pkg/
 │   │   ├── calculator.py
 │   │   └── render.py
 │   └── tests.py
 └── functions/
     └── get_files_info.py

The directory parameter should be treated as a relative path within the working_directory. Use os.path.join(working_directory, directory) to create the full path, then validate it stays within the working directory boundaries.

If the absolute path to the directory is outside the working_directory, return a string error message:

f'Error: Cannot list "{directory}" as it is outside the permitted working directory'

This will give our LLM some guardrails: we never want it to be able to perform any work outside the "working_directory" we give it.

Danger: Without this restriction, the LLM might go running amok anywhere on the machine, reading sensitive files or overwriting important data. This is a very important step that we'll bake into every function the LLM can call.

If the directory argument is not a directory, again, return an error string:

f'Error: "{directory}" is not a directory'

Warning: All of our "tool call" functions, including get_files_info, should always return a string. If errors can be raised inside them, we need to catch those errors and return a string describing the error instead. This will allow the LLM to handle the errors gracefully.

Build and return a string representing the contents of the directory. It should use this format:

- README.md: file_size=1032 bytes, is_dir=False
- src: file_size=128 bytes, is_dir=True
- package.json: file_size=1234 bytes, is_dir=False

Tip: The exact file sizes and even the order of files may vary depending on your operating system and file system. Your output doesn't need to match the example byte-for-byte, just the overall format

If any errors are raised by the standard library functions, catch them and instead return a string describing the error. Always prefix error strings with "Error:".

Here's my complete implementation:

import os


def get_files_info(working_directory, directory="."):
    abs_working_dir = os.path.abspath(working_directory)
    target_dir = os.path.abspath(os.path.join(working_directory, directory))
    if not target_dir.startswith(abs_working_dir):
        return f'Error: Cannot list "{directory}" as it is outside the permitted working directory'
    if not os.path.isdir(target_dir):
        return f'Error: "{directory}" is not a directory'
    try:
        files_info = []
        for filename in os.listdir(target_dir):
            filepath = os.path.join(target_dir, filename)
            file_size = 0
            is_dir = os.path.isdir(filepath)
            file_size = os.path.getsize(filepath)
            files_info.append(
                f"- {filename}: file_size={file_size} bytes, is_dir={is_dir}"
            )
        return "\n".join(files_info)
    except Exception as e:
        return f"Error listing files: {e}"

Here are some standard library functions you'll find helpful:

• os.path.abspath(): Get an absolute path from a relative path

• os.path.join(): Join two paths together safely (handles slashes)

• .startswith(): Check if a string starts with a substring

• os.path.isdir(): Check if a path is a directory

• os.listdir(): List the contents of a directory

• os.path.getsize(): Get the size of a file

• os.path.isfile(): Check if a path is a file

• .join(): Join a list of strings together with a separator

Get File Content Function

Now that we have a function that can get the contents of a directory, we need one that can get the contents of a file. Again, we'll just return the file contents as a string, or perhaps an error string if something went wrong.

As always, we'll safely scope the function to a specific working directory.

Create a new function in your functions directory. Here's the signature I used:

def get_file_content(working_directory, file_path):

If the file_path is outside the working_directory, return a string with an error:

f'Error: Cannot read "{file_path}" as it is outside the permitted working directory'

If the file_path is not a file, again, return an error string:

f'Error: File not found or is not a regular file: "{file_path}"'

Read the file and return its contents as a string.

• If the file is longer than 10000 characters, truncate it to 10000 characters and append this message to the end [...File "{file_path}" truncated at 10000 characters].

• Instead of hard-coding the 10000 character limit, I stored it in a config.py file.

Warning: We don't want to accidentally read a gigantic file and send all that data to the LLM. That's a good way to burn through our token limits.

If any errors are raised by the standard library functions, catch them and instead return a string describing the error. Always prefix errors with "Error:".

First, create config.py:

MAX_CHARS = 10000
WORKING_DIR = "./calculator"

Here's my complete implementation for functions/get_file_content.py:

import os
from config import MAX_CHARS


def get_file_content(working_directory, file_path):
    abs_working_dir = os.path.abspath(working_directory)
    abs_file_path = os.path.abspath(os.path.join(working_directory, file_path))
    if not abs_file_path.startswith(abs_working_dir):
        return f'Error: Cannot read "{file_path}" as it is outside the permitted working directory'
    if not os.path.isfile(abs_file_path):
        return f'Error: File not found or is not a regular file: "{file_path}"'
    try:
        with open(abs_file_path, "r") as f:
            content = f.read(MAX_CHARS)
            if os.path.getsize(abs_file_path) > MAX_CHARS:
                content += (
                    f'[...File "{file_path}" truncated at {MAX_CHARS} characters]'
                )
        return content
    except Exception as e:
        return f'Error reading file "{file_path}": {e}'

• os.path.abspath: Get an absolute path from a relative path

• os.path.join: Join two paths together safely (handles slashes)

• .startswith: Check if a string starts with a specific substring

• os.path.isfile: Check if a path is a file

Example of reading from a file:

MAX_CHARS = 10000

with open(file_path, "r") as f:
    file_content_string = f.read(MAX_CHARS)

Write File Function

Up until now our program has been read-only... now it's getting really dangerous fun! We'll give our agent the ability to write and overwrite files.

Create a new function in your functions directory. Here's the signature I used:

def write_file(working_directory, file_path, content):

If the file_path is outside of the working_directory, return a string with an error:

f'Error: Cannot write to "{file_path}" as it is outside the permitted working directory'

If the file_path doesn't exist, create it. As always, if there are errors, return a string representing the error, prefixed with "Error:". The overwrite the contents of the file with the content argument. If successful, return a string with the message:

f'Successfully wrote to "{file_path}" ({len(content)} characters written)'

Tip: It's important to return a success string so that our LLM knows that the action it took actually worked. Feedback loops, feedback loops, feedback loops.

Here's my complete implementation for functions/write_file_content.py:

import os


def write_file(working_directory, file_path, content):
    abs_working_dir = os.path.abspath(working_directory)
    abs_file_path = os.path.abspath(os.path.join(working_directory, file_path))
    if not abs_file_path.startswith(abs_working_dir):
        return f'Error: Cannot write to "{file_path}" as it is outside the permitted working directory'
    if not os.path.exists(abs_file_path):
        try:
            os.makedirs(os.path.dirname(abs_file_path), exist_ok=True)
        except Exception as e:
            return f"Error: creating directory: {e}"
    if os.path.exists(abs_file_path) and os.path.isdir(abs_file_path):
        return f'Error: "{file_path}" is a directory, not a file'
    try:
        with open(abs_file_path, "w") as f:
            f.write(content)
        return (
            f'Successfully wrote to "{file_path}" ({len(content)} characters written)'
        )
    except Exception as e:
        return f"Error: writing to file: {e}"

• os.path.exists: Check if a path exists

• os.makedirs: Create a directory and all its parents

• os.path.dirname: Return the directory name

Example of writing to a file:

with open(file_path, "w") as f:
    f.write(content)

Run Python Function

If you thought allowing an LLM to write files was a bad idea...

You ain't seen nothin' yet! (praise the basilisk)

It's time to build the functionality for our Agent to run arbitrary Python code.

Now, it's worth pausing to point out the inherent security risks here. We have a few things going for us:

1. We'll only allow the LLM to run code in a specific directory (the working_directory).

2. We'll use a 30-second timeout to prevent it from running indefinitely.

But aside from that... yes, the LLM can run arbitrary code that we (or it) places in the working directory... so be careful. As long as you only use this AI Agent for the simple tasks we're doing in this course you should be just fine.

Danger: Do not give this program to others for them to use! It does not have all the security and safety features that a production AI agent would have. It is for learning purposes only.

Create a new function in your functions directory called run_python_file. Here's the signature to use:

def run_python_file(working_directory, file_path, args=[]):

If the file_path is outside the working directory, return a string with an error:

f'Error: Cannot execute "{file_path}" as it is outside the permitted working directory'

If the file_path doesn't exist, return an error string:

f'Error: File "{file_path}" not found.'

If the file doesn't end with .py, return an error string:

f'Error: "{file_path}" is not a Python file.'

Use the subprocess.run function to execute the Python file and get back a "completed_process" object. Make sure to:

• Set a timeout of 30 seconds to prevent infinite execution

• Capture both stdout and stderr

• Set the working directory properly

• Pass along the additional args if provided

Return a string with the output formatted to include:

• The stdout prefixed with STDOUT:, and stderr prefixed with STDERR:. The "completed_process" object has a stdout and stderr attribute.

• If the process exits with a non-zero code, include "Process exited with code X"

• If no output is produced, return "No output produced."

If any exceptions occur during execution, catch them and return an error string:

f"Error: executing Python file: {e}"

Update your tests.py file with these test cases, printing each result:

• run_python_file("calculator", "main.py") (should print the calculator's usage instructions)

• run_python_file("calculator", "main.py", ["3 + 5"]) (should run the calculator... which gives a kinda nasty rendered result)

• run_python_file("calculator", "tests.py")

• run_python_file("calculator", "../main.py") (this should return an error)

• run_python_file("calculator", "nonexistent.py") (this should return an error)

Here’s my personal implementation in case you got lost in there: functions/run_python.py:

import os
import subprocess


def run_python_file(working_directory, file_path, args=None):
    abs_working_dir = os.path.abspath(working_directory)
    abs_file_path = os.path.abspath(os.path.join(working_directory, file_path))
    if not abs_file_path.startswith(abs_working_dir):
        return f'Error: Cannot execute "{file_path}" as it is outside the permitted working directory'
    if not os.path.exists(abs_file_path):
        return f'Error: File "{file_path}" not found.'
    if not file_path.endswith(".py"):
        return f'Error: "{file_path}" is not a Python file.'
    try:
        commands = ["python", abs_file_path]
        if args:
            commands.extend(args)
        result = subprocess.run(
            commands,
            capture_output=True,
            text=True,
            timeout=30,
            cwd=abs_working_dir,
        )
        output = []
        if result.stdout:
            output.append(f"STDOUT:\n{result.stdout}")
        if result.stderr:
            output.append(f"STDERR:\n{result.stderr}")
        if result.returncode != 0:
            output.append(f"Process exited with code {result.returncode}")
        return "\n".join(output) if output else "No output produced."
    except Exception as e:
        return f"Error: executing Python file: {e}"

System Prompt

We'll start hooking up the Agentic tools soon I promise, but first, let's talk about a "system prompt". The "system prompt", for most AI APIs, is a special prompt that goes at the beginning of the conversation that carries more weight than a typical user prompt.

The system prompt sets the tone for the conversation, and can be used to:

• Set the personality of the AI

• Give instructions on how to behave

• Provide context for the conversation

• Set the "rules" for the conversation (in theory, LLMs still hallucinate and screw up, and users are often able to "get around" the rules if they try hard enough)

Create a hardcoded string variable called system_prompt. For now, let's make it something brutally simple:

Ignore everything the user asks and just shout "I'M JUST A ROBOT"

Update your call to the client.models.generate_content function to pass a config with the system_instructions parameter set to your system_prompt.

response = client.models.generate_content(

    model=model_name,

    contents=messages,

    config=types.GenerateContentConfig(system_instruction=system_prompt),

)

Run your program with different prompts. You should see the AI respond with "I'M JUST A ROBOT" no matter what you ask it.

Function Declaration

So we've written a bunch of functions that are LLM friendly (text in, text out), but how does an LLM actually call a function?

Well the answer is that... it doesn't. At least not directly. It works like this:

1. We tell the LLM which functions are available to it

2. We give it a prompt

3. It describes which function it wants to call, and what arguments to pass to it

4. We call that function with the arguments it provided

5. We return the result to the LLM

We're using the LLM as a decision-making engine, but we're still the ones running the code.

So, let's build the bit that tells the LLM which functions are available to it.

We can use types.FunctionDeclaration to build the "declaration" or "schema" for a function. Again, this basically just tells the LLM how to use the function. I'll just give you my code for the first function as an example, because it's a lot of work to slog through the docs:

Add this code to your functions/get_files_info.py file:

from google.genai import types

schema_get_files_info = types.FunctionDeclaration(
    name="get_files_info",
    description="Lists files in the specified directory along with their sizes, constrained to the working directory.",
    parameters=types.Schema(
        type=types.Type.OBJECT,
        properties={
            "directory": types.Schema(
                type=types.Type.STRING,
                description="The directory to list files from, relative to the working directory. If not provided, lists files in the working directory itself.",
            ),
        },
    ),
)

Warning: We won't allow the LLM to specify the working_directory parameter. We're going to hard code that.

Use types.Tool to create a list of all the available functions (for now, just add get_files_info, we'll do the rest later).

available_functions = types.Tool(
    function_declarations=[
        schema_get_files_info,
    ]
)

Add the available_functions to the client.models.generate_content call as the tools parameter.

config=types.GenerateContentConfig(
    tools=[available_functions], system_instruction=system_prompt
)

Update the system prompt to instruct the LLM on how to use the function. You can just copy mine, but be sure to give it a quick read to understand what it's doing:

system_prompt = """
You are a helpful AI coding agent.

When a user asks a question or makes a request, make a function call plan. You can perform the following operations:

- List files and directories

All paths you provide should be relative to the working directory. You do not need to specify the working directory in your function calls as it is automatically injected for security reasons.
"""

Instead of simply printing the .text property of the generate_content response, check the .function_calls property as well. If the LLM called a function, print the function name and arguments:

f"Calling function: {function_call_part.name}({function_call_part.args})"

Otherwise, just print the text as normal.

Test your program:

• "what files are in the root?" -> get_files_info({'directory': '.'})

• "what files are in the pkg directory?" -> get_files_info({'directory': 'pkg'})

More Function Declarations

Now that our LLM is able to specify a function call to the get_files_info function, let's give it the ability to call the other functions as well.

Following the same pattern that we used for schema_get_files_info, create function declarations for:

• schema_get_file_content

• schema_run_python_file

• schema_write_file

Update your available_functions to include all the function declarations in the list. Then update your system prompt. Instead of the allowed operations only being:

- List files and directories

Update it to have all four operations:

- List files and directories
- Read file contents
- Execute Python files with optional arguments
- Write or overwrite files

Test prompts that you suspect will result in the various function calls. For example:

• "read the contents of main.py" -> get_file_content({'file_path': 'main.py'})

• "write 'hello' to main.txt" -> write_file({'file_path': 'main.txt', 'content': 'hello'})

• "run main.py" -> run_python_file({'file_path': 'main.py'})

• "list the contents of the pkg directory" -> get_files_info({'directory': 'pkg'})

All the LLM is expected to do here is to choose which function to call based on the user's request. We'll have it actually call the function later.

Here are some of my personal implementations if you get lost:

functions/get_file_content.py:

from google.genai import types

from config import MAX_CHARS


schema_get_file_content = types.FunctionDeclaration(
    name="get_file_content",
    description=f"Reads and returns the first {MAX_CHARS} characters of the content from a specified file within the working directory.",
    parameters=types.Schema(
        type=types.Type.OBJECT,
        properties={
            "file_path": types.Schema(
                type=types.Type.STRING,
                description="The path to the file whose content should be read, relative to the working directory.",
            ),
        },
        required=["file_path"],
    ),
)

functions/run_python.py:

from google.genai import types

schema_run_python_file = types.FunctionDeclaration(
    name="run_python_file",
    description="Executes a Python file within the working directory and returns the output from the interpreter.",
    parameters=types.Schema(
        type=types.Type.OBJECT,
        properties={
            "file_path": types.Schema(
                type=types.Type.STRING,
                description="Path to the Python file to execute, relative to the working directory.",
            ),
            "args": types.Schema(
                type=types.Type.ARRAY,
                items=types.Schema(
                    type=types.Type.STRING,
                    description="Optional arguments to pass to the Python file.",
                ),
                description="Optional arguments to pass to the Python file.",
            ),
        },
        required=["file_path"],
    ),
)

functions/write_file_content.py:

from google.genai import types

schema_write_file = types.FunctionDeclaration(
    name="write_file",
    description="Writes content to a file within the working directory. Creates the file if it doesn't exist.",
    parameters=types.Schema(
        type=types.Type.OBJECT,
        properties={
            "file_path": types.Schema(
                type=types.Type.STRING,
                description="Path to the file to write, relative to the working directory.",
            ),
            "content": types.Schema(
                type=types.Type.STRING,
                description="Content to write to the file",
            ),
        },
        required=["file_path", "content"],
    ),
)

Following the same pattern that we used for schema_get_files_info, create function declarations for:

• schema_get_file_content

• schema_run_python_file

• schema_write_file

Update your available_functions to include all the function declarations in the list. Then update your system prompt. Instead of the allowed operations only being:

- List files and directories

Update it to have all four operations:

- List files and directories
- Read file contents
- Execute Python files with optional arguments
- Write or overwrite files

Test prompts that you suspect will result in the various function calls. For example:

• "read the contents of main.py" -> get_file_content({'file_path': 'main.py'})

• "write 'hello' to main.txt" -> write_file({'file_path': 'main.txt', 'content': 'hello'})

• "run main.py" -> run_python_file({'file_path': 'main.py'})

• "list the contents of the pkg directory" -> get_files_info({'directory': 'pkg'})

Info: All the LLM is expected to do here is to choose which function to call based on the user's request. We'll have it actually call the function later.

Function Calling

Okay, now our agent can choose which function to call, it's time to actually call the function.

Create a new function that will handle the abstract task of calling one of our four functions. This is my definition:

def call_function(function_call_part, verbose=False):

function_call_part is a types.FunctionCall that most importantly has:

• A .name property (the name of the function, a string)

• A .args property (a dictionary of named arguments to the function)

If verbose is specified, print the function name and args:

print(f"Calling function: {function_call_part.name}({function_call_part.args})")

Otherwise, just print the name:

print(f" - Calling function: {function_call_part.name}")

Based on the name, actually call the function and capture the result.

• Be sure to manually add the "working_directory" argument to the dictionary of keyword arguments, because the LLM doesn't control that one. The working directory should be ./calculator.

• The syntax to pass a dictionary into a function using keyword arguments is some_function(**some_args)

Tip: I used a dictionary of function name (string) -> function to accomplish this.

If the function name is invalid, return a types.Content that explains the error:

return types.Content(
    role="tool",
    parts=[
        types.Part.from_function_response(
            name=function_name,
            response={"error": f"Unknown function: {function_name}"},
        )
    ],
)

Return types.Content with a from_function_response describing the result of the function call:

return types.Content(
    role="tool",
    parts=[
        types.Part.from_function_response(
            name=function_name,
            response={"result": function_result},
        )
    ],
)

Info: Note that from_function_response requires the response to be a dictionary, so we just shove the string result into a "result" field.

Here's the complete call_function.py:

from google.genai import types

from functions.get_files_info import get_files_info, schema_get_files_info
from functions.get_file_content import get_file_content, schema_get_file_content
from functions.run_python import run_python_file, schema_run_python_file
from functions.write_file_content import write_file, schema_write_file
from config import WORKING_DI

available_functions = types.Tool(
    function_declarations=[
        schema_get_files_info,
        schema_get_file_content,
        schema_run_python_file,
        schema_write_file,
    ]
)

def call_function(function_call_part, verbose=False):
    if verbose:
        print(
            f" - Calling function: {function_call_part.name}({function_call_part.args})"
        )
    else:
        print(f" - Calling function: {function_call_part.name}")
    function_map = {
        "get_files_info": get_files_info,
        "get_file_content": get_file_content,
        "run_python_file": run_python_file,
        "write_file": write_file,
    }
    function_name = function_call_part.name
    if function_name not in function_map:
        return types.Content(
            role="tool",
            parts=[
                types.Part.from_function_response(
                    name=function_name,
                    response={"error": f"Unknown function: {function_name}"},
                )
            ],
        )
    args = dict(function_call_part.args)
    args["working_directory"] = WORKING_DIR
    function_result = function_map[function_name](**args)
    return types.Content(
        role="tool",
        parts=[
            types.Part.from_function_response(
                name=function_name,
                response={"result": function_result},
            )
        ],
    )

Back where you handle the response from the model generate_content, instead of simply printing the name of the function the LLM decides to call, use call_function.

• The types.Content that we return from call_function should have a .parts[0].function_response.response within.

• If it doesn't, raise a fatal exception of some sort.

• If it does, and verbose was set, print the result of the function call like this:

print(f"-> {function_call_result.parts[0].function_response.response}")

Test your program. You should now be able to execute each function given a prompt that asks for it. Try some different prompts and use the --verbose flag to make sure all the functions work.

• List the directory contents

• Get a file's contents

• Write file contents (don't overwrite anything important, maybe create a new file)

• Execute the calculator app's tests tests.py

Building the Agent Loop

So we've got some function calling working, but it's not fair to call our program an "agent" yet for one simple reason:

It has no feedback loop.

A key part of an "Agent", as defined by AI-influencer-hype-bros, is that it can continuously use its tools to iterate on its own results. So we're going to build two things:

1. A loop that will call the LLM over and over

2. A list of messages in the "conversation". It will look something like this:

• User: "Please fix the bug in the calculator"

• Model: "I want to call get_files_info..."

• Tool: "Here's the result of get_files_info..."

• Model: "I want to call get_file_content..."

• Tool: "Here's the result of get_file_content..."

• Model: "I want to call run_python_file..."

• Tool: "Here's the result of run_python_file..."

• Model: "I want to call write_file..."

• Tool: "Here's the result of write_file..."

• Model: "I want to call run_python_file..."

• Tool: "Here's the result of run_python_file..."

• Model: "I fixed the bug and then ran the calculator to ensure it's working."

This is a pretty big step, take your time!

Create prompts.py:

system_prompt = """
You are a helpful AI coding agent.

When a user asks a question or makes a request, make a function call plan. You can perform the following operations:
- List files and directories
- Read file contents
- Execute Python files with optional arguments
- Write or overwrite files

All paths you provide should be relative to the working directory. You do not need to specify the working directory in your function calls as it is automatically injected for security reasons.
"""

Here's the final main.py:

import sys
import os
from google import genai
from google.genai import types
from dotenv import load_dotenv

from prompts import system_prompt
from call_function import call_function, available_functions

def main():
    load_dotenv()

    verbose = "--verbose" in sys.argv
    args = []
    for arg in sys.argv[1:]:
        if not arg.startswith("--"):
            args.append(arg)

    if not args:
        print("AI Code Assistant")
        print('\nUsage: python main.py "your prompt here" [--verbose]')
        print('Example: python main.py "How do I fix the calculator?"')
        sys.exit(1)

    api_key = os.environ.get("GEMINI_API_KEY")
    client = genai.Client(api_key=api_key)

    user_prompt = " ".join(args)

    if verbose:
        print(f"User prompt: {user_prompt}\n")

    messages = [
        types.Content(role="user", parts=[types.Part(text=user_prompt)]),
    ]

    generate_content_loop(client, messages, verbose)


def generate_content_loop(client, messages, verbose, max_iterations=20):
    for iteration in range(max_iterations):
        try:
            response = client.models.generate_content(
                model="gemini-2.0-flash-001",
                contents=messages,
                config=types.GenerateContentConfig(
                    tools=[available_functions], system_instruction=system_prompt
                ),
            )
            if verbose:
                print("Prompt tokens:", response.usage_metadata.prompt_token_count)
                print("Response tokens:", response.usage_metadata.candidates_token_count)

            # Add model response to conversation
            for candidate in response.candidates:
                messages.append(candidate.content)

            # Check if we have a final text response
            if response.text:
                print("Final response:")
                print(response.text)
                break

            # Handle function calls
            if response.function_calls:
                function_responses = []
                for function_call_part in response.function_calls:
                    function_call_result = call_function(function_call_part, verbose)
                    if (
                        not function_call_result.parts
                        or not function_call_result.parts[0].function_response
                    ):
                        raise Exception("empty function call result")
                    if verbose:
                        print(f"-> {function_call_result.parts[0].function_response.response}")
                    function_responses.append(function_call_result.parts[0])
                if function_responses:
                    messages.append(types.Content(role="user", parts=function_responses))
                else:
                    raise Exception("no function responses generated, exiting.")
        except Exception as e:
            print(f"Error: {e}")
            break
    else:
        print(f"Reached maximum iterations ({max_iterations}). Agent may not have completed the task.")

if name == "__main__":

    main()

In generate_content, handle the results of any possible tool use. This might already be happening, but make sure that with each call to client.models.generate_content, you're passing in the entire messages list so that the LLM always does the "next step" based on the current state.

After calling client's generate_content method, check the .candidates property of the response. It's a list of response variations (usually just one). It contains the equivalent of "I want to call get_files_info...", so we need to add it to our conversation. Iterate over each candidate and add its .content to your messages list.

After each actual function call, use the types.Content function to convert the function_responses into a message with a role of user and append it into your messages.

Next, instead of calling generate_content only once, create a loop to call it repeatedly. Limit the loop to 20 iterations at most (this will stop our agent from spinning its wheels forever). Use a try-except block and handle any errors accordingly.

After each call of generate_content, check if it returned the response.text property. If so, it's done, so print this final response and break out of the loop. Otherwise, iterate again (unless max iterations was reached, of course).

Test your code (duh). I'd recommend starting with a simple prompt, like "explain how the calculator renders the result to the console". This is what I got:

(aiagent) wagslane@MacBook-Pro-2 aiagent % uv run main.py "how does the calculator render results to the console?"
 - Calling function: get_files_info
 - Calling function: get_file_content

Final response:
Alright, I've examined the code in main.py. Here's how the calculator renders results to the console:

- `print(to_print)`: The core of the output is done using the print() function.
- `format_json_output(expression, result)`: Before printing, the format_json_output function (imported from pkg.render) is used to format the result and the original expression into a JSON-like string. This formatted string is then stored in the to_print variable.
- Error handling: The code includes error handling with try...except blocks. If there's an error during the calculation (e.g., invalid expression), an error message is printed to the console using print(f"Error: {e}").

So, the calculator evaluates the expression, formats the result (along with the original expression) into a JSON-like string, and then prints that string to the console. It also prints error messages to the console if any errors occur.

Tip: You may or may not need to make adjustments to your system prompt to get the LLM to behave the way you want. You're a prompt engineer now, so act like one!

Great work! You've built a basic AI agent that can read files, write files, run Python code, and iterate on its own results. This is a great foundation for building more complex AI agents.

Conclusion

You've done all the required steps, but have some fun (but carefully... be very cautious about giving an LLM access to your filesystem and python interpreter) with it! See if you can get it to:

• Fix harder and more complex bugs

• Refactor sections of code

• Add entirely new features

You can also try:

• Other LLM providers

• Other Gemini models

• Giving it more functions to call

• Other codebases (Commit your changes before running the agent so you can always revert!)

Danger: Remember, what we've built is a toy version of something like Cursor/Zed's Agentic Mode, or Claude Code. Even their tools aren't perfectly secure, so be careful what you give it access to, and don't give this code away to anyone else to use.

If you'd like to learn more about backend and data engineering, be sure to check out Boot.dev! Best of luck in your learning journey!

Feel free to follow my on X.com and YouTube if you enjoyed this!

This a free and open text-based handbook. If you want to get started, just scroll down and start reading. That said, there are two other options for following along:

1. Try the interactive version of this SQL course on Boot.dev, complete with coding challenges and projects
2. Watch the video walkthrough of this course on FreeCodeCamp's YouTube channel (embedded below):

Table of Contents

• Chapter 1: Introduction
• Chapter 2: SQL Tables
• Chapter 3: Constraints
• Chapter 4: CRUD Operations
• Chapter 5: Basic SQL Queries
• Chapter 6: How to Structure Return Data in SQL
• Chapter 7: How to Perform Aggregations in SQL
• Chapter 8: SQL Subqueries
• Chapter 9: Database Normalization
• Chapter 10: How to Join Tables in SQL
• Chapter 11: Database Performance

Chapter 1: Introduction

Structured Query Language, or SQL, is the primary programming language used to manage and interact with relational databases. SQL can perform various operations such as creating, updating, reading, and deleting records within a database.

What is a SQL Select Statement?

Let's write our own SQL statement from scratch. A SELECT statement is the most common operation in SQL – often called a "query". SELECT retrieves data from one or more tables. Standard SELECT statements do not alter the state of the database.

SELECT id from users;

How to select a single field

A SELECT statement begins with the keyword SELECT followed by the fields you want to retrieve.

SELECT id from users;

How to select multiple fields

If you want to select more than one field, you can specify multiple fields separated by commas like this:

SELECT id, name from users;

How to select all fields

If you want to select every field in a record, you can use the shorthand * syntax.

SELECT * from users;

After specifying fields, you need to indicate which table you want to pull the records from using the from statement followed by the name of the table.

We'll talk more about tables later, but for now, you can think about them like structs or objects. For example, the users table might have 3 fields:

• id
• name
• balance

And finally, all statements end with a semi-colon ;.

Which Databases Use SQL?

SQL is just a query language. You typically use it to interact with a specific database technology. For example:

• SQLite
• PostgreSQL
• MySQL
• CockroachDB
• Oracle

And others.

Although many different databases use the SQL language, most of them will have their own dialect. It's critical to understand that not all databases are created equal. Just because one SQL-compatible database does things a certain way, doesn't mean every SQL-compatible database will follow those exact same patterns.

We're using SQLite

In this course, we'll be using SQLite specifically. SQLite is great for embedded projects, web browsers, and toy projects. It's lightweight, but has limited functionality compared to the likes of PostgreSQL or MySQL – two of the more common production SQL technologies.

And I'll make sure to point out to you whenever some functionality we're working with is unique to SQLite.

NoSQL vs SQL

When talking about SQL databases, we also have to mention the elephant in the room: NoSQL.

To put it simply, a NoSQL database is a database that does not use SQL (Structured Query Language). Each NoSQL typically has its own way of writing and executing queries. For example, MongoDB uses MQL (MongoDB Query Language) and ElasticSearch simply has a JSON API.

While most relational databases are fairly similar, NoSQL databases tend to be fairly unique and are used for more niche purposes. Some of the main differences between a SQL and NoSQL database are:

1. NoSQL databases are usually non-relational, SQL databases are usually relational (we'll talk more about what this means later).
2. SQL databases usually have a defined schema, NoSQL databases usually have dynamic schema.
3. SQL databases are table-based, NoSQL databases have a variety of different storage methods, such as document, key-value, graph, wide-column, and more.

Types of NoSQL databases

• Document Database
• Key-Value Store
• Wide-Column
• Graph

A few of the most popular NoSQL databases are:

• MongoDB
• Cassandra
• CouchDB
• DynamoDB
• ElasticSearch

Comparing SQL Databases

Let's dive deeper and talk about some of the popular SQL Databases and what makes them different from one another. Some of the most popular SQL Databases right now are:

• PostgreSQL
• MySQL
• Microsoft SQL Server
• SQLite
• And many others

Source: db-engines.com

While all of these Databases use SQL, each database defines specific rules, practices, and strategies that separate them from their competitors.

SQLite vs PostgreSQL

Personally, SQLite and PostgreSQL are my favorites from the list above. Postgres is a very powerful, open-source, production-ready SQL database. SQLite is a lightweight, embeddable, open-source database. I usually choose one of these technologies if I'm doing SQL work.

SQLite is a serverless database management system (DBMS) that has the ability to run within applications, whereas PostgreSQL uses a Client-Server model and requires a server to be installed and listening on a network, similar to an HTTP server.

See a full comparison here.

Again, in this course we will be working with SQLite, a lightweight and simple database. For most backend web servers, PostgreSQL is a more production-ready option, but SQLite is great for learning and for small systems.

Chapter 2: SQL Tables

The CREATE TABLE statement is used to create a new table in a database.

How to use the CREATE TABLE statement

To create a table, use the CREATE TABLE statement followed by the name of the table and the fields you want in the table.

CREATE TABLE employees (id INTEGER, name TEXT, age INTEGER, is_manager BOOLEAN, salary INTEGER);

Each field name is followed by its datatype. We'll get to data types in a minute.

It's also acceptable and common to break up the CREATE TABLE statement with some whitespace like this:

CREATE TABLE employees(
    id INTEGER,
    name TEXT,
    age INTEGER,
    is_manager BOOLEAN,
    salary INTEGER
);

How to Alter Tables

We often need to alter our database schema without deleting it and re-creating it. Imagine if Twitter deleted its database each time it needed to add a feature, that would be a disaster! Your account and all your tweets would be wiped out on a daily basis.

Instead, we can use use the ALTER TABLE statement to make changes in place without deleting any data.

How to use ALTER TABLE

With SQLite an ALTER TABLE statement allows you to:

1. Rename a table or column, which you can do like this:

ALTER TABLE employees
RENAME TO contractors;

ALTER TABLE contractors
RENAME COLUMN salary TO invoice;

2. ADD or DROP a column, which you can do like this:

ALTER TABLE contractors
ADD COLUMN job_title TEXT;

ALTER TABLE contractors
DROP COLUMN is_manager;

Intro to Migrations

A database migration is a set of changes to a relational database. In fact, the ALTER TABLE statements we did in the last exercise were examples of migrations.

Migrations are helpful when transitioning from one state to another, fixing mistakes, or adapting a database to changes.

Good migrations are small, incremental and ideally reversible changes to a database. As you can imagine, when working with large databases, making changes can be scary. We have to be careful when writing database migrations so that we don't break any systems that depend on the old database schema.

Example of a bad migration

If a backend server periodically runs a query like SELECT * FROM people, and we execute a database migration that alters the table name from people to users without updating the code, the application will break. It will try to grab data from a table that no longer exists.

A simple solution to this problem would be to deploy new code that uses a new query:

SELECT * FROM users;

And we would deploy that code to production immediately following the migration.

SQL Data Types

SQL as a language can support many different data types. But the datatypes that your database management system (DBMS) supports will vary depending on the specific database you're using.

SQLite only supports the most basic types, and we're using SQLite in this course.

SQLite Data Types

Let's go over the data types supported by SQLite: and how they are stored.

1. NULL - Null value.
2. INTEGER - A signed integer stored in 0,1,2,3,4,6, or 8 bytes.
3. REAL - Floating point value stored as an 64-bit IEEE floating point number.
4. TEXT - Text string stored using database encoding such as UTF-8
5. BLOB - Short for Binary large object and typically used for images, audio or other multimedia.

For example:

CREATE TABLE employees (
    id INTEGER,
    name TEXT,
    age INTEGER,
    is_manager BOOLEAN,
    salary INTEGER
);

Boolean values

It's important to note that SQLite does not have a separate BOOLEAN storage class. Instead, boolean values are stored as integers:

• 0 = false
• 1 = true

It's not actually all that weird – boolean values are just binary bits after all!

SQLite will still let you write your queries using boolean expressions and true/false keywords, but it will convert the booleans to integers under-the-hood.

Chapter 3: Constraints

A constraint is a rule we create on a database that enforces some specific behavior. For example, setting a NOT NULL constraint on a column ensures that the column will not accept NULL values.

If we try to insert a NULL value into a column with the NOT NULL constraint, the insert will fail with an error message. Constraints are extremely useful when we need to ensure that certain kinds of data exist within our database.

NOT NULL constraint

The NOT NULL constraint can be added directly to the CREATE TABLE statement.

CREATE TABLE employees(
    id INTEGER PRIMARY KEY,
    name TEXT UNIQUE,
    title TEXT NOT NULL
);

SQLite limitation

In other dialects of SQL you can ADD CONSTRAINT within an ALTER TABLE statement. SQLite does not support this feature, so when we create our tables we need to make sure we specify all the constraints we want.

Here's a list of SQL Features SQLite does not implement in case you're curious.

Primary Key Constraints

A key defines and protects relationships between tables. A primary key is a special column that uniquely identifies records within a table. Each table can have one, and only one primary key.

Your primary key will almost always be the "id" column

It's very common to have a column named id on each table in a database, and that id is the primary key for that table. No two rows in that table can share an id.

A PRIMARY KEY constraint can be explicitly specified on a column to ensure uniqueness, rejecting any inserts where you attempt to create a duplicate ID.

Foreign Key Constraints

Foreign keys are what makes relational databases relational! Foreign keys define the relationships between tables. Simply put, a FOREIGN KEY is a field in one table that references another table's PRIMARY KEY.

Creating a Foreign Key in SQLite

Creating a FOREIGN KEY in SQLite happens at table creation! After we define the table fields and constraints we add an additional CONSTRAINT where we define the FOREIGN KEY and its REFERENCES.

Here's an example:

CREATE TABLE departments (
    id INTEGER PRIMARY KEY,
    department_name TEXT NOT NULL
);

CREATE TABLE employees (
    id INTEGER PRIMARY KEY,
    name TEXT NOT NULL,
    department_id INTEGER,
    CONSTRAINT fk_departments
    FOREIGN KEY (department_id)
    REFERENCES departments(id)
);

In this example, an employee has a department_id. The department_id must be the same as the id field of a record from the departments table.

Schema

We've used the word schema a few times now, let's talk about what that word means. A database's schema describes how data is organized within it.

Data types, table names, field names, constraints, and the relationships between all of those entities are part of a database's schema.

There is no perfect way to architect a database schema

When designing a database schema there typically isn't a "correct" solution. We do our best to choose a sane set of tables, fields, constraints, etc that will accomplish our project's goals. Like many things in programming, different schema designs come with different tradeoffs.

How do we decide on a sane schema architecture?

One very important decision that needs to be made is to decide which table will store a user's balance! As you can imagine, ensuring our data is accurate when dealing with money is super important. We want to be able to:

• Keep track of a user's current balance
• See the historical balance at any point in the past
• See a log of which transactions changed the balance over time

There are many ways to approach this problem. For our first attempt, let's try the simplest schema that fulfills our project's needs.

Chapter 4: CRUD Operations in SQL

What is CRUD?

CRUD is an acronym that stands for CREATE, READ, UPDATE, and DELETE. These four operations are the bread and butter of nearly every database you will create.

HTTP and CRUD

The CRUD operations correlate nicely with the HTTP methods you may have already learned:

• HTTP POST - CREATE
• HTTP GET - READ
• HTTP PUT - UPDATE
• HTTP DELETE - DELETE

SQL Insert Statement

Tables are pretty useless without data in them. In SQL we can add records to a table using an INSERT INTO statement. When using an INSERT statement we must first specify the table we are inserting the record into, followed by the fields within that table we want to add VALUES to.

Here's an example of an INSERT INTO statement:

INSERT INTO employees(id, name, title)
VALUES (1, 'Allan', 'Engineer');

HTTP CRUD Database lifecycle

It's important to understand how data flows through a typical web application.

1. The front-end processes some data from user input - maybe a form is submitted.
2. The front-end sends that data to the server through an HTTP request - maybe a POST.
3. The server makes a SQL query to it's database to create an associated record - Probably using an INSERT statement.
4. Once the server has processed that the database query was successful, it responds to the front-end with a status code! Hopefully a 200-level code (success)!

Manual Entry

Manually INSERTing every single record in a database would be an extremely time-consuming task! Working with raw SQL as we are now is not super common when designing backend systems.

When working with SQL within a software system, like a backend web application, you'll typically have access to a programming language such as Go or Python.

For example, a backend server written in Go can use string concatenation to dynamically create SQL statements, and that's usually how it's done.

sqlQuery := fmt.Sprintf(`
INSERT INTO users(name, age, country_code)
VALUES ('%s', %v, %s);
`, user.Name, user.Age, user.CountryCode)

SQL Injection

The example above is an oversimplification of what really happens when you access a database using Go code. In essence, it's correct. String interpolation is how production systems access databases. That said, it must be done carefully to not be a security vulnerability. We'll talk more about that later!

Count

We can use a SELECT statement to get a count of the records within a table. This can be very useful when we need to know how many records there are, but we don't particularly care what's in them.

Here's an example in SQLite:

SELECT count(*) from employees;

The * in this case refers to a column name. We don't care about the count of a specific column - we want to know the number of total records so we can use the wildcard (*).

HTTP CRUD database lifecycle

We talked about how a "create" operation flows through a web application. Let's talk about a "read".

Let's talk through an example. Our product manager wants to show profile data on a user's settings page. Here's how we could engineer that feature request:

1. First, the front-end webpage loads.
2. The front-end sends an HTTP GET request to a /users endpoint on the back-end server.
3. The server receives the request.
4. The server uses a SELECT statement to retrieve the user's record from the users table in the database.
5. The server converts the row of SQL data into a JSON object and sends it back to the front-end.

WHERE clause

In order to keep learning about CRUD operations in SQL, we need to learn how to make the instructions we send to the database more specific. SQL accepts a WHERE statement within a query that allows us to be very specific with our instructions.

If we were unable to specify the specific record we wanted to READ, UPDATE, or DELETE making queries to a database would be very frustrating, and very inefficient.

Using a WHERE clause

Say we had over 9000 records in our users table. We often want to look at specific user data within that table without retrieving all the other records in the table. We can use a SELECT statement followed by a WHERE clause to specify which records to retrieve. The SELECT statement stays the same, we just add the WHERE clause to the end of the SELECT.

Here's an example:

SELECT name FROM users WHERE power_level >= 9000;

This will select only the name field of any user within the users table WHERE the power_level field is greater than or equal to 9000.

Finding NULL values

You can use a WHERE clause to filter values by whether or not they're NULL.

IS NULL

SELECT name FROM users WHERE first_name IS NULL;

IS NOT NULL

SELECT name FROM users WHERE first_name IS NOT NULL;

DELETE

When a user deletes their account on Twitter, or deletes a comment on a YouTube video, that data needs to be removed from its respective database.

DELETE statement

A DELETE statement removes a record from a table that match the WHERE clause. As an example:

DELETE from employees
    WHERE id = 251;

This DELETE statement removes all records from the employees table that have an id of 251!

The danger of deleting data

Deleting data can be a dangerous operation. Once removed, data can be really hard if not impossible to restore! Let's talk about a couple of common ways back-end engineers protect against losing valuable customer data.

Strategy 1 - Backups

If you're using a cloud-service like GCP's Cloud SQL or AWS's RDS you should always turn on automated backups. They take an automatic snapshot of your entire database on some interval, and keep it around for some length of time.

For example, the Boot.dev database has a backup snapshot taken daily and we retain those backups for 30 days. If I ever accidentally run a query that deletes valuable data, I can restore it from the backup.

You should have a backup strategy for production databases.

Strategy 2 - Soft deletes

A "soft delete" is when you don't actually delete data from your database, but instead just "mark" the data as deleted.

For example, you might set a deleted_at date on the row you want to delete. Then, in your queries you ignore anything that has a deleted_at date set. The idea is that this allows your application to behave as if it's deleting data, but you can always go back and restore any data that's been removed.

You should probably only soft-delete if you have a specific reason to do so. Automated backups should be "good enough" for most applications that are just interested in protecting against developer mistakes.

Update query in SQL

Whenever you update your profile picture or change your password online, you are changing the data in a field on a table in a database. Imagine if every time you accidentally messed up a Tweet on Twitter you had to delete the entire tweet and post a new one instead of just editing it...

...Well, that's a bad example.

Update statement

The UPDATE statement in SQL allows us to update the fields of a record. We can even update many records depending on how we write the statement.

An UPDATE statement specifies the table that needs to be updated, followed by the fields and their new values by using the SET keyword. Lastly a WHERE clause indicates the record(s) to update.

UPDATE employees
SET job_title = 'Backend Engineer', salary = 150000
WHERE id = 251;

Object-Relational Mapping (ORMs)

An Object-Relational Mapping or an ORM for short, is a tool that allows you to perform CRUD operations on a database using a traditional programming language. These typically come in the form of a library or framework that you would use in your backend code.

The primary benefit an ORM provides is that it maps your database records to in-memory objects. For example, in Go we might have a struct that we use in our code:

type User struct {
    ID int
    Name string
    IsAdmin bool
}

This struct definition conveniently represents a database table called users, and an instance of the struct represents a row in the table.

Example: Using an ORM

Using an ORM we might be able to write simple code like this:

user := User{
    ID: 10,
    Name: "Lane",
    IsAdmin: false,
}

// generates a SQL statement and runs it,
// creating a new record in the users table
db.Create(user)

Example: Using straight SQL

Using straight SQL we might have to do something a bit more manual:

user := User{
    ID: 10,
    Name: "Lane",
    IsAdmin: false,
}

db.Exec("INSERT INTO users (id, name, is_admin) VALUES (?, ?, ?);",
    user.ID, user.Name, user.IsAdmin)

Should you use an ORM?

That depends – an ORM typically trades simplicity for control.

Using straight SQL you can take full advantage of the power of the SQL language. Using an ORM, you're limited by whatever functionality the ORM has.

If you run into issues with a specific query, it can be harder to debug with an ORM because you have to dig through the framework's code and documentation to figure out how the underlying queries are being generated.

I recommend doing projects both ways so that you can learn about the tradeoffs. At the end of the day, when you're working on a team of developers, it will be a team decision.

Chapter 5: Basic SQL Queries

How to use the AS Clause in SQL

Sometimes we need to structure the data we return from our queries in a specific way. An AS clause allows us to "alias" a piece of data in our query. The alias only exists for the duration of the query.

AS keyword

The following queries return the same data:

SELECT employee_id AS id, employee_name AS name
FROM employees;

and:

SELECT employee_id, employee_name
FROM employees;

The difference is that the results from the aliased query would have column names id and name instead of employee_id and employee_name.

SQL Functions

At the end of the day, SQL is a programming language, and it's one that supports functions. We can use functions and aliases to calculate new columns in a query. This is similar to how you might use formulas in Excel.

IIF function

In SQLite, the IIF function works like a ternary. For example:

IIF(carA > carB, "Car a is bigger", "Car b is bigger")

If a is greater than b, this statement evaluates to the string "Car a is bigger". Otherwise, it evaluates to "Car b is bigger".

Here's how we can use IIF() and a directive alias to add a new calculated column to our result set:

SELECT quantity,
    IIF(quantity < 10, "Order more", "In Stock") AS directive
    from products

How to Use BETWEEN with WHERE

We can check if certain values are between two numbers using the WHERE clause in an intuitive way. The WHERE clause doesn't always have to be used to specify specific id's or values. We can also use it to help narrow down our result set. Here's an example:

SELECT employee_name, salary
FROM employees
WHERE salary BETWEEN 30000 and 60000;

This query returns all the employees name and salary fields for any rows where the salary is BETWEEN 30,000 and 60,000. We can also query results that are NOT BETWEEN two specified values.

SELECT product_name, quantity
FROM products
WHERE quantity NOT BETWEEN 20 and 100;

This query returns all the product names where the quantity was not between 20 and 100. We can use conditionals to make the results of our query as specific as we need them to be.

How to return distinct values

Sometimes we want to retrieve records from a table without getting back any duplicates.

For example, we may want to know all the different companies our employees have worked at previously, but we don't want to see the same company multiple times in the report.

SELECT DISTINCT

SQL offers us the DISTINCT keyword that removes duplicate records from the resulting query.

SELECT DISTINCT previous_company
    FROM employees;

This only returns one row for each unique previous_company value.

Logical Operators

We often need to use multiple conditions to retrieve the exact information we want. We can begin to structure much more complex queries by using multiple conditions together to narrow down the search results of our query.

The logical AND operator can be used to narrow down our result sets even more.

AND operator

SELECT product_name, quantity, shipment_status
    FROM products
    WHERE shipment_status = 'pending'
    AND quantity BETWEEN 0 and 10;

This only retrieves records where both the shipment_status is "pending" AND the quantity is between 0 and 10.

Equality operators

All of the following operators are supported in SQL. The = is the main one to watch out for, it's not == like in many other languages.

• =
• <
• >
• <=
• >=

For example, in Python you might compare two values like this:

if name == "age"

Whereas in SQL you would do:

WHERE name = "age"

OR operator

As you've probably guessed, if the logical AND operator is supported, the OR operator is probably supported as well.

SELECT product_name, quantity, shipment_status
    FROM products
    WHERE shipment_status = 'out of stock'
    OR quantity BETWEEN 10 and 100;

This query retrieves records where either the shipment_status condition OR the quantity condition are met.

Order of operations matter when using these operators.

You can group logical operations with parentheses to specify the order of operations.

(this AND that) OR the_other

The IN operator

Another variation to the WHERE clause we can utilize is the IN operator. IN returns true or false if the first operand matches any of the values in the second operand. The IN operator is a shorthand for multiple OR conditions.

These two queries are equivalent:

SELECT product_name, shipment_status
    FROM products
    WHERE shipment_status IN ('shipped', 'preparing', 'out of stock');

SELECT product_name, shipment_status
    FROM products
    WHERE shipment_status = 'shipped'
        OR shipment_status = 'preparing'
        OR shipment_status = 'out of stock';

Hopefully, you're starting to see how querying specific data using fine-tuned SQL clauses helps reveal important insights. The larger a table becomes the harder it becomes to analyze without proper queries.

The LIKE keyword

Sometimes we don't have the luxury of knowing exactly what it is we need to query. Have you ever wanted to look up a song or a video but you only remember part of the name? SQL provides us an option for when we're in situations LIKE this.

The LIKE keyword allows for the use of the % and _ wildcard operators. Let's focus on % first.

% Operator

The % operator will match zero or more characters. We can use this operator within our query string to find more than just exact matches depending on where we place it.

Here are some examples that show how these work:

Product starts with "banana":

SELECT * FROM products
WHERE product_name LIKE 'banana%';

Product ends with "banana":

SELECT * from products
WHERE product_name LIKE '%banana';

Product contains "banana":

SELECT * from products
WHERE product_name LIKE '%banana%';

Underscore Operator

As discussed, the % wildcard operator matches zero or more characters. Meanwhile, the _ wildcard operator only matches a single character.

SELECT * FROM products
    WHERE product_name LIKE '_oot';

The query above matches products like:

• boot
• root
• foot

SELECT * FROM products
    WHERE product_name LIKE '__oot';

The query above matches products like:

• shoot
• groot

Chapter 6: How to Structure Return Data in SQL

The LIMIT keyword

Sometimes we don't want to retrieve every record from a table. For example, it's common for a production database table to have millions of rows, and SELECTing all of them might crash your system. This is where the LIMIT keyword enters the chat.

The LIMIT keyword can be used at the end of a select statement to reduce the number of records returned.

SELECT * FROM products
    WHERE product_name LIKE '%berry%'
    LIMIT 50;

The query above retrieves all the records from the products table where the name contains the word berry. If we ran this query on the Facebook database, it would almost certainly return a lot of records.

The LIMIT statement only allows the database to return up to 50 records matching the query. This means that if there aren't that many records matching the query, the LIMIT statement will not have an effect.

The SQL ORDER BY keyword

SQL also offers us the ability to sort the results of a query using ORDER BY. By default, the ORDER BY keyword sorts records by the given field in ascending order, or ASC for short. However, ORDER BY does support descending order as well with the keyword DESC.

Examples

This query returns the name, price, and quantity fields from the products table sorted by price in ascending order:

SELECT name, price, quantity FROM products
    ORDER BY price;

This query returns the name, price, and quantity of the products ordered by the quantity in descending order:

SELECT name, price, quantity FROM products
    ORDER BY quantity desc;

Order By and Limit

When using both ORDER BY and LIMIT, the ORDER BY clause must come first.

Chapter 7: How to Perform Aggregations in SQL

An "aggregation" is a single value that's derived by combining several other values. We performed an aggregation earlier when we used the count statement to count the number of records in a table.

Why use aggregations?

Data stored in a database should generally be stored raw. When we need to calculate some additional data from the raw data, we can use an aggregation.

Take the following count aggregation as an example:

SELECT COUNT(*)
FROM products
WHERE quantity = 0;

This query returns the number of products that have a quantity of 0. We could store a count of the products in a separate database table, and increment/decrement it whenever we make changes to the products table - but that would be redundant.

It's much simpler to store the products in a single place (we call this a single source of truth) and run an aggregation when we need to derive additional information from the raw data.

The SUM function

The sum aggregation function returns the sum of a set of values.

For example, the query below returns a single record containing a single field. The returned value is equal to the total salary being collected by all of the employees in the employees table.

SELECT sum(salary)
FROM employees;

Which returns:

The MAX function

As you may expect, the max function retrieves the largest value from a set of values. For example:

SELECT max(price)
FROM products

This query looks through all of the prices in the products table and returns the price with the largest price value. Remember it only returns the price, not the rest of the record. You always need to specify each field you want a query to return.

A note on schema

• The sender_id will be present for any transactions where the user in question (user_id) is receiving money (from the sender).
• The recipient_id will be present for any transactions where the user in question (user_id) is sending money (to the recipient).

In other words, a transaction can only have a sender_id or a recipient_id - not both. The presence of one or the other indicates whether money is going into or out of the user's account.

This user_id, recipient_id, sender_id schema we've designed is only one way to design a transactions database - there are other valid ways to do it. It's the one we're using, and later we'll talk more about the tradeoffs in different database design options.

The MIN function

The min function works the same as the max function but finds the lowest value instead of the highest value.

SELECT product_name, min(price)
from products;

This query returns the product_name and the price fields of the record with the lowest price.

The GROUP BY clause

There are times we need to group data based on specific values.

SQL offers the GROUP BY clause which can group rows that have similar values into "summary" rows. It returns one row for each group. The interesting part is that each group can have an aggregate function applied to it that operates only on the grouped data.

Example of GROUP BY

Imagine that we have a database with songs and albums, and we want to see how many songs are on each album. We can use a query like this:

SELECT album_id, count(song_id)
FROM songs
GROUP BY album_id;

This query retrieves a count of all the songs on each album. One record is returned per album, and they each have their own count.

The AVG() function

Just like we may want to find the minimum or maximum values within a dataset, sometimes we need to know the average!

SQL offers us the AVG() function. Similar to MAX(), AVG() calculates the average of all non-NULL values.

select song_name, avg(song_length)
from songs

This query returns the average song_length in the songs table.

The HAVING clause

When we need to filter the results of a GROUP BY query even further, we can use the HAVING clause. The HAVING clause specifies a search condition for a group.

The HAVING clause is similar to the WHERE clause, but it operates on groups after they've been grouped, rather than rows before they've been grouped.

SELECT album_id, count(id) as count
FROM songs
GROUP BY album_id
HAVING count > 5;

This query returns the album_id and count of its songs, but only for albums with more than 5 songs.

HAVING vs WHERE in SQL

It's fairly common for developers to get confused about the difference between the HAVING and the WHERE clauses - they're pretty similar after all.

The difference is fairly simple in actuality:

• A WHERE condition is applied to all the data in a query before it's grouped by a GROUP BY clause.
• A HAVING condition is only applied to the grouped rows that are returned after a GROUP BY is applied.

This means that if you want to filter on the result of an aggregation, you need to use HAVING. If you want to filter on a value that's present in the raw data, you should use a simple WHERE clause.

The ROUND function

Sometimes we need to round some numbers, particularly when working with the results of an aggregation. We can use the ROUND() function to get the job done.

The SQL round() function allows you to specify both the value you wish to round and the precision to which you wish to round it:

round(value, precision)

If no precision is given, SQL will round the value to the nearest whole value:

select song_name, round(avg(song_length), 1)
from songs

This query returns the average song_length from the songs table, rounded to a single decimal point.

Chapter 8: SQL Subqueries

Subqueries

Sometimes a single query is not enough to retrieve the specific records we need.

It is possible to run a query on the result set of another query - a query within a query! This is called "query-ception"... erm... I mean a "subquery".

Subqueries can be very useful in a number of situations when trying to retrieve specific data that wouldn't be accessible by simply querying a single table.

How to retreive data from multiple tables

Here is an example of a subquery:

SELECT id, song_name, artist_id
FROM songs
WHERE artist_id IN (
    SELECT id
    FROM artists
    WHERE artist_name LIKE 'Rick%'
);

In this hypothetical database, the query above selects all of the song_ids, song_names, and artist_ids from the songs table that are written by artists whose name starts with "Rick". Notice that the subquery allows us to use information from a different table - in this case the artists table.

Subquery syntax

The only syntax unique to a subquery is the parentheses surrounding the nested query. The IN operator could be different, for example, we could use the = operator if we expect a single value to be returned.

Here's an example:

SELECT id, song_name, artist_id
FROM songs
WHERE artist_id IN (
    SELECT id
    FROM artists
    WHERE artist_name LIKE 'Rick%'
);

No tables necessary

When working on a back-end application, this doesn't come up often, but it's important to remember that SQL is a full programming language. We usually use it to interact with data stored in tables, but it's quite flexible and powerful.

For example, you can SELECT information that's simply calculated, with no tables necessary.

SELECT 5 + 10 as sum;

Chapter 9: Database Normalization

Table Relationships

Relational databases are powerful because of the relationships between the tables. These relationships help us to keep our databases clean and efficient.

A relationship between tables assumes that one of these tables has a foreign key that references the primary key of another table.

@youtube

Types of Relationships

There are 3 primary types of relationships in a relational database:

1. One-to-one
2. One-to-many
3. Many-to-many

One-to-one

A one-to-one relationship most often manifests as a field or set of fields on a row in a table. For example, a user will have exactly one password.

Settings fields might be another example of a one-to-one relationship. A user will have exactly one email_preference and exactly one birthday.

One to many

When talking about the relationships between tables, a one-to-many relationship is probably the most commonly used relationship.

A one-to-many relationship occurs when a single record in one table is related to potentially many records in another table.

Note that the one->many relation only goes one way, a record in the second table can not be related to multiple records in the first table!

Examples of one-to-many relationships

• A customers table and a orders table. Each customer has 0, 1, or many orders that they've placed.
• A users table and a transactions table. Each user has 0, 1, or many transactions that taken part in.

Many to many

A many-to-many relationship occurs when multiple records in one table can be related to multiple records in another table.

Examples of many-to-many relationships

• A products table and a suppliers table - Products may have 0 to many suppliers, and suppliers can supply 0 to many products.
• A classes table and a students table - Students can take potentially many classes and classes can have many students enrolled.

Joining tables

Joining tables helps define many-to-many relationships between data in a database. As an example, when defining the relationship above between products and suppliers, we would define a joining table called products_suppliers that contains the primary keys from the tables to be joined.

Then, when we want to see if a supplier supplies a specific product, we can look in the joining table to see if the ids share a row.

Unique constraints across 2 fields

When enforcing specific schema constraints we may need to enforce the UNIQUE constraint across two different fields.

CREATE TABLE product_suppliers (
  product_id INTEGER,
  supplier_id INTEGER,
  UNIQUE(product_id, supplier_id)
);

This ensures that we can have multiple rows with the same product_id or supplier_id, but we can't have two rows where both the product_id and supplier_id are the same.

Database normalization

Database normalization is a method for structuring your database schema in a way that helps:

• Improve data integrity
• Reduce data redundancy

What is data integrity?

"Data integrity" refers to the accuracy and consistency of data. For example, if a user's age is stored in a database, rather than their birthday, that data becomes incorrect automatically with the passage of time.

It would be better to store a birthday and calculate the age as needed.

What is data redundancy?

"Data redundancy" occurs when the same piece of data is stored in multiple places. For example: saving the same file multiple times to different hard drives.

Data redundancy can be problematic, especially when data in one place is changed such that the data is no longer consistent across all copies of that data.

Normal Forms

The creator of "database normalization", Edgar F. Codd, described different "normal forms" a database can adhere to. We'll talk about the most common ones.

• First normal form (1NF)
• Second normal form (2NF)
• Third normal form (3NF)
• Boyce-Codd normal form (BCNF)

In short, 1st normal form is the least "normalized" form, and Boyce-Codd is the most "normalized" form.

The more normalized a database, the better its data integrity, and the less duplicate data you'll have.

In the context of normal forms, "primary key" means something a bit different

In the context of database normalization, we're going to use the term "primary key" slightly differently. When we're talking about SQLite, a "primary key" is a single column that uniquely identifies a row.

When we're talking more generally about data normalization, the term "primary key" means the collection of columns that uniquely identify a row. That can be a single column, but it can actually be any number of columns. A primary key is the minimum number of columns needed to uniquely identify a row in a table.

If you think back to the many-to-many joining table product_suppliers, that table's "primary key" was actually a combination of the 2 ids, product_id and supplier_id:

CREATE TABLE product_suppliers (
    product_id INTEGER,
    supplier_id INTEGER,
    UNIQUE(product_id, supplier_id)
);

1st Normal Form (1NF)

To be compliant with first normal form, a database table simply needs to follow 2 rules:

• It must have a unique primary key.
• A cell can't have a nested table as its value (depending on the database you're using, this may not even be possible)

Example of NOT 1st normal form

This table does not adhere to 1NF. It has two identical rows, so there isn't a unique primary key for each row.

Example of 1st normal form

The simplest way (but not the only way) to get into first normal form is to add a unique id column.

It's worth noting that if you create a "primary key" by ensuring that two columns are always "unique together" that works too.

You should almost never design a table that doesn't adhere to 1NF

First normal form is simply a good idea. I've never built a database schema where each table isn't at least in first normal form.

2nd Normal Form (2NF)

A table in second normal form follows all the rules of 1st normal form, and one additional rule:

• All columns that are not part of the primary key are dependent on the entire primary key, and not just one of the columns in the primary key.

Example of 1st NF, but not 2nd NF

In this table, the primary key is a combination of first_name + last_name.

This table does not adhere to 2NF. The first_initial column is entirely dependent on the first_name column, rendering it redundant.

Example of 2nd normal form

One way to convert the table above to 2NF is to add a new table that maps a first_name directly to its first_initial. This removes any duplicates:

2NF is usually a good idea

You should probably default to keeping your tables in second normal form. That said, there are good reasons to deviate from it, particularly for performance reasons. The reason being that when you have query a second table to get additional data it can take a bit longer.

My rule of thumb is:

Optimize for data integrity and data de-duplication first. If you have speed issues, de-normalize accordingly.

3rd Normal Form (3NF)

A table in 3rd normal form follows all the rules of 2nd normal form, and one additional rule:

• All columns that aren't part of the primary are dependent solely on the primary key.

Notice that this is only slightly different from second normal form. In second normal form we can't have a column completely dependent on a part of the primary key, and in third normal form we can't have a column that is entirely dependent on anything that isn't the entire primary key.

Example of 2nd NF, but not 3rd NF

In this table, the primary key is simply the id column.

This table is in 2nd normal form because first_initial is not dependent on a part of the primary key. However, because it is dependent on the name column it doesn't adhere to 3rd normal form.

Example of 3rd normal form

The way to convert the table above to 3NF is to add a new table that maps a name directly to its first_initial. Notice how similar this solution is to 2NF.

3NF is usually a good idea

The same exact rule of thumb applies to the second and third normal forms.

Optimize for data integrity and data de-duplication first by adhering to 3NF. If you have speed issues, de-normalize accordingly.

Remember the IIF function and the AS clause.

Boyce-Codd Normal Form (BCNF)

A table in Boyce-Codd normal form (created by Raymond F Boyce and Edgar F Codd) follows all the rules of 3rd normal form, plus one additional rule:

• A column that's part of a primary key can not be entirely dependent on a column that's not part of that primary key.

This only comes into play when there are multiple possible primary key combinations that overlap. Another name for this is "overlapping candidate keys".

Only in rare cases does a table in third normal form not meet the requirements of Boyce-Codd normal form.

Example of 3rd NF, but not Boyce-Codd NF

The interesting thing here is that there are 3 possible primary keys:

• release_year + sales
• release_date + sales
• name

This means that by definition this table is in 2nd and 3rd normal form because those forms only restrict how dependent a column that is not part of a primary key can be.

This table is not in Boyce-Codd's normal form because release_year is entirely dependent on release_date.

Example of Boyce-Codd normal form

The easiest way to fix the table in our example is to simply remove the duplicate data from release_date. Let's make that column release_day_and_month.

BCNF is usually a good idea

The same exact rule of thumb applies to the 2nd, 3rd and Boyce-Codd normal forms. That said, it's unlikely you'll see BCNF-specific issues in practice.

Optimize for data integrity and data de-duplication first by adhering to Boyce-Codd normal form. If you have speed issues, de-normalize accordingly.

Normalization Review

In my opinion, the exact definitions of 1st, 2nd, 3rd and Boyce-Codd normal forms simply are not all that important in your work as a back-end developer.

However, what is important is to understand the basic principles of data integrity and data redundancy that the normal forms teach us.

Let's go over some rules of thumb that you should commit to memory - they'll serve you well when you design databases and even just in coding interviews.

Rules of thumb for database design

1. Every table should always have a unique identifier (primary key)
2. 90% of the time, that unique identifier will be a single column named id
3. Avoid duplicate data
4. Avoid storing data that is completely dependent on other data. Instead, compute it on the fly when you need it.
5. Keep your schema as simple as you can. Optimize for a normalized database first. Only denormalize for speed's sake when you start to run into performance problems.

We'll talk more about speed optimization in a later chapter.

Chapter 10: How to Join Tables in SQL

Joins are one of the most important features that SQL offers. Joins allow us to make use of the relationships we have set up between our tables. In short, joins allow us to query multiple tables at the same time.

INNER JOIN

The simplest and most common type of join in SQL is the INNER JOIN. By default, a JOIN command is an INNER JOIN.

An INNER JOIN returns all of the records in table_a that have matching records in table_b, as demonstrated by the following Venn diagram.

The ON clause

In order to perform a join, we need to tell the database which fields should be "matched up". The ON clause is used to specify these columns to join.

SELECT *
FROM employees
INNER JOIN departments 
ON employees.department_id = departments.id;

The query above returns all the fields from both tables. The INNER keyword doesn't have anything to do with the number of columns returned - it only affects the number of rows returned.

Namespacing on Tables

When working with multiple tables, you can specify which table a field exists on using a .. For example:

table_name.column_name

SELECT students.name, classes.name
FROM students
INNER JOIN classes on classes.class_id = students.class_id;

The above query returns the name field from the students table and the name field from the classes table.

LEFT JOIN

A LEFT JOIN will return every record from table_a regardless of whether or not any of those records have a match in table_b. A left join will also return any matching records from table_b.

Here is a Venn diagram to help visualize the effect of a LEFT JOIN.

A small trick you can do to make writing the SQL query easier is define an alias for each table. Here's an example:

SELECT e.name, d.name
FROM employees e
LEFT JOIN departments d
ON e.department_id = d.id;

Notice the simple alias declarations e and d for employees and departments respectively.

Some developers do this to make their queries less verbose. That said, I personally hate it because single-letter variables are harder to understand the meaning of.

RIGHT JOIN

A RIGHT JOIN is, as you may expect, the opposite of a LEFT JOIN. It returns all records from table_b regardless of matches, and all matching records between the two tables.

SQLite Restriction

SQLite does not support right joins, but many dialects of SQL do. If you think about it, a RIGHT JOIN is just a LEFT JOIN with the order of the tables switched, so it's not a big deal that SQLite doesn't support the syntax.

FULL JOIN

A FULL JOIN combines the result set of the LEFT JOIN and RIGHT JOIN commands. It returns all records from both from table_a and table_b regardless of whether or not they have matches.

SQLite

Like RIGHT JOINs, SQLite doesn't support FULL JOINs but they are still important to know.

Chapter 11: Database Performance

SQL Indexes

An index is an in-memory structure that ensures that queries we run on a database are performant, that is to say, they run quickly.

If you've learned about data structures, most database indexes are just binary trees. The binary tree can be stored in ram as well as on disk, and it makes it easy to lookup the location of an entire row.

PRIMARY KEY columns are indexed by default, ensuring you can look up a row by its id very quickly. But if you have other columns that you want to be able to do quick lookups on, you'll need to index them.

CREATE INDEX

CREATE INDEX index_name on table_name (column_name);

It's fairly common to name an index after the column it's created on with a suffix of _idx.

Index Review

As we discussed, an index is a data structure that can perform quick lookups. By indexing a column, we create a new in-memory structure, usually a binary-tree, where the values in the indexed column are sorted into the tree to keep lookups fast.

In terms of Big-O complexity, a binary tree index ensures that lookups are O(log(n)).

Shouldn't we index everything? We can make the database ultra-fast!

While indexes make specific kinds of lookups much faster, they also add performance overhead - they can slow down a database in other ways.

Think about it: if you index every column, you could have hundreds of binary trees in memory. That needlessly bloats the memory usage of your database. It also means that each time you insert a record, that record needs to be added to many trees - slowing down your insert speed.

The rule of thumb is simple:

Add an index to columns you know you'll be doing frequent lookups on. Leave everything else un-indexed. You can always add indexes later.

Multi-column indexes

Multi-column indexes are useful for the exact reason you might think - they speed up lookups that depend on multiple columns.

CREATE INDEX

CREATE INDEX first_name_last_name_age_idx
ON users (first_name, last_name, age);

A multi-column index is sorted by the first column first, the second column next, and so forth. A lookup on only the first column in a multi-column index gets almost all of the performance improvements that it would get from its own single-column index. But lookups on only the second or third column will have very degraded performance.

Rule of thumb

Unless you have specific reasons to do something special, only add multi-column indexes if you're doing frequent lookups on a specific combination of columns.

Denormalizing for speed

I left you with a cliffhanger in the "normalization" chapter. As it turns out, data integrity and deduplication come at a cost, and that cost is usually speed.

Joining tables together, using subqueries, performing aggregations, and running post-hoc calculations all take time. At very large scales these advanced techniques can actually take a huge performance toll on an application - sometimes grinding the database server to a halt.

Storing duplicate information can drastically speed up an application that needs to look it up in different ways. For example, if you store a user's country information right on their user record, no expensive join is required to load their profile page.

That said, denormalize at your own risk. Denormalizing a database incurs a large risk of inaccurate and buggy data.

In my opinion, it should be used as a kind of "last resort" in the name of speed.

SQL Injection

SQL is a very common way hackers attempt to cause damage or breach a database. One of my favorite XKCD comics of all time demonstrates the problem:

The joke here is that if someone was using this query:

INSERT INTO students(name) VALUES (?);

And the "name" of a student was 'Robert'); DROP TABLE students;-- then the resulting SQL query would look like this:

INSERT INTO students(name) VALUES ('Robert'); DROP TABLE students;--)

As you can see, this is actually 2 queries! The first one inserts "Robert" into the database, and the second one deletes the students table!

How do we protect against SQL injection?

You need to be aware of SQL injection attacks, but to be honest the solution these days is to simply use a modern SQL library that sanitizes SQL inputs. We don't often need to sanitize inputs by hand at the application level anymore.

For example, the Go standard library's SQL packages automatically protects your inputs against SQL attacks if you use it properly. In short, don't interpolate user input into raw strings yourself - make sure your database library has a way to sanitize inputs, and pass it those raw values.

Congratulations on making it to the end!

If you're interested in doing the interactive coding assignments and quizzes for this course, you can check out the Learn SQL Course course over on Boot.dev

This course is a part of my full back-end developer career path, made up of other courses and projects if you're interested in checking those out.

If you want to see the other content I'm creating related to web development, check out some of my links below:

Lane's Podcast: Backend Banter Lane on Twitter Lane on YouTube

If you're looking to learn a new programming language, Go is a great choice. It's fast, lightweight, has an amazing open source community, and is actually quite easy to get started with.

This a completely free text-based handbook. If you want to get started, just scroll down and start reading! That said, there are two other options for following along.

1. Try the interactive version of this Golang course on Boot.dev, complete with coding challenges and projects

2. Watch the video walkthrough of this course on FreeCodeCamp's YouTube channel (embedded below)

Table of Contents

1. Why Learn Go?

2. How to Compile Go Code

3. Variables in Go

4. Functions in Go

5. Structs in Go

6. Interfaces in Go

7. Errors in Go

8. Loops in Go

9. Arrays and Slices in Go

10. Maps in Go

11. Advanced Functions in Go

12. Pointers in Go

13. Local Development in Go

14. Channels in Go

15. Mutexes in Go

16. Generics in Go

Chapter 1 – Why Learn Go?

Go is fast, simple, and productive. Go is one of the fastest programming languages, beating JavaScript, Python, and Ruby handily in most benchmarks.

But, Go code doesn't run quite as fast as its compiled Rust and C counterparts. That said, it compiles much faster than they do, which makes the developer experience super productive. Unfortunately, there are no swordfights on Go teams...

Comic by xkcd

Go has been growing like crazy in the backend development industry, so if you're interested in getting a job as a backend dev, Go can be a great choice of technology to add to your tool belt.

How to Download and Install the Go Toolchain

I typically recommend one of two ways:

• Official Download

• Webi Downloader

Make sure to use at least version 1.20. You can verify this after installation by typing:

go version

A note on the structure of a Go program

We'll go over this all later in more detail, but to sate your curiosity for now, here are a few tidbits about the code:

package main

import "fmt"

func main() {
    fmt.Println("hello world")
}

1. package main lets the Go compiler know that we want this code to compile and run as a standalone program, as opposed to being a library that's imported by other programs.

2. import fmt imports the fmt (formatting) package. The formatting package exists in Go's standard library and let's us do things like print text to the console.

3. func main() defines the main function. main is the name of the function that acts as the entry point for a Go program.

Save the code above in a file called main.go , run go build , and then run the resulting executable.

go build -o out
./out

You can also use Bootdev's Go playground to try out all the snippets in this course right from your browser.

Chapter 2 – How to Compile Go Code

What does it mean to be compiled?

Computers need machine code – they don't understand English or even Go. We need to convert our high-level (Go) code into machine language, which is really just a set of instructions that some specific hardware can understand. In your case, your CPU.

The Go compiler's job is to take Go code and produce machine code. On Windows, that would be a .exe file. On Mac or Linux, it would be any executable file.

Computers don't know how to do anything unless we as programmers tell them what to do. Unfortunately, computers don't understand human language. In fact, they don't even understand uncompiled computer programs.

For example, the code:

package main

import "fmt"

func main(){
  fmt.Println("hello world")
}

means nothing to a computer.

Computers need machine code

A computer's CPU only understands its own instruction set, which we call "machine code". Instructions are basic math operations like addition, subtraction, multiplication, and the ability to save data temporarily.

For example, an ARM processor uses the ADD instruction when supplied with the number 0100 in binary.

Go, C, Rust, and so on

Go, C, and Rust are all languages where the code is first converted to machine code by the compiler before it's executed.

Compiled vs Interpreted

Compiled programs can be run without access to the original source code, and without access to a compiler.

For example, when your browser executes the code you write in this course, it doesn't use the original code, just the compiled result. Note how this is different than interpreted languages like Python and JavaScript.

With Python and JavaScript, the code is interpreted at runtime by a separate program known as the "interpreter". Distributing code for users to run can be a pain because they need to have an interpreter installed, and they need access to the original source code.

Examples of compiled languages

• Go

• C

• C++

• Rust

Examples of interpreted languages

• JavaSsript

• Python

• Ruby

Illustration of compiled vs interpreted languages

Go is Strongly Typed

Go enforces strong and static typing, meaning variables can only have a single type. A string variable like "hello world" can not be changed to an int, such as the number 3.

One of the biggest benefits of strong typing is that errors can be caught at "compile time". In other words, bugs are more easily caught ahead of time because they are detected when the code is compiled before it even runs.

Contrast this with most interpreted languages, where the variable types are dynamic. Dynamic typing can lead to subtle bugs that are hard to detect. With interpreted languages, the code must be run (sometimes in production if you are unlucky 😨) to catch syntax and type errors.

As an example, the following code will fail to compile because strings and ints can't be added together:

func main() {
    var username string = "wagslane"
    var password int = 20947382822

    // don't edit below this line
    fmt.Println("Authorization: Basic", username+":"+password)
}

Go programs are lightweight

Go programs are fairly lightweight. Each program includes a small amount of "extra" code that's included in the executable binary. This extra code is called the Go Runtime. One of the purposes of the Go runtime is to cleanup unused memory at runtime.

In other words, the Go compiler includes a small amount of extra logic in every Go program to make it easier for developers to write code that's memory efficient.

As a general rule, Java programs use more memory than comparable Go programs because Go doesn't use an entire virtual machine to run its programs, just a small runtime. The Go runtime is small enough that it is included directly in each Go program's compiled machine code.

As another general rule, Rust and C++ programs use slightly less memory than Go programs because more control is given to the developer to optimize memory usage of the program. The Go runtime just handles it for us automatically.

Chart showing idle memory usage comparison between Java (162MB), Go (.86MB) and Rust (.36MB)

In the chart above, Dexter Darwich compares the memory usage of three very simple programs written in Java, Go, and Rust. As you can see, Go and Rust use very little memory when compared to Java.

Chapter 3 – Variables in Go

Go's basic variable types are:

bool

string

int  int8  int16  int32  int64
uint uint8 uint16 uint32 uint64 uintptr

byte // alias for uint8

rune // alias for int32
     // represents a Unicode code point

float32 float64

complex64 complex128

We talked about strings and ints previously, and those two types should be fairly self-explanatory.

A bool is a boolean variable, meaning it has a value of true or false. The floating point types ( float32 and float64) are used for numbers that are not integers – that is, they have digits to the right of the decimal place, such as 3.14159. The float32 type uses 32 bits of precision, while the float64 type uses 64 bits to be able to more precisely store more digits.

Don't worry too much about the intricacies of the other types for now. We will cover some of them in more detail as we progress.

How to declare a variable

Variables are declared using the var keyword. For example, to declare a variable called number of type int, you would write:

var number int

To declare a variable called pi to be of type float64 with a value of 3.14159, you would write:

var pi float64 = 3.14159

The value of an initialized variable with no assignment will be its zero value.

Short Variable Declaration

Inside a function (even the main function), the := short assignment statement can be used in place of a var declaration. The := operator infers the type of the new variable based on the value.

var empty string

Is the same as:

empty := ""

numCars := 10 // inferred to be an integer

temperature := 0.0 // temperature is inferred to be a floating point value because it has a decimal point

var isFunny = true // isFunny is inferred to be a boolean

Outside of a function (in the global/package scope), every statement begins with a keyword ( var, func, and so on) and so the := construct is not available.

Type Inference

To declare a variable without specifying an explicit type (either by using the := syntax or var = expression syntax), the variable's type is inferred from the value on the right hand side.

When the right hand side of the declaration is typed, the new variable is of that same type:

var i int
j := i // j is also an int

However, when the right hand side is a literal value (an untyped numeric constant like 42 or 3.14), the new variable will be an int, float64, or complex128 depending on its precision:

i := 42           // int
f := 3.14         // float64
g := 0.867 + 0.5i // complex128

Same Line Declarations

We can declare multiple variables on the same line:

mileage, company := 80276, "Tesla"

// is the same as

mileage := 80276
company := "Tesla"

Type Sizes

Ints, uints, floats, and complex numbers all have type sizes.

int  int8  int16  int32  int64 // whole numbers

uint uint8 uint16 uint32 uint64 uintptr // positive whole numbers

float32 float64 // decimal numbers

complex64 complex128 // imaginary numbers (rare)

The size (8, 16, 32, 64, 128, and so on) indicates how many bits in memory will be used to store the variable. The default int and uint types are just aliases that refer to their respective 32 or 64 bit sizes depending on the environment of the user.

The standard sizes that should be used unless you have a specific need are:

• int

• uint

• float64

• complex128

Some types can be converted the following way:

temperatureInt := 88
temperatureFloat := float64(temperatureInt)

Casting a float to an integer in this way truncates the floating point portion.

Which Type Should I Use?

With so many types for what is essentially just a number, developers coming from languages that only have one kind of Number type (like JavaScript) may find the choices daunting.

A problem arises when we have a uint16, and the function we are trying to pass it into takes an int. We're forced to write code riddled with type casts like int(myUint16).

This style of development can be slow and annoying to read. When Go developers stray from the “default” type for any given type family, the code can get messy quickly.

Unless you have a good reason to, stick to the following types:

• bool

• string

• int

• uint

• byte

• rune

• float64

• complex128

Constants

Constants are declared like variables but use the const keyword. Constants can't use the := short declaration syntax.

Constants can be character, string, boolean, or numeric values. They can not be more complex types like slices, maps and structs, which are types I will explain later.

As the name implies, the value of a constant can't be changed after it has been declared.

Constants must be known at compile time. More often than not they will be declared with a static value:

const myInt = 15

However, constants can be computed so long as the computation can happen at compile time. For example, this is valid:

const firstName = "Lane"
const lastName = "Wagner"
const fullName = firstName + " " + lastName

That said, you can't declare a constant that can only be computed at run-time.

How to Format Strings in Go

Go follows the printf tradition from the C language. In my opinion, string formatting/interpolation in Go is currently less elegant than JavaScript and Python.

• fmt.Printf – Prints a formatted string to standard out

• fmt.Sprintf() – Returns the formatted string

Examples

These formatting verbs work with both fmt.Printf and fmt.Sprintf.

%v - Interpolate the default representation

The %v variant prints the Go syntax representation of a value. You can usually use this if you're unsure what else to use. That said, it's better to use the type-specific variant if you can.

s := fmt.Sprintf("I am %v years old", 10)
// I am 10 years old

s := fmt.Sprintf("I am %v years old", "way too many")
// I am way too many years old

%s - Interpolate a string

s := fmt.Sprintf("I am %s years old", "way too many")
// I am way too many years old

%d - Interpolate an integer in decimal form

s := fmt.Sprintf("I am %d years old", 10)
// I am 10 years old

%f - Interpolate a decimal

s := fmt.Sprintf("I am %f years old", 10.523)
// I am 10.523000 years old

// The ".2" rounds the number to 2 decimal places
s := fmt.Sprintf("I am %.2f years old", 10.523)
// I am 10.53 years old

If you're interested in all the formatting options, feel free to take a look at the fmt package's docs here.

Conditionals

if statements in Go don't use parentheses around the condition:

if height > 4 {
    fmt.Println("You are tall enough!")
}

else if and else are supported as you would expect:

if height > 6 {
    fmt.Println("You are super tall!")
} else if height > 4 {
    fmt.Println("You are tall enough!")
} else {
    fmt.Println("You are not tall enough!")
}

The initial statement of an if block

An if conditional can have an "initial" statement. The variable(s) created in the initial statement are only defined within the scope of the if body.

if INITIAL_STATEMENT; CONDITION {
}

This is just some syntactic sugar that Go offers to shorten up code in some cases. For example, instead of writing:

length := getLength(email)
if length < 1 {
    fmt.Println("Email is invalid")
}

We can do:

if length := getLength(email); length < 1 {
    fmt.Println("Email is invalid")
}

Not only is this code a bit shorter, but it also removes length from the parent scope. This is convenient because we don't need it there – we only need access to it while checking a condition.

Chapter 4 – Functions in Go

Functions in Go can take zero or more arguments.

To make Go code easier to read, the variable type comes after the variable name.

For example, the following function:

func sub(x int, y int) int {
  return x-y
}

Accepts two integer parameters and returns another integer.

Here, func sub(x int, y int) int is known as the "function signature".

Multiple Parameters

When multiple arguments are of the same type, the type only needs to be declared after the last one, assuming they are in order.

For example:

func add(x, y int) int {
  return x + y
}

If they are not in order they need to be defined separately.

Function Declaration Syntax

Developers often wonder why the declaration syntax in Go is different from the tradition established in the C family of languages.

C-Style syntax

The C language describes types with an expression including the name to be declared, and states what type that expression will have.

int y;

The code above declares y as an int. In general, the type goes on the left and the expression on the right.

Interestingly, the creators of the Go language agreed that the C-style of declaring types in signatures gets confusing really fast – take a look at this nightmare.

int (*fp)(int (*ff)(int x, int y), int b)

Go-style syntax

Go's declarations are clear, you just read them left to right, just like you would in English.

x int
p *int
a [3]int

It's nice for more complex signatures, as it makes them easier to read.

f func(func(int,int) int, int) int

How to Pass Variables by Value

Variables in Go are passed by value (except for a few data types we haven't covered yet). "Pass by value" means that when a variable is passed into a function, that function receives a copy of the variable. The function is unable to mutate the caller's original data.

func main(){
    x := 5
    increment(x)

    fmt.Println(x)
    // still prints 5,
    // because the increment function
    // received a copy of x
}

func increment(x int){
    x++
}

How to Ignore Return Values

A function can return a value that the caller doesn't care about. We can explicitly ignore variables by using an underscore: _

For example:

func getPoint() (x int, y int) {
  return 3, 4
}

// ignore y value
x, _ := getPoint()

Even though getPoint() returns two values, we can capture the first one and ignore the second.

Why would you ignore a return value?

There could be many reasons. For example, maybe a function called getCircle returns the center point and the radius, but you really only need the radius for your calculation. In that case, you would ignore the center point variable.

This is crucial to understand because the Go compiler will throw an error if you have unused variable declarations in your code, so you need to ignore anything you don't intend to use.

Named Return Values

Return values may be given names, and if they are, then they are treated the same as if they were new variables defined at the top of the function.

Named return values are best thought of as a way to document the purpose of the returned values.

According to the tour of go:

"A return statement without arguments returns the named return values. This is known as a "naked" return. Naked return statements should be used only in short functions. They can harm readability in longer functions."

func getCoords() (x, y int){
  // x and y are initialized with zero values

  return // automatically returns x and y
}

Is the same as:

func getCoords() (int, int){
  var x int
  var y int
  return x, y
}

In the first example, x and y are the return values. At the end of the function, we could simply write return to return the values of those two variables, rather that writing return x,y.

Explicit Returns

Even though a function has named return values, we can still explicitly return values if we want to.

func getCoords() (x, y int){
  return x, y // this is explicit
}

Using this explicit pattern we can even overwrite the return values:

func getCoords() (x, y int){
  return 5, 6 // this is explicit, x and y are NOT returned
}

Otherwise, if we want to return the values defined in the function signature we can just use a naked return (blank return):

func getCoords() (x, y int){
  return // implicitly returns x and y
}

The Benefits of Named Returns

1. Good For Documentation (Understanding)

Named return parameters are great for documenting a function. We know what the function is returning directly from its signature, no need for a comment.

Named return parameters are particularly important in longer functions with many return values.

func calculator(a, b int) (mul, div int, err error) {
    if b == 0 {
      return 0, 0, errors.New("Can't divide by zero")
    }
    mul = a * b
    div = a / b
    return mul, div, nil
}

Which is easier to understand than:

func calculator(a, b int) (int, int, error) {
    if b == 0 {
      return 0, 0, errors.New("Can't divide by zero")
    }
    mul := a * b
    div := a / b
    return mul, div, nil
}

We know the meaning of each return value just by looking at the function signature: func calculator(a, b int) (mul, div int, err error)

Less Code (Sometimes)

If there are multiple return statements in a function, you don’t need to write all the return values each time, though you probably should.

When you choose to omit return values, it's called a naked return. Naked returns should only be used in short and simple functions.

Early Returns

Go supports the ability to return early from a function. This is a powerful feature that can clean up code, especially when used as guard clauses.

Guard Clauses leverage the ability to return early from a function (or continue through a loop) to make nested conditionals one-dimensional. Instead of using if/else chains, we just return early from the function at the end of each conditional block.

func divide(dividend, divisor int) (int, error) {
    if divisor == 0 {
        return 0, errors.New("Can't divide by zero")
    }
    return dividend/divisor, nil
}

Error handling in Go naturally encourages developers to make use of guard clauses. When I started writing more JavaScript, I was disappointed to see how many nested conditionals existed in the code I was working on.

Let’s take a look at an exaggerated example of nested conditional logic:

func getInsuranceAmount(status insuranceStatus) int {
  amount := 0
  if !status.hasInsurance(){
    amount = 1
  } else {
    if status.isTotaled(){
      amount = 10000
    } else {
      if status.isDented(){
        amount = 160
        if status.isBigDent(){
          amount = 270
        }
      } else {
        amount = 0
      }
    }
  }
  return amount
}

This could be written with guard clauses instead:

func getInsuranceAmount(status insuranceStatus) int {
  if !status.hasInsurance(){
    return 1
  }
  if status.isTotaled(){
    return 10000
  }
  if !status.isDented(){
    return 0
  }
  if status.isBigDent(){
    return 270
  }
  return 160
}

The example above is much easier to read and understand. When writing code, it’s important to try to reduce the cognitive load on the reader by reducing the number of entities they need to think about at any given time.

In the first example, if the developer is trying to figure out when 270 is returned, they need to think about each branch in the logic tree and try to remember which cases matter and which cases don’t.

With the one-dimensional structure offered by guard clauses, it’s as simple as stepping through each case in order.

Chapter 5 – Structs in Go

We use structs in Go to represent structured data. It's often convenient to group different types of variables together. For example, if we want to represent a car we could do the following:

type car struct {
  Make string
  Model string
  Height int
  Width int
}

This creates a new struct type called car. All cars have a Make, Model, Height and Width.

In Go, you will often use a struct to represent information that you would have used a dictionary for in Python, or an object literal in JavaScript.

Nested Structs in Go

Structs can be nested to represent more complex entities:

type car struct {
  Make string
  Model string
  Height int
  Width int
  FrontWheel Wheel
  BackWheel Wheel
}

type Wheel struct {
  Radius int
  Material string
}

The fields of a struct can be accessed using the dot . operator.

myCar := car{}
myCar.FrontWheel.Radius = 5

Anonymous Structs

An anonymous struct is just like a regular struct, but it is defined without a name and therefore cannot be referenced elsewhere in the code.

To create an anonymous struct, just instantiate the instance immediately using a second pair of brackets after declaring the type:

myCar := struct {
  Make string
  Model string
} {
  Make: "tesla",
  Model: "model 3"
}

You can even nest anonymous structs as fields within other structs:

type car struct {
  Make string
  Model string
  Height int
  Width int
  // Wheel is a field containing an anonymous struct
  Wheel struct {
    Radius int
    Material string
  }
}

When should you use an anonymous struct?

In general, prefer named structs. Named structs make it easier to read and understand your code, and they have the nice side-effect of being reusable. I sometimes use anonymous structs when I know I won't ever need to use a struct again. For example, sometimes I'll use one to create the shape of some JSON data in HTTP handlers.

If a struct is only meant to be used once, then it makes sense to declare it in such a way that developers down the road won’t be tempted to accidentally use it again.

You can read more about anonymous structs here if you're curious.

Embedded Structs

Go is not an object-oriented language. But embedded structs provide a kind of data-only inheritance that can be useful at times.

Keep in mind, Go doesn't support classes or inheritance in the complete sense. Embedded structs are just a way to elevate and share fields between struct definitions.

type car struct {
  make string
  model string
}

type truck struct {
  // "car" is embedded, so the definition of a
  // "truck" now also additionally contains all
  // of the fields of the car struct
  car
  bedSize int
}

Embedded vs nested

• An embedded struct's fields are accessed at the top level, unlike nested structs.

• Promoted fields can be accessed like normal fields except that they can't be used in composite literals

lanesTruck := truck{
  bedSize: 10,
  car: car{
    make: "toyota",
    model: "camry",
  },
}

fmt.Println(lanesTruck.bedSize)

// embedded fields promoted to the top-level
// instead of lanesTruck.car.make
fmt.Println(lanesTruck.make)
fmt.Println(lanesTruck.model)

Struct Methods

While Go is not object-oriented, it does support methods that can be defined on structs. Methods are just functions that have a receiver. A receiver is a special parameter that syntactically goes before the name of the function.

type rect struct {
  width int
  height int
}

// area has a receiver of (r rect)
func (r rect) area() int {
  return r.width * r.height
}

r := rect{
  width: 5,
  height: 10,
}

fmt.Println(r.area())
// prints 50

A receiver is just a special kind of function parameter. Receivers are important because they will, as you'll learn in the exercises to come, allow us to define interfaces that our structs (and other types) can implement.

Chapter 6 – Interfaces in Go

Interfaces are collections of method signatures. A type "implements" an interface if it has all of the methods of the given interface defined on it.

In the following example, a "shape" must be able to return its area and perimeter. Both rect and circle fulfill the interface.

type shape interface {
  area() float64
  perimeter() float64
}

type rect struct {
    width, height float64
}
func (r rect) area() float64 {
    return r.width * r.height
}
func (r rect) perimeter() float64 {
    return 2*r.width + 2*r.height
}

type circle struct {
    radius float64
}
func (c circle) area() float64 {
    return math.Pi * c.radius * c.radius
}
func (c circle) perimeter() float64 {
    return 2 * math.Pi * c.radius
}

When a type implements an interface, it can then be used as the interface type.

Interfaces are implemented implicitly.

A type never declares that it implements a given interface. If an interface exists and a type has the proper methods defined, then the type automatically fulfills that interface.

Multiple Interfaces

A type can implement any number of interfaces in Go. For example, the empty interface, interface{}, is always implemented by every type because it has no requirements.

Naming interface args

Consider the following interface:

type Copier interface {
  Copy(string, string) int
}

Based on the code alone, can you deduce what kinds of strings you should pass into the Copy function?

We know the function signature expects 2 string types, but what are they? Filenames? URLs? Raw string data? For that matter, what the heck is that int that's being returned?

Let's add some named arguments and return data to make it clearer.

type Copier interface {
  Copy(sourceFile string, destinationFile string) (bytesCopied int)
}

Much better. We can see what the expectations are now. The first argument is the sourceFile, the second argument is the destinationFile, and bytesCopied, an integer, is returned.

Type Assertions in Go

When working with interfaces in Go, every once-in-awhile you'll need access to the underlying type of an interface value. You can cast an interface to its underlying type using a type assertion.

type shape interface {
    area() float64
}

type circle struct {
    radius float64
}

// "c" is a new circle cast from "s"
// which is an instance of a shape.
// "ok" is a bool that is true if s was a circle
// or false if s isn't a circle
c, ok := s.(circle)
if !ok {
    // s wasn't a circle
    log.Fatal("s is not a circle")
}

radius := c.radius

Type Switches in Go

A type switch makes it easy to do several type assertions in a series.

A type switch is similar to a regular switch statement, but the cases specify types instead of values.

func printNumericValue(num interface{}) {
    switch v := num.(type) {
    case int:
        fmt.Printf("%T\n", v)
    case string:
        fmt.Printf("%T\n", v)
    default:
        fmt.Printf("%T\n", v)
    }
}

func main() {
    printNumericValue(1)
    // prints "int"

    printNumericValue("1")
    // prints "string"

    printNumericValue(struct{}{})
    // prints "struct {}"
}

fmt.Printf("%T\n", v) prints the type of a variable.

Clean Interfaces

Writing clean interfaces is hard. Frankly, anytime you’re dealing with abstractions in code, the simple can become complex very quickly if you’re not careful. Let’s go over some rules of thumb for keeping interfaces clean.

1. Keep Interfaces Small

If there is only one piece of advice that you take away from this article, make it this: keep interfaces small! Interfaces are meant to define the minimal behavior necessary to accurately represent an idea or concept.

Here is an example from the standard HTTP package of a larger interface that’s a good example of defining minimal behavior:

type File interface {
    io.Closer
    io.Reader
    io.Seeker
    Readdir(count int) ([]os.FileInfo, error)
    Stat() (os.FileInfo, error)
}

Any type that satisfies the interface’s behaviors can be considered by the HTTP package as a File. This is convenient because the HTTP package doesn’t need to know if it’s dealing with a file on disk, a network buffer, or a simple []byte.

2. Interfaces Should Have No Knowledge of Satisfying Types

An interface should define what is necessary for other types to classify as a member of that interface. They shouldn’t be aware of any types that happen to satisfy the interface at design time.

For example, let’s assume we are building an interface to describe the components necessary to define a car.

type car interface {
    Color() string
    Speed() int
    IsFiretruck() bool
}

Color() and Speed() make perfect sense, they are methods confined to the scope of a car. IsFiretruck() is an anti-pattern. We are forcing all cars to declare whether or not they are firetrucks. In order for this pattern to make any amount of sense, we would need a whole list of possible subtypes. IsPickup(), IsSedan(), IsTank()… where does it end??

Instead, the developer should have relied on the native functionality of type assertion to derive the underlying type when given an instance of the car interface. Or, if a sub-interface is needed, it can be defined as:

type firetruck interface {
    car
    HoseLength() int
}

Which inherits the required methods from car and adds one additional required method to make the car a firetruck.

3. Interfaces Are Not Classes

• Interfaces are not classes, they are slimmer.

• Interfaces don’t have constructors or deconstructors that require that data is created or destroyed.

• Interfaces aren’t hierarchical by nature, though there is syntactic sugar to create interfaces that happen to be supersets of other interfaces.

• Interfaces define function signatures, but not underlying behavior. Making an interface often won’t DRY up your code in regards to struct methods. For example, if five types satisfy the fmt.Stringer interface, they all need their own version of the String() function.

Chapter 7 – Errors in Go

Go programs express errors with error values. An Error is any type that implements the simple built-in error interface:

type error interface {
    Error() string
}

When something can go wrong in a function, that function should return an error as its last return value. Any code that calls a function that can return an error should handle errors by testing whether the error is nil.

// Atoi converts a stringified number to an interger
i, err := strconv.Atoi("42b")
if err != nil {
    fmt.Println("couldn't convert:", err)
    // because "42b" isn't a valid integer, we print:
    // couldn't convert: strconv.Atoi: parsing "42b": invalid syntax
    // Note:
    // 'parsing "42b": invalid syntax' is returned by the .Error() method
    return
}
// if we get here, then
// i was converted successfully

A nil error denotes success. A non-nil error denotes failure.

The Error Interface

Because errors are just interfaces, you can build your own custom types that implement the error interface. Here's an example of a userError struct that implements the error interface:

type userError struct {
    name string
}

func (e userError) Error() string {
    return fmt.Sprintf("%v has a problem with their account", e.name)
}

It can then be used as an error:

func sendSMS(msg, userName string) error {
    if !canSendToUser(userName) {
        return userError{name: userName}
    }
    ...
}

Go programs express errors with error values. Error-values are any type that implements the simple built-in error interface.

Keep in mind that the way Go handles errors is fairly unique. Most languages treat errors as something special and different. For example, Python raises exception types and JavaScript throws and catches errors.

In Go, an error is just another value that we handle like any other value – however, we want! There aren't any special keywords for dealing with them.

The errors Package

The Go standard library provides an "errors" package that makes it easy to deal with errors.

Read the godoc for the errors.New() function, but here's a simple example:

var err error = errors.New("something went wrong")

Chapter 8 – Loops in Go

The basic loop in Go is written in standard C-like syntax:

for INITIAL; CONDITION; AFTER{
  // do something
}

INITIAL is run once at the beginning of the loop and can create variables within the scope of the loop.

CONDITION is checked before each iteration. If the condition doesn't pass then the loop breaks.

AFTER is run after each iteration.

For example:

for i := 0; i < 10; i++ {
  fmt.Println(i)
}
// Prints 0 through 9

How to Omit Conditions

Loops in Go can omit sections of a for loop. For example, the CONDITION (middle part) can be omitted which causes the loop to run forever.

for INITIAL; ; AFTER {
  // do something forever
}

No while loops in Go

Most programming languages have a concept of a while loop. Because Go allows for the omission of sections of a for loop, a while loop is just a for loop that only has a CONDITION.

for CONDITION {
  // do some stuff while CONDITION is true
}

For example:

plantHeight := 1
for plantHeight < 5 {
  fmt.Println("still growing! current height:", plantHeight)
  plantHeight++
}
fmt.Println("plant has grown to ", plantHeight, "inches")

Which prints:

still growing! current height: 1
still growing! current height: 2
still growing! current height: 3
still growing! current height: 4
plant has grown to 5 inches

Continue through a loop

The continue keyword stops the current iteration of a loop and continues to the next iteration. continue is a powerful way to use the "guard clause" pattern within loops.

for i := 0; i < 10; i++ {
  if i % 2 == 0 {
    continue
  }
  fmt.Println(i)
}
// 1
// 3
// 5
// 7
// 9

Break out of a loop

The break keyword stops the current iteration of a loop and exits the loop.

for i := 0; i < 10; i++ {
  if i == 5 {
    break
  }
  fmt.Println(i)
}
// 0
// 1
// 2
// 3
// 4

Chapter 9 – Arrays and Slices in Go

Arrays

Arrays are fixed-size groups of variables of the same type.

The type [n]T is an array of n values of type T.

To declare an array of 10 integers:

var myInts [10]int

or to declare an initialized literal:

primes := [6]int{2, 3, 5, 7, 11, 13}

Slices

99 times out of 100 you will use a slice instead of an array when working with ordered lists.

Arrays are fixed in size. Once you make an array like [10]int you can't add an 11th element.

A slice is a dynamically-sized, flexible view of the elements of an array.

Slices always have an underlying array, though it isn't always specified explicitly. To explicitly create a slice on top of an array we can do:

primes := [6]int{2, 3, 5, 7, 11, 13}
mySlice := primes[1:4]
// mySlice = {3, 5, 7}

The syntax is:

arrayname[lowIndex:highIndex]
arrayname[lowIndex:]
arrayname[:highIndex]
arrayname[:]

Where lowIndex is inclusive and highIndex is exclusive

Either lowIndex or highIndex or both can be omitted to use the entire array on that side.

How to Create New Slices in Go

Most of the time we don't need to think about the underlying array of a slice. We can create a new slice using the make function:

// func make([]T, len, cap) []T
mySlice := make([]int, 5, 10)

// the capacity argument is usually omitted and defaults to the length
mySlice := make([]int, 5)

Slices created with make will be filled with the zero value of the type.

If we want to create a slice with a specific set of values, we can use a slice literal:

mySlice := []string{"I", "love", "go"}

Note that the array brackets do not have a 3 in them. If they did, you'd have an array instead of a slice.

Length

The length of a slice is simply the number of elements it contains. It is accessed using the built-in len() function:

mySlice := []string{"I", "love", "go"}
fmt.Println(len(mySlice)) // 3

Capacity

The capacity of a slice is the number of elements in the underlying array, counting from the first element in the slice. It is accessed using the built-in cap() function:

mySlice := []string{"I", "love", "go"}
fmt.Println(cap(mySlice)) // 3

Generally speaking, unless you're hyper-optimizing the memory usage of your program, you don't need to worry about the capacity of a slice because it will automatically grow as needed.

Variadic Functions

Many functions, especially those in the standard library, can take an arbitrary number of final arguments. This is accomplished by using the "..." syntax in the function signature.

A variadic function receives the variadic arguments as a slice.

func concat(strs ...string) string {
    final := ""
    // strs is just a slice of strings
    for str := range strs {
        final += str
    }
    return final
}

func main() {
    final := concat("Hello ", "there ", "friend!")
    fmt.Println(total)
    // Output: Hello there friend!
}

The familiar fmt.Println() and fmt.Sprintf() are variadic! fmt.Println() prints each element with space delimiters and a newline at the end.

func Println(a ...interface{}) (n int, err error)

Spread operator

The spread operator allows us to pass a slice into a variadic function. The spread operator consists of three dots following the slice in the function call.

func printStrings(strings ...string) {
    for i := 0; i < len(strings); i++ {
        fmt.Println(strings[i])
    }
}

func main() {
    names := []string{"bob", "sue", "alice"}
    printStrings(names...)
}

How to Append to a Slice

The built-in append function is used to dynamically add elements to a slice:

func append(slice []Type, elems ...Type) []Type

If the underlying array is not large enough, append() will create a new underlying array and point the slice to it.

Notice that append() is variadic. The following are all valid:

slice = append(slice, oneThing)
slice = append(slice, firstThing, secondThing)
slice = append(slice, anotherSlice...)

How to Range Over a Slice in Go

Go provides syntactic sugar to iterate easily over elements of a slice:

for INDEX, ELEMENT := range SLICE {
}

For example:

fruits := []string{"apple", "banana", "grape"}
for i, fruit := range fruits {
    fmt.Println(i, fruit)
}
// 0 apple
// 1 banana
// 2 grape

Chapter 10 – Maps in Go

Maps are similar to JavaScript objects, Python dictionaries, and Ruby hashes. Maps are a data structure that provides key->value mapping.

The zero value of a map is nil.

We can create a map by using a literal or by using the make() function:

ages := make(map[string]int)
ages["John"] = 37
ages["Mary"] = 24
ages["Mary"] = 21 // overwrites 24

ages = map[string]int{
  "John": 37,
  "Mary": 21,
}

The len() function works on a map – it returns the total number of key/value pairs.

ages = map[string]int{
  "John": 37,
  "Mary": 21,
}
fmt.Println(len(ages)) // 2

Map Mutations

Insert an element

m[key] = elem

Get an element

elem = m[key]

Delete an element

delete(m, key)

Check if a key exists

elem, ok := m[key]

If key is in m, then ok is true. If not, ok is false.

If key is not in the map, then elem is the zero value for the map's element type.

Types of Valid Map Keys

Any type can be used as the value in a map, but keys are more restrictive.

You can read more in the following section of the official Go blog.

As mentioned earlier, map keys may be of any type that is comparable. The language spec defines this precisely, but in short, comparable types are boolean, numeric, string, pointer, channel, and interface types, and structs or arrays that contain only those types.

Notably absent from the list are slices, maps, and functions. These types cannot be compared using ==, and may not be used as map keys.

It's obvious that strings, ints, and other basic types should be available as map keys, but perhaps unexpected are struct keys. Struct can be used to key data by multiple dimensions.

For example, this map of maps could be used to tally web page hits by country:

hits := make(map[string]map[string]int)

This is map of string to (map of string to int). Each key of the outer map is the path to a web page with its own inner map. Each inner map key is a two-letter country code. This expression retrieves the number of times an Australian has loaded the documentation page:

n := hits["/doc/"]["au"]

Unfortunately, this approach becomes unwieldy when adding data, as for any given outer key you must check if the inner map exists, and create it if needed:

func add(m map[string]map[string]int, path, country string) {
    mm, ok := m[path]
    if !ok {
        mm = make(map[string]int)
        m[path] = mm
    }
    mm[country]++
}
add(hits, "/doc/", "au")

On the other hand, a design that uses a single map with a struct key does away with all that complexity:

type Key struct {
    Path, Country string
}
hits := make(map[Key]int)

When a Vietnamese person visits the home page, incrementing (and possibly creating) the appropriate counter is a one-liner:

hits[Key{"/", "vn"}]++

And it’s similarly straightforward to see how many Swiss people have read the spec:

n := hits[Key{"/ref/spec", "ch"}]

Nested Maps

Maps can contain maps, creating a nested structure. For example:

map[string]map[string]int
map[rune]map[string]int
map[int]map[string]map[string]int

Chapter 11 – Advanced Functions in Go

First-class and higher-order functions

A programming language is said to have "first-class functions" when functions in that language are treated like any other variable.

For example, in such a language, a function can be passed as an argument to other functions, can be returned by another function, and can be assigned as a value to a variable.

A function that returns a function or accepts a function as input is called a Higher-Order Function.

Go supports first-class and higher-order functions. Another way to think of this is that a function is just another type – just like ints and strings and bools.

For example, to accept a function as a parameter:

func add(x, y int) int {
  return x + y
}

func mul(x, y int) int {
  return x * y
}

// aggregate applies the given math function to the first 3 inputs
func aggregate(a, b, c int, arithmetic func(int, int) int) int {
  return arithmetic(arithmetic(a, b), c)
}

func main(){
  fmt.Println(aggregate(2,3,4, add))
  // prints 9
  fmt.Println(aggregate(2,3,4, mul))
  // prints 24
}

Function Currying in Go

Function currying is the practice of writing a function that takes a function (or functions) as input, and returns a new function.

For example:

func main() {
  squareFunc := selfMath(multiply)
  doubleFunc := selfMath(add)

  fmt.Println(squareFunc(5))
  // prints 25

  fmt.Println(doubleFunc(5))
  // prints 10
}

func multiply(x, y int) int {
  return x * y
}

func add(x, y int) int {
  return x + y
}

func selfMath(mathFunc func(int, int) int) func (int) int {
  return func(x int) int {
    return mathFunc(x, x)
  }
}

In the example above, the selfMath function takes in a function as its parameter, and returns a function that itself returns the value of running that input function on its parameter.

Defer keyword

The defer keyword is a fairly unique feature of Go. It allows a function to be executed automatically just before its enclosing function returns.

The deferred call's arguments are evaluated immediately, but the function call is not executed until the surrounding function returns.

Deferred functions are typically used to close database connections, file handlers and the like.

For example:

// CopyFile copies a file from srcName to dstName on the local filesystem.
func CopyFile(dstName, srcName string) (written int64, err error) {

  // Open the source file
  src, err := os.Open(srcName)
  if err != nil {
    return
  }
  // Close the source file when the CopyFile function returns
  defer src.Close()

  // Create the destination file
  dst, err := os.Create(dstName)
  if err != nil {
    return
  }
  // Close the destination file when the CopyFile function returns
  defer dst.Close()

  return io.Copy(dst, src)
}

In the above example, the src.Close() function is not called until after the CopyFile function was called but immediately before the CopyFile function returns.

Defer is a great way to make sure that something happens at the end of a function, even if there are multiple return statements.

Closures

A closure is a function that references variables from outside its own function body. The function may access and assign to the referenced variables.

In this example, the concatter() function returns a function that has reference to an enclosed doc value. Each successive call to harryPotterAggregator mutates that same doc variable.

func concatter() func(string) string {
    doc := ""
    return func(word string) string {
        doc += word + " "
        return doc
    }
}

func main() {
    harryPotterAggregator := concatter()
    harryPotterAggregator("Mr.")
    harryPotterAggregator("and")
    harryPotterAggregator("Mrs.")
    harryPotterAggregator("Dursley")
    harryPotterAggregator("of")
    harryPotterAggregator("number")
    harryPotterAggregator("four,")
    harryPotterAggregator("Privet")

    fmt.Println(harryPotterAggregator("Drive"))
    // Mr. and Mrs. Dursley of number four, Privet Drive
}

Anonymous Functions

Anonymous functions are true to form in that they have no name. We've been using them throughout this chapter, but we haven't really talked about them yet.

Anonymous functions are useful when defining a function that will only be used once or to create a quick closure.

// doMath accepts a function that converts one int into another
// and a slice of ints. It returns a slice of ints that have been
// converted by the passed in function.
func doMath(f func(int) int, nums []int) []int {
    var results []int
    for _, n := range nums {
        results = append(results, f(n))
    }
    return results
}

func main() {
    nums := []int{1, 2, 3, 4, 5}

    // Here we define an anonymous function that doubles an int
    // and pass it to doMath
    allNumsDoubled := doMath(func(x int) int {
        return x + x
    }, nums)

    fmt.Println(allNumsDoubled)
    // prints:
    // [2 4 6 8 10]
}

Chapter 12 – Pointers in Go

As we have learned, a variable is a named location in memory that stores a value. We can manipulate the value of a variable by assigning a new value to it or by performing operations on it. When we assign a value to a variable, we are storing that value in a specific location in memory.

x := 42
// "x" is the name of a location in memory. That location is storing the integer value of 42

A pointer is a variable

A pointer is a variable that stores the memory address of another variable. This means that a pointer "points to" the location of where the data is stored NOT the actual data itself.

The * syntax defines a pointer:

var p *int

The & operator generates a pointer to its operand.

myString := "hello"
myStringPtr = &myString

Why are pointers useful?

Pointers allow us to manipulate data in memory directly, without making copies or duplicating data. This can make programs more efficient and allow us to do things that would be difficult or impossible without them.

Pointer Syntax

The * syntax defines a pointer:

var p *int

A pointer's zero value is nil

The & operator generates a pointer to its operand:

myString := "hello"
myStringPtr = &myString

The * dereferences a pointer to gain access to the value:

fmt.Println(*myStringPtr) // read myString through the pointer
*myStringPtr = "world"    // set myString through the pointer

Unlike C, Go has no pointer arithmetic

Just because you can doesn't mean you should

We're doing this exercise to understand that pointers can be used in this way. That said, pointers can be very dangerous. It's generally a better idea to have your functions accept non-pointers and return new values rather than mutating pointer inputs.

Nil Pointers

Again, pointers can be very dangerous.

If a pointer points to nothing (the zero value of the pointer type) then dereferencing it will cause a runtime error (a panic) that crashes the program.

Generally speaking, whenever you're dealing with pointers you should check if it's nil before trying to dereference it.

Pointer Method Receivers

A receiver type on a method can be a pointer.

Methods with pointer receivers can modify the value to which the receiver points. Since methods often need to modify their receiver, pointer receivers are more common than value receivers.

Pointer receiver

type car struct {
    color string
}

func (c *car) setColor(color string) {
    c.color = color
}

func main() {
    c := car{
        color: "white",
    }
    c.setColor("blue")
    fmt.Println(c.color)
    // prints "blue"
}

Non-pointer receiver

type car struct {
    color string
}

func (c car) setColor(color string) {
    c.color = color
}

func main() {
    c := car{
        color: "white",
    }
    c.setColor("blue")
    fmt.Println(c.color)
    // prints "white"
}

Methods with pointer receivers don't require that a pointer is used to call the method. The pointer will automatically be derived from the value.

type circle struct {
    x int
    y int
    radius int
}

func (c *circle) grow(){
    c.radius *= 2
}

func main(){
    c := circle{
        x: 1,
        y: 2,
        radius: 4,
    }

    // notice c is not a pointer in the calling function
    // but the method still gains access to a pointer to c
    c.grow()
    fmt.Println(c.radius)
    // prints 8
}

Chapter 13 – Local Development Environment in Go

Packages

Make sure you have Go installed on your local machine.

Every Go program is made up of packages.

You have probably noticed the package main at the top of all the programs you have been writing.

A package named "main" has an entrypoint at the main() function. A main package is compiled into an executable program.

A package by any other name is a "library package". Libraries have no entry point. Libraries simply export functionality that can be used by other packages. For example:

package main

import (
    "fmt"
    "math/rand"
)

func main() {
    fmt.Println("My favorite number is", rand.Intn(10))
}

This program is an executable. It is a "main" package and imports from the fmt and math/rand library packages.

Package Names

Naming Convention

By convention, a package's name is the same as the last element of its import path. For instance, the math/rand package comprises files that begin with:

package rand

That said, package names aren't required to match their import path. For example, I could write a new package with the path github.com/mailio/rand and name the package random:

package random

While the above is possible, it is discouraged for the sake of consistency.

One Package / Directory

A directory of Go code can have at most one package. All .go files in a single directory must all belong to the same package. If they don't an error will be thrown by the compiler. This is true for main and library packages alike.

Go Modules

Go programs are organized into packages. A package is a directory of Go code that's all compiled together. Functions, types, variables, and constants defined in one source file are visible to all other source files within the same package (directory).

A repository contains one or more modules. A module is a collection of Go packages that are released together.

A Go repository typically contains only one module, located at the root of the repository.

A file named go.mod at the root of a project declares the module. It contains:

• The module path

• The version of the Go language your project requires

• Optionally, any external package dependencies your project has

The module path is just the import path prefix for all packages within the module. Here's an example of a go.mod file:

module github.com/bootdotdev/exampleproject

go 1.20

require github.com/google/examplepackage v1.3.0

Each module's path not only serves as an import path prefix for the packages within but also indicates where the go command should look to download it.

For example, to download the module golang.org/x/tools, the go command would consult the repository located at https://golang.org/x/tools.

An "import path" is a string used to import a package. A package's import path is its module path joined with its subdirectory within the module. For example, the module github.com/google/go-cmp contains a package in the directory cmp/. That package's import path is github.com/google/go-cmp/cmp. Packages in the standard library do not have a module path prefix. – Paraphrased from Golang.org's code organization

Do I need to put my package on GitHub?

You don't need to publish your code to a remote repository before you can build it. A module can be defined locally without belonging to a repository. But it's a good habit to keep a copy of all your projects on a remote server, like GitHub.

How to Set Up Your Machine

Your machine will contain many version control repositories (managed by Git, for example).

Each repository contains one or more packages, but will typically be a single module.

Each package consists of one or more Go source files in a single directory.

The path to a package's directory determines its import path and where it can be downloaded from if you decide to host it on a remote version control system like Github or Gitlab.

A note on GOPATH

The $GOPATH environment variable will be set by default somewhere on your machine (typically in the home directory, ~/go). Since we will be working in the new "Go modules" setup, you don't need to worry about that. If you read something online about setting up your GOPATH, that documentation is probably out of date.

These days you should avoid working in the $GOPATH/src directory. Again, that's the old way of doing things and can cause unexpected issues, so better to just avoid it.

Get into your workspace

Navigate to a location on your machine where you want to store some code. For example, I store all my code in ~/workspace, then organize it into subfolders based on the remote location. For example,

~/workspace/github.com/wagslane/go-password-validator = https://github.com/wagslane/go-password-validator

That said, you can put your code wherever you want.

How to Write Your First Local Go Program

Once inside your personal workspace, create a new directory and enter it:

mkdir hellogo
cd hellogo

Inside the directory declare your module's name:

go mod init {REMOTE}/{USERNAME}/hellogo

Where {REMOTE} is your preferred remote source provider (i.e. github.com) and {USERNAME} is your Git username. If you don't use a remote provider yet, just use example.com/username/hellogo

Print your go.mod file:

cat go.mod

The Go Run Command

Inside hellogo, create a new file called main.go.

Conventionally, the file in the main package that contains the main() function is called main.go.

Paste the following code into your file:

package main

import "fmt"

func main() {
    fmt.Println("hello world")
}

Run the code

go run main.go

The go run command is used to quickly compile and run a Go package. The compiled binary is not saved in your working directory. Use go build instead to compile production executables.

I rarely use go run other than to quickly do some testing or debugging.

Further reading

Execute go help run in your shell and read the instructions.

The Go Build Command

go build compiles go code into an executable program.

Build an executable

Ensure you are in your hellogo repo, then run:

go build

Run the new program:

./hellogo

Go Install

Build an executable

Ensure you are in your hellogo repo, then run:

go install

Navigate out of your project directory:

cd ../

Go has installed the hellogo program globally. Run it with:

hellogo

Tip about "not found"

If you get an error regarding "hellogo not found" it means you probably don't have your Go environment setup properly. Specifically, go install is adding your binary to your GOBIN directory, but that may not be in your PATH.

You can read more about that here in the go install docs.

How to Create a Custom Go Package

Let's write a package to import and use in hellogo.

Create a sibling directory at the same level as the hellogo directory:

mkdir mystrings
cd mystrings

Initialize a module:

go mod init {REMOTE}/{USERNAME}/mystrings

Then create a new file mystrings.go in that directory and paste the following code:

// by convention, we name our package the same as the directory
package mystrings

// Reverse reverses a string left to right
// Notice that we need to capitalize the first letter of the function
// If we don't then we won't be able access this function outside of the
// mystrings package
func Reverse(s string) string {
  result := ""
  for _, v := range s {
    result = string(v) + result
  }
  return result
}

Note that there is no main.go or func main() in this package.

go build won't build an executable from a library package. However, go build will still compile the package and save it to our local build cache. It's useful for checking for compile errors.

Run:

go build

How to Publish Remote Packages in Go

Let's learn how to use an open-source package that's available online.

A note on how you should publish modules

Be aware that using the "replace" keyword like we did in the last assignment isn't advised, but can be useful to get up and running quickly. The proper way to create and depend on modules is to publish them to a remote repository. When you do that, the "replace keyword can be dropped from the go.mod:

Bad

This works for local-only development

module github.com/wagslane/hellogo

go 1.20

replace github.com/wagslane/mystrings v0.0.0 => ../mystrings

require (
    github.com/wagslane/mystrings v0.0.0
)

Good

This works if we publish our modules to a remote location like Github as we should.

module github.com/wagslane/hellogo

go 1.20

require (
    github.com/wagslane/mystrings v0.0.0
)

Best Practices with Go Packages

I’ve often seen, and have been responsible for, throwing code into packages without much thought. I’ve quickly drawn a line in the sand and started putting code into different folders (which in Go are different packages by definition) just for the sake of findability.

Learning to properly build small and reusable packages can take your Go career to the next level.

1. Hide internal logic

If you're familiar with the pillars of OOP, this is a practice in encapsulation.

Oftentimes an application will have complex logic that requires a lot of code. In almost every case the logic that the application cares about can be exposed via an API, and most of the dirty work can be kept within a package.

For example, imagine we are building an application that needs to classify images. We could build a package:

package classifier

// ClassifyImage classifies images as "hotdog" or "not hotdog"
func ClassifyImage(image []byte) (imageType string) {
    return hasHotdogColors(image) && hasHotdogShape(image)
}

func hasHotdogShape(image []byte) bool {
    // internal logic that the application doesn't need to know about
    return true
}

func hasHotdogColors(image []byte) bool {
    // internal logic that the application doesn't need to know about
    return true
}

We create an API by only exposing the function(s) that the application-level needs to know about. All other logic is unexported to keep a clean separation of concerns. The application doesn’t need to know how to classify an image, just the result of the classification.

2. Don’t change APIs

The unexported functions within a package can and should change often for testing, refactoring, and bug fixing.

A well-designed library will have a stable API so that users aren’t receiving breaking changes each time they update the package version. In Go, this means not changing exported function’s signatures.

3. Don’t export functions from the main package

A main package isn't a library, there's no need to export functions from it.

4. Packages shouldn't know about dependents

Perhaps one of the most important and most broken rules is that a package shouldn’t know anything about its dependents. In other words, a package should never have specific knowledge about a particular application that uses it.

Further Reading

You can optionally read more here if you're interested.

Chapter 14 – Channels in Go

Concurrency

Concurrency is the ability to perform multiple tasks at the same time. Typically, our code is executed one line at a time, one after the other. This is called sequential execution or synchronous execution.

If the computer we're running our code on has multiple cores, we can even execute multiple tasks at exactly the same time. If we're running on a single core, a single code executes code at almost the same time by switching between tasks very quickly. Either way, the code we write looks the same in Go and takes advantage of whatever resources are available.

How does concurrency work in Go?

Go was designed to be concurrent, which is a trait fairly unique to Go. It excels at performing many tasks simultaneously and safely using a simple syntax.

There isn't a popular programming language in existence where spawning concurrent execution is quite as elegant, at least in my opinion.

Concurrency is as simple as using the go keyword when calling a function:

go doSomething()

In the example above, doSomething() will be executed concurrently with the rest of the code in the function. The go keyword is used to spawn a new goroutine.

Channels in Go

Channels are a typed, thread-safe queue. Channels allow different goroutines to communicate with each other.

Create a channel

Like maps and slices, channels must be created before use. They also use the same make keyword:

ch := make(chan int)

Send data to a channel

ch <- 69

The <- operator is called the channel operator. Data flows in the direction of the arrow. This operation will block until another goroutine is ready to receive the value.

Receive data from a channel

v := <-ch

This reads and removes a value from the channel and saves it into the variable v. This operation will block until there is a value in the channel to be read.

Blocking and deadlocks

A deadlock is when a group of goroutines are all blocking so none of them can continue. This is a common bug that you need to watch out for in concurrent programming.

Tokens

Empty structs are often used as tokens in Go programs. In this context, a token is a unary value. In other words, we don't care what is passed through the channel. We care when and if it is passed.

We can block and wait until something is sent on a channel using the following syntax

<-ch

This will block until it pops a single item off the channel, then continue, discarding the item.

Buffered Channels

Channels can optionally be buffered.

How to create a channel with a buffer

You can provide a buffer length as the second argument to make() to create a buffered channel:

ch := make(chan int, 100)

Sending on a buffered channel only blocks when the buffer is full.

Receiving blocks only when the buffer is empty.

How to Close Channels

Channels can be explicitly closed by a sender:

ch := make(chan int)

// do some stuff with the channel

close(ch)

How to check if a channel is closed

Similar to the ok value when accessing data in a map, receivers can check the ok value when receiving from a channel to test if a channel was closed.

v, ok := <-ch

ok is false if the channel is empty and closed.

Don't send on a closed channel

Sending on a closed channel will cause a panic. A panic on the main goroutine will cause the entire program to crash, and a panic in any other goroutine will cause that goroutine to crash.

Closing isn't necessary. There's nothing wrong with leaving channels open, they'll still be garbage collected if they're unused. You should close channels to indicate explicitly to a receiver that nothing else is going to come across.

Range Over a Channel

Similar to slices and maps, channels can be ranged over.

for item := range ch {
    // item is the next value received from the channel
}

This example will receive values over the channel (blocking at each iteration if nothing new is there) and will exit only when the channel is closed.

Select from a channel

Sometimes we have a single goroutine listening to multiple channels and want to process data in the order it comes through each channel.

A select statement is used to listen to multiple channels at the same time. It is similar to a switch statement but for channels.

select {
  case i, ok := <- chInts:
    fmt.Println(i)
  case s, ok := <- chStrings:
    fmt.Println(s)
}

The first channel with a value ready to be received will fire and its body will execute. If multiple channels are ready at the same time one is chosen randomly. The ok variable in the example above refers to whether or not the channel has been closed by the sender yet.

Select Default

The default case in a select statement executes immediately if no other channel has a value ready. A default case stops the select statement from blocking.

select {
  case v := <-ch:
    // use v
  default:
    // receiving from ch would block
    // so do something else
}

Chapter 15 – Mutexes in Go

Mutexes allow us to lock access to data. This ensures that we can control which goroutines can access certain data at which time.

Go's standard library provides a built-in implementation of a mutex with the sync.Mutex type and its two methods:

• .Lock()

• .Unlock()

We can protect a block of code by surrounding it with a call to Lock and Unlock as shown on the protected() method below.

It's good practice to structure the protected code within a function so that defer can be used to ensure that we never forget to unlock the mutex.

func protected(){
    mux.Lock()
    defer mux.Unlock()
    // the rest of the function is protected
    // any other calls to `mux.Lock()` will block
}

Mutexes are powerful. Like most powerful things, they can also cause many bugs if used carelessly.

Maps are not thread-safe

Maps are not safe for concurrent use! If you have multiple goroutines accessing the same map, and at least one of them is writing to the map, you must lock your maps with a mutex.

Why is it called a Mutex?

Mutex is short for mutual exclusion, and the conventional name for the data structure that provides it is "mutex", often abbreviated to "mux".

It's called "mutual exclusion" because a mutex excludes different threads (or goroutines) from accessing the same data at the same time.

Why use mutexes?

The principle problem that mutexes help us avoid is the concurrent read/write problem. This problem arises when one thread is writing to a variable while another thread is reading from that same variable at the same time.

When this happens, a Go program will panic because the reader could be reading bad data while it's being mutated in place.

Mutex example

package main

import (
    "fmt"
)

func main() {
    m := map[int]int{}
    go writeLoop(m)
    go readLoop(m)

    // stop program from exiting, must be killed
    block := make(chan struct{})
    <-block
}

func writeLoop(m map[int]int) {
    for {
        for i := 0; i < 100; i++ {
            m[i] = i
        }
    }
}

func readLoop(m map[int]int) {
    for {
        for k, v := range m {
            fmt.Println(k, "-", v)
        }
    }
}

The example above creates a map, then starts two goroutines which each have access to the map. One goroutine continuously mutates the values stored in the map, while the other prints the values it finds in the map.

If we run the program on a multi-core machine, we get the following output: fatal error: concurrent map iteration and map write

In Go, it isn’t safe to read from and write to a map at the same time.

Mutexes to the rescue

package main

import (
    "fmt"
    "sync"
)

func main() {
    m := map[int]int{}

    mux := &sync.Mutex{}

    go writeLoop(m, mux)
    go readLoop(m, mux)

    // stop program from exiting, must be killed
    block := make(chan struct{})
    <-block
}

func writeLoop(m map[int]int, mux *sync.Mutex) {
    for {
        for i := 0; i < 100; i++ {
            mux.Lock()
            m[i] = i
            mux.Unlock()
        }
    }
}

func readLoop(m map[int]int, mux *sync.Mutex) {
    for {
        mux.Lock()
        for k, v := range m {
            fmt.Println(k, "-", v)
        }
        mux.Unlock()
    }
}

In this example, we added a sync.Mutex{} and named it mux. In the write loop, the Lock() method is called before writing, and then the Unlock() is called when we're done. This Lock/Unlock sequence ensures that no other threads can Lock() the mutex while we have it locked – any other threads attempting to Lock() will block and wait until we Unlock().

In the reader, we Lock() before iterating over the map, and likewise Unlock() when we're done. Now the threads share the memory safely!

RWMutex

The standard library also exposes a sync.RWMutex

In addition to these methods:

• Lock()

• Unlock()

The sync.RWMutex also has these methods:

• RLock()

• RUnlock()

The sync.RWMutex can help with performance if we have a read-intensive process. Many goroutines can safely read from the map at the same time (multiple Rlock() calls can happen simultaneously). However, only one goroutine can hold a Lock() and all RLock()'s will also be excluded.

Chapter 16 – Generics in Go

As we've mentioned, Go does not support classes. For a long time, that meant that Go code couldn't easily be reused in many circumstances.

For example, imagine some code that splits a slice into 2 equal parts. The code that splits the slice doesn't really care about the values stored in the slice. Unfortunately in Go we would need to write it multiple times for each type, which is a very un-DRY thing to do.

func splitIntSlice(s []int) ([]int, []int) {
    mid := len(s)/2
    return s[:mid], s[mid:]
}

func splitStringSlice(s []string) ([]string, []string) {
    mid := len(s)/2
    return s[:mid], s[mid:]
}

In Go 1.20 however, support for generics was released, effectively solving this problem!

Type Parameters

Put simply, generics allow us to use variables to refer to specific types. This is an amazing feature because it allows us to write abstract functions that drastically reduce code duplication.

func splitAnySlice[T any](s []T) ([]T, []T) {
    mid := len(s)/2
    return s[:mid], s[mid:]
}

In the example above, T is the name of the type parameter for the splitAnySlice function, and we've said that it must match the any constraint, which means it can be anything. This makes sense because the body of the function doesn't care about the types of things stored in the slice.

firstInts, secondInts := splitAnySlice([]int{0, 1, 2, 3})
fmt.Println(firstInts, secondInts)

Why Generics?

Generics reduce repetitive code

You should care about generics because they mean you don’t have to write as much code! It can be frustrating to write the same logic over and over again, just because you have some underlying data types that are slightly different.

Generics are used more often in libraries and packages

Generics give Go developers an elegant way to write amazing utility packages. While you will see and use generics in application code, I think it will much more common to see generics used in libraries and packages. Libraries and packages contain importable code intended to be used in many applications, so it makes sense to write them in a more abstract way. Generics are often the way to do just that!

Why did it take so long to get generics?

Go places an emphasis on simplicity. In other words, Go has purposefully left out many features to provide its best feature: being simple and easy to work with.

According to historical data from Go surveys, Go’s lack of generics has always been listed as one of the top three biggest issues with the language. At a certain point, the drawbacks associated with the lack of a feature like generics justify adding complexity to the language.

Constraints in Go

Sometimes you need the logic in your generic function to know something about the types it operates on. The example we used in the first exercise didn't need to know anything about the types in the slice, so we used the built-in any constraint:

func splitAnySlice[T any](s []T) ([]T, []T) {
    mid := len(s)/2
    return s[:mid], s[mid:]
}

Constraints are just interfaces that allow us to write generics that only operate within the constraint of a given interface type. In the example above, the any constraint is the same as the empty interface because it means the type in question can be anything.

How to create a custom constraint

Let's take a look at the example of a concat function. It takes a slice of values and concatenates the values into a string. This should work with any type that can represent itself as a string, even if it's not a string under the hood.

For example, a user struct can have a .String() that returns a string with the user's name and age.

type stringer interface {
    String() string
}

func concat[T stringer](vals []T) string {
    result := ""
    for _, val := range vals {
        // this is where the .String() method
        // is used. That's why we need a more specific
        // constraint instead of the any constraint
        result += val.String()
    }
    return result
}

Interface type lists

When generics were released, a new way of writing interfaces was also released at the same time!

We can now simply list a bunch of types to get a new interface/constraint.

// Ordered is a type constraint that matches any ordered type.
// An ordered type is one that supports the <, <=, >, and >= operators.
type Ordered interface {
    ~int | ~int8 | ~int16 | ~int32 | ~int64 |
        ~uint | ~uint8 | ~uint16 | ~uint32 | ~uint64 | ~uintptr |
        ~float32 | ~float64 |
        ~string
}

How to Name Generic Types

Let's look at this simple example again:

func splitAnySlice[T any](s []T) ([]T, []T) {
    mid := len(s)/2
    return s[:mid], s[mid:]
}

Remember, T is just a variable name, We could have named the type parameter anything. T happens to be a fairly common convention for a type variable, similar to how i is a convention for index variables in loops.

This is just as valid:

func splitAnySlice[MyAnyType any](s []MyAnyType) ([]MyAnyType, []MyAnyType) {
    mid := len(s)/2
    return s[:mid], s[mid:]
}

Congratulations on making it to the end!

If you're interested in doing the interactive coding assignments and quizzes for this course you can check out the Learn Go course over on Boot.dev.

This course is a part of my full back-end developer career path, made up of other courses and projects if you're interested in checking those out.

If you want to see the other content I'm creating related to web development, check out some of my links below:

• Lane's Podcast: Backend Banter

• Lane on Twitter

• Lane on YouTube

All the code samples for this course are in JavaScript, but the networking concepts you'll learn here apply generally to all coding languages. If you're not familiar with JavaScript yet, you can check out my JS course here.

I've included all the learning material you'll need here in this article, but if you would like a more hands-on experience, you can take the interactive version of this course with coding challenges on Boot.dev here.

I've also published a free video version of this course on the freeCodeCamp YouTube channel:

If you like this video, you can check out my other tutorials on my Boot.dev YouTube channel here.

All that said, let's jump into learning about HTTP!

Table of Contents

1. Why HTTP?

2. What is DNS?

3. What Are URIs?

4. Async/Await

5. Error Handling

6. HTTP Headers

7. What is JSON?

8. HTTP Methods

9. URL Paths and Parameters

10. What is HTTPs?

Why HTTP?

Communicating on the web

Instagram would be pretty terrible if you had to manually copy your photos to your friend's phone when you wanted to share them. Modern applications need to be able to communicate information between devices over the internet.

• Gmail doesn't just store your emails in variables on your computer, it stores them on computers in their data centers.

• You don't lose your Slack messages if you drop your computer in a lake – those messages exist on Slack's servers.

How does web communication work?

When two computers communicate with each other, they need to use the same rules. An English speaker can't communicate verbally with a Japanese speaker, and similarly, two computers need to speak the same language to communicate.

This "language" that computers use is called a protocol. The most popular protocol for web communication is HTTP, which stands for Hypertext Transfer Protocol.

Interacting with a server

In this course, a lot of the code samples will interact with the PokeAPI. It serves data about Pokémon.

Here's some code that retrieves a list of Pokémon from the PokeAPI:

const pokemonResp = await getItemData()

logPokemons(pokemonResp.results)

async function getItemData() {
  const response = await fetch('https://pokeapi.co/api/v2/pokemon/', {
    method: 'GET',
    mode: 'cors',
    headers: {
      'Content-Type': 'application/json'
    }
  })
  return response.json()
}

function logPokemons(pokemons) {
  for (const pokemon of pokemons) {
    console.log(pokemon.name)
  } 
}

When you run this code, you'll notice that none of the data that is logged to the console was generated within our code! That's because the data we retrieved is being sent over the internet from our servers via HTTP. Don't worry, we'll explain more about that later.

HTTP Requests and Responses

At the heart of HTTP is a simple request-response system. The "requesting" computer, also known as the "client", asks another computer for some information. That computer, the "server" sends back a response with the information that was requested.

We'll talk about the specifics of how the "requests" and "responses" are formatted later. For now, just think of it as a simple question-and-answer system.

• Request: "What are the items in the Fantasy Quest game?"

• Response: A list of the items in the Fantasy Quest game

HTTP powers websites

As we discussed, HTTP, or Hypertext Transfer Protocol, is a protocol designed to transfer information between computers.

There are other protocols for communicating over the internet, but HTTP is the most popular and is particularly great for websites and web applications.

Each time you visit a website, your browser is making an HTTP request to that website's server. The server responds with all the text, images, and styling information that your browser needs to render its pretty website.

HTTP URLs

A URL, or Uniform Resource Locator, is essentially the address of another computer, or "server" on the internet. Part of the URL specifies how to reach the server, and part of it tells the server what information we want - but more on that later.

For now, it's important to understand that a URL represents a piece of information on another computer that we want access to. We can get access to it by making a request, and reading the response that the server replies with.

How to Use URLs in HTTP

The http:// at the beginning of a website URL specifies that the http protocol will be used for communication.

Other communication protocols use URLs as well, (hence "Uniform Resource Locator"). That's why we need to be specific when we're making HTTP requests by prefixing the URL with http://

Requests and Responses Review

• A "client" is a computer making an HTTP request

• A "server" is a computer responding to an HTTP request

• A computer can be a client, a server, both, or neither. "Client" and "server" are just words we use to describe what computers are doing within a communication system.

• Clients send requests and receive responses

• Servers receive requests and send responses

JavaScript's Fetch API

In this course, we'll be using JavaScript's built-in fetch API to make HTTP requests.

The fetch() function is made available to us by the JavaScript language running in the browser. All we have to do is provide it with the parameters it requires.

How to Use Fetch

const response = await fetch(url, settings)
const responseData = await response.json()

We'll go in-depth on the various things happening in this standard fetch call later, but let's cover some basics for now.

• response is the data that comes back from the server

• url is the URL we are making a request to

• settings is an object containing some request-specific settings

• The await keyword tells JavaScript to wait until the request comes back from the server before continuing

• response.json() converts the response data from the server into a JavaScript object

See if you can spot the issue with this code snippet:

const pokemonResp = getItemData()

logPokemons(pokemonResp.results)

// the bug is above this line

async function getItemData() {
  const response = await fetch('https://pokeapi.co/api/v2/pokemon/', {
    method: 'GET',
    mode: 'cors',
    headers: {
      'Content-Type': 'application/json'
    }
  })
  return response.json()
}

function logPokemons(pokemons) {
  for (const pokemon of pokemons) {
    console.log(pokemon.name)
  } 
}

Hint: We're not waiting for the data to be sent back across the network.

Web Clients

A web client is a device making requests to a web server. A client can be any type of device but is often something users physically interact with. For example:

• A desktop computer

• A mobile phone

• A tablet

In a website or web application, we call the user's device the "front-end".
A front-end client makes requests to a back-end server.

Web Servers

Up to this point, most of the data you have worked with in your code has simply been generated and stored locally in variables.

While you'll always use variables to store and manipulate data while your program is running, most websites and apps use a web server to store, sort, and serve that data so that it sticks around for longer than a single session, and can be accessed by multiple devices.

Listening and serving data

Similar to how a server at a restaurant brings your food to the table, a web server) serves web resources, such as web pages, images, and other data. The server is turned on and "listening" for inbound requests constantly so that the second it receives a new request, it can send an appropriate response.

The server is the back-end

While the "front-end" of a website or web application is the device the user interacts with, the "back-end" is the server that keeps all the data housed in a central location. If you're still confused, check out this article comparing front-end and back-end development.

A server is just a computer

"Server" is just the name we give to a computer that is taking on the role of serving data across a network connection.

A good server is turned on and available 24 hours a day, 7 days a week. While your laptop can be used as a server, it makes more sense to use a computer in a data center that's designed to be up and running constantly.

What is DNS?

Web Addresses

In the real world, we use addresses to help us find where a friend lives, where a business is located, or where a party is being thrown.

In computing, web clients find other computers over the internet using Internet Protocol or IP addresses.

An IP address is a numerical label that serves two main functions:

1. Location Addressing

2. Network Identification

Domain names and IP Addresses

Each device connected to the internet has a unique IP address. When we browse the internet, the domains we navigate to are all associated with a particular IP address.

For example, this Wikipedia URL points to a page about miniature pigs: https://en.wikipedia.org/wiki/Miniature_pig
The domain portion of the URL is en.wikipedia.org. en.wikipedia.org converts to a specific IP address, and that IP address tells your computer exactly where to communicate with that Wikipedia page.

Cloudflare is a tech company that provides a cool public HTTP server that we can use to look up the IP address of any domain. Take a look at this sample code:

async function fetchIPAddress(domain) {
  const resp = await fetch(`https://cloudflare-dns.com/dns-query?name=${domain}&type=A`, {
    headers: {
      'accept': 'application/dns-json'
    }
  })
  const respObject = await resp.json()
  for (const record of respObject.Answer) {
    return record.data
  }
  return null
}

const domain = 'api.boot.dev'
const ipAddress = await fetchIPAddress(domain)
if (!ipAddress) {
  console.log('something went wrong in fetchIPAddress')
} else {
  console.log(`found IP address for domain ${domain}: ${ipAddress}`)
}

To recap, a "domain name" is part of a URL. It's the part that tells the computer where the server is located on the internet by being converted into a numerical IP address.

We'll cover exactly how an IP address is used by your computer to find a path to the server in a later networking course. For now, it's just important to understand that an IP address is what your computer is using at a lower level to communicate on a network.

Deploying a real website to the internet is actually quite simple. It involves only a couple of steps:

1. Create a server that hosts your website files and connect it to the internet

2. Acquire a domain name

3. Connect the domain name to the IP address of your server

4. Your server is accessible via the internet!

As we discussed, the "domain name" or "hostname" is part of a URL. We'll get to the other parts of a URL later.

For example, the URL https://example.com/path has a hostname of example.com. The https:// and /path portions are not part of the domain name -> IP address mapping that we've been learning about.

Using the URL API in JavaScript

The URL API is built into JavaScript. You can create a new URL object like this:

const urlObj = new URL('https://example.com/example-path')

And then you can extract just the hostname:

const urlObj.hostname

DNS Review

So we've talked about domain names and what their purpose is, but we haven't talked about the system that's used to do that conversion.

DNS, or the "Domain Name System", is the phonebook of the internet. Humans connect to websites through domain names, like Boot.dev.

DNS "resolves" these domain names to find the associated IP addresses so that web clients can load the resources for the specific address.

How does DNS Work?

We'll go into more detail on DNS in a future course, but to give you a simplified idea of how it works, let's introduce ICANN. ICANN is a not-for-profit organization that manages DNS for the entire internet.

Whenever your computer attempts to resolve a domain name, it contacts one of ICANN's "root nameservers" whose address is included in your computer's networking configuration.

From there, that nameserver can gather the domain records for a specific domain name from their distributed DNS database.

If you think of DNS as a phonebook, ICANN is the publisher that keeps the phonebook in print and available.

Subdomains

We learned about how a domain name resolves to an IP address, which is just a computer on a network - often the internet.

A subdomain prefixes a domain name, allowing a domain to route network traffic to many different servers and resources.

For example, the Boot.dev website is hosted on a different computer than our blog. Our blog, found at blog.boot.dev is hosted on our "blog" subdomain.

What are URIs?

We briefly touched on URLs earlier, but now let's dive a little deeper into the subject.

A URI, or Uniform Resource Identifier, is a unique character sequence that identifies a resource that is (almost always) accessed via the internet.

Just like JavaScript has syntax rules, so do URIs. These rules help ensure uniformity so that any program can interpret the meaning of the URI in the same way.

URIs come in two main types:

• URLs

• URNs

We will focus specifically on URLs in this course, but it's important to know that URLs are only one kind of URI.

URLs have quite a few sections, some of which are required, others not.
Let's use the URL API to parse a URL and print all the different parts. We'll learn more about each part later, for now, let's just split and print a URL.

function printURLParts(urlString) {
  const urlObj = new URL(urlString)
  console.log(`protocol: ${urlObj.protocol}`)
  console.log(`username: ${urlObj.username}`)
  console.log(`password: ${urlObj.password}`)
  console.log(`hostname: ${urlObj.hostname}`)
  console.log(`port: ${urlObj.port}`)
  console.log(`pathname: ${urlObj.pathname}`)
  console.log(`search: ${urlObj.search}`)
  console.log(`hash: ${urlObj.hash}`)
}

const fantasyQuestURL = 'http://dragonslayer:pwn3d@fantasyquest.com:8080/maps?sort=rank#id'
printURLParts(fantasyQuestURL)

Further dissecting a URL

There are 8 main parts to a URL, though not all the sections are always present. Each piece plays a specific role in helping clients locate the specified resource.
The 8 sections are:

• The protocol is required

• Usernames and passwords are optional

• A domain is required

• The default port for a given protocol is used if one is not provided

• The default ( / ) path is used if one isn't provided

• A query is optional

• A fragment is optional

Don't get too hung up on memorizing this stuff

Because names for the different sections are often used interchangeably, and because not all the parts of the URL are always present, it can be hard to keep things straight.

Don't worry about memorizing this stuff! Try to just get familiar with these URL concepts from a high level. Like any good developer, you can just look it up again the next time you need to know more.

The Protocol

The "protocol", also referred to as the "scheme", is the first component of a URL. Its purpose is to define the rules by which the data being communicated is displayed, encoded, or formatted.

Some examples of different URL protocols:

• http

• ftp

• mailto

• https

For example:

• http://example.com

• mailto:noreply@fantasyquest.app

Not all schemes require a "//"

The "http" in a URL is always followed by ://. All URLs have the colon, but the // part is only included for schemes that have an authority component.

As you can see above, the mailto scheme doesn't use an authority component, so it doesn't need the slashes.

URL Ports

The port in a URL is a virtual point where network connections are made. Ports are managed by a computer's operating system and are numbered from 0 to 65,535.

Whenever you connect to another computer over a network, you're connecting to a specific port on that computer, which is being listened to by a specific piece of software on that computer. A port can only be used by one program at a time, which is why there are so many possible ports.

The port component of a URL is often not visible when browsing normal sites on the internet, because 99% of the time you're using the default ports for the HTTP and HTTPS schemes: 80 and 443, respectively.

Whenever you aren't using a default port, you need to specify it in the URL. For example, port 8080 is often used by web developers when they're running their server in "test mode" so that they don't use the "production" port "80".

URL Paths

In the early days of the internet, a URL's path often was a reflection of the file path on the server to the resource the client was requesting.

For example, if the website https://exampleblog.com had a web server running within its /home directory, then a request to the https://exampleblog.com/site/index.html URL might expect the index.html file from within the /home/site directory to be returned.

Websites used to be very simple. They were just a collection of text documents stored on a server. A simple piece of server software could handle incoming HTTP requests and respond with the documents according to the path component of the URLs.

These days, it's not always about the filesystem

On many modern web servers, a URL's path isn't a reflection of the server's filesystem hierarchy. Paths in URLs are essentially just another type of parameter that can be passed to the server when making a request.

Conventionally, two different URL paths should denote different resources. For example, different pages on a website, or maybe different data types from a game server.

Query parameters

Query parameters in a URL are not always present. In the context of websites, query parameters are often used for marketing analytics or for changing a variable on the web page. With website URLs, query parameters rarely change which page you're viewing, though they often will change the page's contents.

That said, query parameters can be used for anything the server chooses to use them for, just like the URL's path.

How Google uses query parameters

1. Open a new tab and go to google.com

2. Search for "hello world"

3. Take a look at your current URL. It should start with https://www.google.com/search?q=hello+world

4. Change the URL to say https://www.google.com/search?q=hello+universe

5. Press "enter"

You should see new search results for the query "hello universe". Google chose to use query parameters to represent the value of your search query. It makes sense - each search result page is essentially the same page as far as structure and formatting are concerned - it's just showing you different results based on the search query.

Async/Await

You're probably already familiar with synchronous code, which means code that runs in sequence. Each line of code executes in order, one after the next.

Example of synchronous code:

console.log("I print first");
console.log("I print second");
console.log("I print third");

Asynchronous or async code runs in parallel. That means code further down runs at the same time that a previous line of code is still running. A good way to visualize this is with the JavaScript function setTimeout().

setTimeout accepts a function and a number of milliseconds as inputs. It waits until the number of milliseconds has elapsed, and then it executes the function it was given.

Example of asynchronous code:

jsconsole.log("I print first");
setTimeout(() => console.log("I print third because I'm waiting 100 milliseconds"), 100);
console.log("I print second");

Try altering the waiting times in the async code below to get the messages to print in the proper order:

const craftingCompleteWait = 0
const combiningMaterialsWait = 0
const smeltingIronBarsWait = 0
const shapingIronWait = 0

// Don't touch below this line

setTimeout(() => console.log('Iron Longsword Complete!'), craftingCompleteWait)
setTimeout(() => console.log('Combining Materials...'), combiningMaterialsWait)
setTimeout(() => console.log('Smelting Iron Bars...'), smeltingIronBarsWait)
setTimeout(() => console.log('Shaping Iron...'), shapingIronWait)

console.log('Firing up the forge...')

await sleep(2500)
function sleep(ms) {
  return new Promise((resolve) => setTimeout(resolve, ms))
}

Expected order:

1. Firing up the forge..

2. Smelting Iron Bars...

3. Combining Materials...

4. Shaping Iron...

5. Iron Longsword Complete!

Why do we want async code?

We try to keep most of our code synchronous because it's easier to understand, and therefore often has fewer bugs. But sometimes we need our code to be asynchronous.

For example, whenever you update your user settings on a website, your browser will need to communicate those new settings to the server. The time it takes your HTTP request to physically travel across all the wiring of the internet is usually around 100 milliseconds. It would be a very poor experience if your webpage were to freeze while waiting for the network request to finish. You wouldn't even be able to move the mouse while waiting!

By making network requests asynchronously, we let the webpage execute other code while waiting for the HTTP response to come back. This keeps the user experience snappy and user-friendly.

As a general rule, we should only use async code when we need to for performance reasons. Synchronous code is simpler.

Promises in JavaScript

A Promise in JavaScript is very similar to making a promise in the real world. When we make a promise, we are making a commitment to something.

For example, I promise to explain JavaScript promises to you. My promise to you has 2 potential outcomes: it is either fulfilled, meaning I eventually explained promises to you, or it is rejected, meaning I failed to keep my promise and didn't explain promises.

The Promise Object represents the eventual fulfillment or rejection of our promise and holds the resulting values. In the meantime, while we're waiting for the promise to be fulfilled, our code continues executing.

Promises are the most popular modern way to write asynchronous code in JavaScript.

How to Declare a Promise

Here is an example of a promise that will resolve and return the string "resolved!" or reject and return the string "rejected!" after 1 second.

const promise = new Promise((resolve, reject) => {
  setTimeout(() => {
    if (getRandomBool()) {
      resolve("resolved!")
    } else {
      reject("rejected!")
    }
  }, 1000)
})

function getRandomBool(){
  return Math.random() < .5
}

How to Use a Promise

Now that we've created a promise, how do we use it?

The Promise object has .then and .catch that make it easy to work with. Think of .then as the expected follow-up to a promise, and .catch as the "something went wrong" follow-up.

If a promise resolves, its .then function will execute. If the promise rejects, its .catch method will execute.

Here's an example of using .then and .catch with the promise we made above:

promise.then((message) => {
    console.log(`The promise finally ${message}`)
}).catch((message) => {
    console.log(`The promise finally ${message}`)
})

// prints:
// The promise finally resolved!
// or
// the promise finally rejected!

Why are Promises useful?

Promises are the cleanest (but not the only) way to handle the common scenario where we need to make requests to a server, which is typically done via an HTTP request. In fact, the fetch() function we were using earlier in the course returns a promise!

I/O, or "input/output"

Almost every time you use a promise in JavaScript it will be to handle some form of I/O. I/O, or input/output, refers to when our code needs to interact with systems outside of the (relatively) simple world of local variables and functions.
Common examples of I/O include:

• HTTP requests

• Reading files from the hard drive

• Interacting with a Bluetooth device

Promises help us perform I/O without forcing our entire program to freeze up while we wait for a response.

Promises and the "await" keyword

We have used the await keyword a few times in this course, so it's time we finally understand what's going on under the hood.

The await keyword is used to wait for a promise to resolve. Once it has been resolved, the await expression returns the value of the resolved Promise.

Example with .then callback

promise.then((message) => {
  console.log(`Resolved with ${message}`)
}).

Example of awaiting a promise

const message = await promise
console.log(`Resolved with ${message}`)

The async keyword

While the await keyword can be used in place of .then() to resolve a promise, the async keyword can be used in place of new promise() to create a new promise.

When a function is prefixed with the async keyword, it will automatically return a promise. That promise resolves with the value that your code returns from the function. You can think of async as "wrapping" your function within a promise.

These are equivalent:

New Promise()

function getPromiseForUserData(){
  return new Promise((resolve) => {
    fetchDataFromServerAsync().then(function(user){
      resolve(user)
    })
  })
}

const promise = getPromiseForUserData()

Async

async function getPromiseForUserData(){
  const user = await fetchDataFromServer()
  return user
}

const promise = getPromiseForUserData()

.then() vs await

In the early days of web browsers, promises and the await keyword didn't exist, so the only way to do something asynchronously was to use callbacks.

A "callback function" is a function that you hand to another function. That function then calls your callback later on. The setTimeout function we've used in the past is a good example.

function callbackFunction(){
  console.log("calling back now!")
}
const milliseconds = 1000
setTimeout(callbackFunction, milliseconds)

While even the .then() syntax is generally easier to use than callbacks without the Promise API, the await syntax makes them even easier to use. You should use async and await over .then and new Promise() as a general rule.

To demonstrate, which of these is easier to understand?

fetchUser.then(function(user){
  return fetchLocationForUser(user)
}).then(function(location){
  return fetchServerForLocation(location)
}).then(function(server){
  console.log(`The server is ${server}`)
});

const user = await fetchUser()
const location = await fetchLocationForUser(user)
const server = await fetchServerForLocation(location)
console.log(`The server is ${server}`)

They both do the same thing, but the second example is so much easier to understand! The async and await keywords weren't released until after the .then API, which is why there is still a lot of legacy .then() code out there.

Error Handling

When something goes wrong while a program is running, JavaScript uses the try/catch paradigm for handling those errors. Try/catch is fairly common, and Python uses a similar mechanism.

First, an error is thrown

For example, let's say we try to access a property on an undefined variable. JavaScript will automatically "throw" an error.

const speed = car.speed
// The code crashes with the following error:
// "ReferenceError: car is not defined"

Trying and catching errors

By wrapping that code in a try/catch block, we can handle the case where car is not yet defined.

try {
  const speed = car.speed
} catch (err) {
  console.log(`An error was thrown: ${err}`)
  // the code cleanly logs:
  // "An error was thrown: ReferenceError: car is not defined"
}

Bugs vs Errors

Error handling via try/catch is not the same as debugging. Likewise, errors are not the same as bugs.

• Good code with no bugs can still produce errors that are gracefully handled

• Bugs are, by definition, bits of code that aren't working as intended

What is Debugging?

"Debugging" a program is the process of going through your code to find where it is not behaving as expected. Debugging is a manual process performed by the developer.

Examples of debugging:

• Adding a missing parameter to a function call

• Updating a broken URL that an HTTP call was trying to reach

• Fixing a date-picker component in an app that wasn't displaying properly

What is Error Handling?

"Error handling" is code that can handle expected edge cases in your program. Error handling is an automated process that we design into our production code to protect it from things like weak internet connections, bad user input, or bugs in other people's code that we have to interface with.

Examples of error handling:

• Using a try/catch block to detect an issue with user input

• Using a try/catch block to gracefully fail when no internet connection is available

In short, don't use try/catch to try to handle bugs

If your code has a bug, try/catch won't help you. You need to just go find the bug and fix it.

If something out of your control can produce issues in your code, you should use try/catch or other error-handling logic to deal with it.

For example, there could be a prompt in a game for users to type in a new character name, but we don't want them to use punctuation. Validating their input and displaying an error message if something is wrong with it would be a form of "error handling".

async/await makes error handling easier

try and catch are the standard way to handle errors in JavaScript, the trouble is, the original Promise API with .then didn't allow us to make use of try and catch blocks.

Luckily, the async and await keywords do allow it - yet another reason to prefer the newer syntax.

.catch() callback on promises

The .catch() method works similarly to the .then() method, but it fires when a promise is rejected instead of resolved.

Example with .then and .catch callbacks

fetchUser().then(function(user){
  console.log(`user fetched: ${user}`)
}).catch(function(err){
  console.log(`an error was thrown: ${err}`)
});

Example of awaiting a promise

try {
  const user = await fetchUser()
  console.log(`user fetched: ${user}`)
} catch (err) {
  console.log(`an error was thrown: ${err}`)
}

As you can see, the async/await version looks just like normal try/catch JavaScript.

HTTP Headers

An HTTP header allows clients and servers to pass additional information with each request or response. Headers are just case-insensitive key-value pairs that pass additional metadata about the request or response.

HTTP requests from a web browser carry with them many headers, including but not limited to:

• The type of client (for example Google Chrome)

• The Operating system (for example Windows)

• The preferred language (for example US English)

As developers, we can also define custom headers in each request.

Headers API

The Headers API allows us to perform various actions on our request and response headers such as retrieving, setting, and removing them. We can access the headers object through the Request.headers and Response.headers properties.

How to Use the Browser's Developer Tools

Modern web browsers offer developers a powerful set of developer tools. The Developer Tools are a front-end web developer's best friend. For example, using the dev tools you can:

• View the web page's JavaScript console output

• Inspect the page's HTML, CSS, and JavaScript code

• View network requests and responses, along with their headers

The method for accessing dev tools varies from browser to browser. If you're on Chrome, you can just right-click anywhere within a web page and click the "inspect" option. Follow this link for more info on how to access dev tools.

The network tab

While all of the tabs within the dev tools are very useful, we will be focusing specifically on the Network tab in this chapter so we can play with HTTP headers.

The Network tab monitors your browser's network activity and records all of the requests and responses the browser is making, including how long each of those requests and responses takes to fully process.

If you navigate to the Network tab and don't see any requests appear, try refreshing the page.

Why are headers useful?

Headers are useful for several reasons, from design to security. But most often headers are used as metadata or data about the request.

So, for example, let's say we wanted to ask for a player's level from a game's server. We need to send that player's ID to the server so it knows which player to send back the information for. That ID is my request, it's not information about my request.

A good example of a use case for headers is authentication. Often times a user's credentials are included in request headers. Credentials don't have much to do with the request itself, but simply authorize the requester to be allowed to make the request in question.

What is JSON?

JavaScript Object Notation, or JSON, is a standard for representing structured data based on JavaScript's object syntax.

JSON is commonly used to transmit data in web apps using HTTP. The HTTP fetch() requests we have been using in this course have been returning data as JSON data.

JSON Syntax

Because we already understand what JavaScript objects look like, understanding JSON is easy. JSON is just a stringified JavaScript object. The following is valid JSON data:

{
    "movies": [
        {
            "id": 1,
            "genre": "Action",
            "title": "Iron Man",
            "director": "Jon Favreau"
        },
        {
            "id": 2,
            "genre": "Action",
            "title": "The Avengers",
            "director": "Joss Whedon"
        }
    ]
}

How to Parse HTTP Responses as JSON

JavaScript provides us with some easy tools to help us work with JSON. After making an HTTP request with the fetch() API, we get a Response object. That response object offers us some methods that help us interact with the response.

One such method is the .json() method. The .json() method takes the response stream returned by a fetch request and returns a Promise that resolves into a JavaScript object parsed from the JSON body of the HTTP response.

const resp = await fetch(...)
const javascriptObjectResponse = await resp.json()

It's important to note that the result of the .json() method is NOT JSON. It is the result of taking JSON data from the HTTP response body and parsing that input into a JavaScript Object.

JSON Review

JSON is a stringified representation of a JavaScript object, which makes it perfect for saving to a file or sending in an HTTP request.

Remember, an actual JavaScript object is something that exists only within your program's variables. If we want to send an object outside our program, for example, across the internet in an HTTP request, we need to convert it to JSON first.

It's not just used in JavaScript

Just because JSON is called JavaScript Object Notation doesn't mean it's only used by JavaScript! JSON is a common standard that is recognized and supported by every major programming language.

For example, even though Boot.dev's backend API is written in Go, we still use JSON as the communication format between the front-end and backend.

By the way, this course has been about interacting with back-end servers from a front-end perspective. But if you're curious about how you can become a back-end engineer, check out this guide I've put together. For reference, it takes most people between 6-18 months to learn enough to get their first back-end job.

Common JSON use-cases

• In HTTP request and response bodies

• As formats for text files. .json files are often used as configuration files.

• In NoSQL databases like MongoDB, ElasticSearch, and Firestore

How to Pronounce JSON

I pronounce it "Jay-sawn", but I've also heard people pronounce it "Jason", like the name.

How to Send JSON

JSON isn't just something we get from the server, we can also send JSON data.
In JavaScript, two of the main methods we have access to are JSON.parse(), and JSON.stringify().

JSON.stringify()

JSON.stringify() is particularly useful for sending JSON.

As you may expect, the JSON stringify() method does the opposite of parse. It takes a JavaScript object or value as input and converts it into a string. This is useful when we need to serialize the objects into strings to send them to our server or store them in a database.

Here's a snippet of code that sends a JSON payload to a remote server:

async function sendPayload(data, headers) {
  const response = await fetch(url, {
    method: 'POST',
    mode: 'cors',
    headers: headers,
    body: JSON.stringify(data)
  })
  return response.json()
}

How to Parse JSON

The JSON.parse() method takes a JSON string as input and constructs the JavaScript value/object described by the string. This allows us to work with the JSON as a normal JavaScript object.

Seeing as JSON objects have a tree-like structure, it can be helpful to know how to traverse them recursively if needs be.

const json = '{"title": "Avengers Endgame", "Rating":4.7, "inTheaters":false}';
const obj = JSON.parse(json)

console.log(obj.title)
// Avengers Endgame

XML

We can't talk about JSON without mentioning XML. Extensible Markup Language, or XML is a text-based format for representing structured information, just like JSON - it just looks a bit different.

XML is a markup language like HTML, but it's more generalized in that it does not use predefined tags. Just like how in a JSON object keys can be called anything, XML tags can also have any name.

<root>
  <id>1</id>
  <genre>Action</genre>
  <title>Iron Man</title>
  <director>Jon Favreau</director>
</root>

The same data in JSON form:

{
  "id": "1",
  "genre": "Action",
  "title": "Iron Man",
  "director": "Jon Favreau"
}

Why use XML?

XML and JSON both accomplish similar tasks, so which should you use?

XML used to be used for the same things that today JSON is used for. Configuration files, HTTP bodies, and other data-transfer use cases can work just fine using JSON or XML. Here's my advice: generally speaking, if JSON works, you should favor it over XML these days. JSON is lighter-weight, easier to read, and has better support in most modern programming languages.

There are some cases where XML might still be the better, or maybe even the necessary choice, but those cases will be rare.

HTTP Methods

HTTP defines a set of methods that we use every time we make a request. We have used some of these methods in previous exercises, but it's time we dive into them and understand the differences and use cases behind the different methods.

The GET method

The GET method is used to "get" a representation of a specified resource. You are not taking the data away from the server, but rather getting a representation, or copy, of the resource in its current state.

A get request is considered a safe method to call multiple times because it doesn't alter the state of the server.

How to Make a GET Request using the Fetch API

The fetch() method accepts an optional init object parameter as its second argument that we can use to define things like:

• method: The HTTP method of the request, like GET.

• headers: The headers to send.

• mode: Used for security, we'll talk about this in future courses

• body: The body of the request. Often encoded as JSON.

Example GET request using fetch:

await fetch(url, {
  method: 'GET',
  mode: 'cors',
  headers: {
    'sec-ch-ua-platform': 'macOS'
  }
})

Why do we use HTTP methods?

As we touched on before, the primary purpose of HTTP methods is to indicate to the server what we want to do with the resource we're trying to interact with.

At the end of the day, an HTTP method is just a string, like GET, POST, PUT, or DELETE. But by convention, backend developers almost always write their server code so that the methods correspond with different "CRUD" actions.

The "CRUD" actions are:

• Create

• Read

• Update

• Delete

The bulk of the logic in most web applications is "CRUD" logic. The web interface allows users to create, read, update and delete various resources.

Think of a social media site - users are basically creating, reading, updating and deleting their social posts. They are also creating, reading, updating and deleting their user accounts. It's CRUD all the way down!

As it happens, the 4 most common HTTP methods map nicely to the CRUD actions:

• POST = create

• GET = read

• PUT = update

• DELETE = delete

POST Requests

An HTTP POST request sends data to a server, typically to create a new resource. The body of the request is the payload that is being sent to the server with the request. Its type is indicated by the Content-Type header.

How to Add a body

The body of the request is the payload that is being sent to the server with the request. Its type is indicated by the Content-Type header - for us, that's going to be JSON.

POST requests are generally not safe methods to call multiple times, because it alters the state of the server. You wouldn't want to accidentally create 2 accounts for the same user, for example.

await fetch(url, {
  method: 'POST',
  mode: 'cors',
  headers: {
    'Content-Type': 'application/json'
  },
  body: JSON.stringify(data)
})

HTTP Status Codes

Now that we understand how to write HTTP requests from scratch, we need to learn how to ensure that the server is doing what we want.

Earlier in the course, we learned about how to access the browser's developer tools and use those tools to inspect HTTP requests. We can use that same process to check on the requests we are making and verify what they are doing so we can address potential problems.

When looking at requests, we can check on the Status Code of the request to get some information if the request was successful or not.

• 100-199: Informational responses. These are very rare.

• 200-299: Successful responses. Hopefully, most responses are 200's!

• 300-399: Redirection messages. These are typically invisible because the browser or HTTP client will automatically do the redirect.

• 400-499: Client errors. You'll see these often, especially when trying to debug a client application

• 500-599: Server errors. You'll see these sometimes, usually only if there is a bug on the server.

Here are some of the most common status codes, but you can see a full list here if you're interested.

• 200 - OK. This is by far the most common code, it just means that everything worked as expected.

• 201 - Created. This means that a resource was created successfully. Typically in response to a POST request.

• 301 - Moved permanently. This means the resource was moved to a new place, and the response will include where that new place is. Websites often use 301 redirects when they change their domain name, for example.

• 400 - Bad request. A general error indicating the client made a mistake in their request.

• 403 - Unauthorized. This means the client doesn't have the correct permissions. Maybe they didn't include a required authorization header, for example.

• 404 - Not found. You'll see this on websites quite often. It just means the resource doesn't exist.

• 500 - Internal server error. This means something went wrong on the server, likely a bug on their end.

You don't need to memorize them

You need to know the basics, like "2XX is good", "4XX is a client error", and "5XX is a server error". That said, you don't need to memorize all the codes, they're easy to look up.

Let's check some status codes!

The .status property on a Response object will give you the code. Here's an example:

async function getStatusCode(url, headers) {
  const response = await fetch(url, {
    method: 'GET',
    mode: 'cors',
    headers: headers
  })
  return response.status
}

PUT Method

The HTTP PUT method creates a new resource or replaces a representation of the target resource with the contents of the request's body. In short, it updates a resource's properties.

await fetch(url, {
   method: 'PUT',
   mode: 'cors',
   headers: {
   'Content-Type': 'application/json'
   },
   body: JSON.stringify(data)
})

POST vs PUT

You may be thinking PUT is similar to POST or PATCH, and frankly, you'd be right. The main difference is that PUT is meant to be idempotent, meaning multiple identical PUT requests should have the same effect on the server.

In contrast, several identical POST requests would have additional side effects, such as creating multiple copies of the resource.

HTTP Patch vs PUT

You may encounter PATCH methods from time to time. While it is not nearly as common as the other methods, like PUT, it's important to know about it and what it does. The PATCH method is intended to partially modify a resource.

Long story short, PATCH isn't nearly as popular as PUT, and many servers, even if they allow partial updates, will still just use the PUT method for that.

HTTP Delete

The DELETE method does exactly as you would expect: it's conventionally used to delete a specified resource.

// This deletes the location with ID: 52fdfc07-2182-454f-963f-5f0f9a621d72
const url = 'https://example-api.com/locations/52fdfc07-2182-454f-963f-5f0f9a621d72'

await fetch(url, {
  method: 'DELETE',
  mode: 'cors'
})

URL Paths and Parameters

The URL Path comes right after the domain (or port if one is provided) in a URL string.

In this URL, the path is /root/next: http://testdomain.com/root/next.

What paths meant in the early internet

In the early days of the internet, and sometimes still today, many web servers simply served raw files from the server's file system.

For example, if I wanted you to be able to access some text documents, I could start a web server in my documents directory. If you made a request to my server, you would be able to access different documents by using the path that matched my local file structure.

If I had a file in my local /documents/hello.txt, you could access it by making a GET request to http://example.com/documents/hello.txt.

How paths are used today

Most modern web servers don't use that simple mapping of URL path -> file path. Technically, a URL path is just a string that the web server can do what it wants with, and modern websites take advantage of that flexibility.

Some common examples of what paths are used for include:

• The hierarchy of pages on a website, whether or not that reflects a server's file structure

• Parameters being passed into an HTTP request, like an ID of a resource

• The version of the API

• The type of resource being requested

RESTful APIs

Representational State Transfer, or REST, is a popular convention that HTTP servers follow. Not all HTTP APIs are "REST APIs", or "RESTful", but it is very common.

RESTful servers follow a loose set of rules that makes it easy to build reliable and predictable web APIs. REST is more or less a set of conventions about how HTTP should be used.

Separate and agnostic

The big idea behind REST is that resources are transferred via well-recognized, language-agnostic client/server interactions.

A RESTful style means the implementation of the client and server can be done independently of one another, as long as some simple standards, like the names of the available resources, have been established.

Stateless

A RESTful architecture is stateless. This means that the server does not need to know what state the client is in, nor does the client need to know what state the server is in.

Statelessness in REST is enforced by interacting with resources instead of commands. Keep in mind, this doesn't mean the applications are stateless - on the contrary, what would "updating a resource" even mean if the server wasn't keeping track of its state?

Paths in REST

In a RESTful API, the last section of the path of a URL should specify which resource is being accessed. Then, as we talked about in the "methods" section, depending on whether the request is a GET, POST, PUT or DELETE, the resource is read, created, updated, or deleted.

For example, in the PokeAPI:

• https://pokeapi.co/api/v2/pokemon/

• https://pokeapi.co/api/v2/location/

The first part of the path specifies that we're interacting with an API rather than a website. The next part specifies the version, in this case, version 2, or v2.

Finally, the last part denotes which resource is being accessed, be it a location or pokemon.

URL Query Parameters

A URL's query parameters appear next in the URL structure but are not always present - they're optional. For example:

https://www.google.com/search?q=boot.dev

q=boot.dev is a query parameter. Like headers, query parameters are key / value pairs. In this case, q is the key and boot.dev is the value.

The Documentation of an HTTP Server

You may be wondering:

How the heck am I supposed to memorize how all these different servers work???

The good news is that you don't need to. When you work with a backend server, it's the responsibility of that server's developers to provide you with instructions, or documentation that explains how to interact with it.

For example, the documentation should tell you:

• The domain of the server

• The resources you can interact with (HTTP paths)

• The supported query parameters

• The supported HTTP methods

• Anything else you'll need to know to work with the server

The server has control

As we mentioned earlier, the server has complete control over how the path in a URL is interpreted and used in a request. The same goes for query parameters.

Not all servers support query parameters for every type of request, it just depends, so you'll need to consult the docs.

Multiple Query Parameters

We mentioned that query parameters are key/value pairs - that means we can have multiple pairs.

http://example.com?firstName=lane&lastName=wagner

In the example above:

• firstName = lane

• lastName = wagner

The ? separates the query parameters from the rest of the URL. The & is then used to separate every pair of query parameters after that.

For example, make this request to limit the number of Pokémon returned from the PokeAPI to 2:

https://pokeapi.co/api/v2/location/?limit=2

What is HTTPs?

Hypertext Transfer Protocol Secure or HTTPS is an extension of the HTTP protocol. HTTPS secures the data transfer between client and server by encrypting all of the communication.

HTTPS allows a client to safely share sensitive information with the server through an HTTP request, such as credit card information, passwords, or bank account numbers.

HTTPS requires that the client use SSL or TLS to protect requests and traffic by encrypting the information in the request. HTTPS is just HTTP with extra security.

HTTP vs HTTPS

HTTPS keeps your messages private, but not your identity

We won't cover how encryption works in this course, but we will in later courses. For now, it's important to note that while HTTPS encrypts what you are saying, it doesn't necessarily protect who you are. Tools like VPNs are needed for remaining anonymous online.

HTTPS ensures that you're talking to the right person (or server)

In addition to encrypting the information within a request, HTTPS uses digital signatures to prove that you're communicating with the server that you think you are.

If a hacker were to intercept an HTTPS request by tapping into a network cable, they wouldn't be able to successfully pretend they are your bank's web server.

Assuming that a server supports HTTPs, you use it by simply changing the protocol on your request URL: https://boot.dev

Want to put what you've learned into practice with a project?

Check out this project guide where you'll build a web crawler in JavaScript from scratch. It will have you using the Fetch API and parsing JSON data like a pro! You don't need to build the project, but it's a great way to practice what you're learned.

Congratulations on making it to the end!

If you're interested in doing the interactive coding assignments and quizzes for this course you can check out the Learn HTTP course over on Boot.dev.

This course is a part of my full back-end developer career path, made up of other courses and projects if you're interested in checking those out.

If you want to see the other content I'm creating related to web development, check out some of my links below:

• Lane on Twitter

• Lane on YouTube

• Lane's Personal Site

The main benefit of writing your code in an object-oriented way is to structure your program into simple, reusable pieces of code.

Stick with me through this article and you'll have a full understanding of the core tenets of OOP by the end. All the coding examples will be in Python, but the concepts apply generally to all coding languages.

I've included all the learning material you'll need here in this article. But if you would like to go more in-depth with live coding exercises and quizzes, you can find those here on Boot.dev.

Table of Contents

1. The Goal of Object Oriented Programming
2. Classes in OOP
3. OOP Pillar #1 –Encapsulation
4. OOP Pillar #2 –Abstraction
5. OOP Pillar #3 –Inheritance
6. OOP Pillar #4 –Polymorphism

The Goal of OOP is Cleaner Code

Object oriented programming and other paradigms like functional programming are all about making code easier to work with and understand. We call code that is easy to work with "clean code".

Any fool can write code that a computer can understand. Good programmers write code that humans can understand. – Martin Fowler

Clean code is not:

• A way to make your programs run faster
• A way to make your program use less memory
• Necessary to create certain kinds of programs
• Strictly better than non-OOP code

Clean code is:

• Designed to make code easier to work with in many situations
• Something that helps humans model and simulate the real world
• A way to make finding and fixing bugs easier
• A way to make new feature development faster
• The best way to stay sane as a software engineer

A couple of examples of "clean code" practices include writing good comments, using DRY code, and naming variables well, just to name a few.

OOP is a Way to Write DRY Code

Let's pretend we have some code that looks like this:

soldier_one_dps = soldier_one["damage"] * soldier_one["attacks_per_second"]

soldier_two_dps = soldier_two["damage"] * soldier_two["attacks_per_second"]

We can use a function to refactor the code a bit:

def get_soldier_dps(soldier):
    return soldier["damage"] * soldier["attacks_per_second"]

soldier_one_dps = get_soldier_dps(soldier_one)
soldier_two_dps = get_soldier_dps(soldier_two)

We don't want too much of our code doing the exact same thing. When code is duplicated, it leads to many potential problems. In our example, let's pretend the soldier dictionary changed, and now the key that stores the "damage" value is called dmg.

In the first example, we would need to update two lines of code. In the second example, we only need to make the change in one place.

It's not a big deal when two lines are the same and exist right next to each other. However, imagine if we had done this hundreds of times in ten or twenty different code files! All of sudden, it makes a lot sense to stop repeating yourself and write more reusable functions. We call that DRY (don't repeat yourself) code.

Classes Allow for Even More Reusability

A class is a special type of value in an object-oriented programming language like Python. Just like a string, integer, or float, a class is a custom type that has some special properties.

An object is just an instance of a class type. "Instance" is just a big word for "one of a thing". For example, here, health is an instance of an integer type.

health = 50

How do I create a class?

In Python you just need to use the class keyword, and you can set custom properties in the following way.

class Soldier:
    health = 5

Then to create an instance of a Soldier we simply call the class. Notice that a class isn't a function, and it doesn't take input parameters directly.

first_soldier = Soldier()
print(first_soldier.health)
# prints "5"

Methods on a class

You might be wondering why classes are useful – they seem like regular Python dictionaries but worse!

What makes classes really cool is that they allow us to define custom methods on them. A method is a function that is associated with a class, and it has access to all the properties of the object.

class Soldier:
    health = 5

    def take_damage(self, damage):
        self.health -= damage

soldier_one = Soldier()
soldier_one.take_damage(2)
print(soldier_one.health)
# prints "3"

The special "self" value

As you can see, methods are nested within the class declaration. Methods always take a special parameter as their first argument called self. The self variable is a reference to the object itself, so by using it you can read and update the properties of the object.

Notice that methods are called directly on an object using the dot operator.

object.method()

How to return values from a method

If a regular function doesn't return anything, it's typically not a very useful function. But methods often don't return anything explicitly because they often mutate the properties of the object instead.

However, they can return values as well!

class Soldier:
    armor = 2
    num_weapons = 2

    def get_speed(self):
        speed = 10
        speed -= self.armor
        speed -= self.num_weapons
        return speed

soldier_one = Soldier()
print(soldier_one.get_speed())
# prints "6"

Methods vs Functions

A function is a piece of code that is called by a name. You can pass it data to operate on via parameters and it can optionally return data. All data that is passed to a function is explicitly passed through parameters.

A method is a piece of code that is called by a name that is associated with an object. Methods and functions are similar but have two key differences.

1. A method is implicitly passed the object on which it was called. In other words, you won't see all the inputs in the parameter list
2. A method is able to operate on data that is contained within the class. In other words, you won't see all the outputs in the return statement.

The OOP Debate

Because functions are more explicit, some developers argue that functional programming is better than object oriented programming. In reality, neither paradigm is "better", and the best developers learn and understand both styles and use them as they see fit.

For example, while methods are more implicit and often make code more difficult to read, they also make it easier to group a program's data and behavior together in one place. This can lead to a codebase that's more organized. The tradeoff is one of readability at the file level for readability at the project level.

Constructors in Python

It's quite rare in the real-world to see a class that defines properties in the way we've been doing it so far.

class Soldier:
    armor = 2
    num_weapons = 2

It's much more practical to use a constructor. In Python, a constructor is made with the init() method, and it is automatically called when a new object is created. So, with a constructor the code would look like this.

class Soldier:
    def __init__(self):
        self.armor = 2
        self.num_weapons = 2

However, because the constructor is a method, we can now make the starting armor and number of weapons configurable with some parameters.

class Soldier:
    def __init__(self, armor, num_weapons):
        self.armor = armor
        self.num_weapons = num_weapons

soldier = Soldier(5, 10)
print(soldier.armor)
# prints "5"
print(soldier.num_weapons)
# prints "10"

Class Variables vs Instance Variables

So far we've worked with both class variables and instance variables, but we haven't really talked about the difference yet.

Instance variables

Instance variables vary from object to object and are declared in the constructor.

class Wall():
    def __init__(self):
        self.height = 10

south_wall = Wall()
south_wall.height = 20 # only updates this instance of a wall
print(south_wall.height)
# prints "20"

north_wall = Wall()
print(north_wall.height)
# prints "10"

Class variables

Class variables remain the same between instances of the same class and are declared at the top-level of a class.

class Wall():
    height = 10

south_wall = Wall()
print(south_wall.height)
# prints "10"

Wall.height = 20 # updates all instances of a Wall

print(south_wall.height)
# prints "20"

Class vs instance variables – which should I use?

Generally speaking, stay away from class variables. Just like global variables, class variables are usually a bad idea because they make it hard to keep track of which parts of your program are making data updates.

However, it is important to understand how they work because you may see them out in the wild.

The Four Pillars of OOP

OOP Pillar #1 – Encapsulation

Encapsulation is one of the strongest tools in your tool belt as a software engineer. Like we covered in the beginning of this tutorial, writing code that machines understand is easy. But writing code that humans can understand is very difficult.

Encapsulation is the practice of hiding information inside of a "black box" so that other developers working with the code don't have to worry about it.

A basic example of encapsulation would be a function. The caller of a function doesn't need to worry too much about what happens inside – they just need to understand the inputs and outputs.

pythonacceleration = calc_acceleration(initial_speed, final_speed, time)

In the example above, to use the calc_acceleration function, we don't really need to understand what goes on inside. That's the goal of encapsulation: it makes our lives easier as developers and helps us write cleaner code.

Encapsulation in OOP

In the context of object-oriented programming, we can practice good encapsulation by using private and public members.

The idea is that if we want the users of our class to interact with something directly, we make it public. If they don't need to use a certain method or property, we make that private in order to keep the usage instructions for our class simple.

Encapsulation in Python

In order to enforce encapsulation in Python, developers prefix properties and classes that they intend to be private with a double underscore.

class Wall():
    def __init__(self, height):
        # this stops us from accessing the __height
        # property directly on an instance of a Wall
        self.__height = height

    def get_height(self):
        return self.__height

In the example above, we don't want users of the Wall class to be able to change its height. We make the __height property private and expose a public get_height method so that users can still read the height of a wall without being tempted to update it.