Go Language - freeCodeCamp.org

How to Create Dynamic Emails in Go with React Email

Orim Dominic Adah — Mon, 20 Apr 2026 20:16:44 +0000

Backend applications are required to send emails to users to deliver notifications and maintain communication outside the application interface. These emails usually contain information specific to each user, such as the user's name or address, making them dynamic.

This article walks you through building a dynamic email template with React Email, converting it to HTML, and injecting data into it using Go templates. It also contains an optional section that shows you how to send and test the email delivery with MailHog.

To follow along with this article, you need to have Go and Node.js installed on your computer. You should also have a basic understanding of React and some familiarity with Go templates, though these aren't strict requirements because you can pick them up as you practise along.

What is React Email?
Go Templates
- Go Template Delimiters
Create Dynamic Emails in Go with React Email
Conclusion

What is React Email?

React Email is a JavaScript library that helps you build dynamic email templates with React. If you already know basic React, React Email provides a better developer experience for building dynamic email templates. Here are some reasons why:

Familiar syntax with React: If you know React already, React Email eliminates the hassle in learning a separate templating language, using inefficient drag-and-drop UIs, or writing emaiil templates from scratch with HTML tables.
Reusable built-in components: React Email provides ready-to-use UI components like Buttons and Footers so you don't have to start from scratch, making email development seamless and fast.
Consistency across email clients: React Email generates email templates that have been tested and work well across popular email clients. This helps eliminate worries over emails rendering inconsistently across email clients.
Email development tooling: React Email has features for previewing and assessing emails built with it. Some of these features include:
- A local development server that lets you preview mobile and desktop views of your emails in your web browser as you develop the emails in real time
- An email delivery feature that sends your email to a real email address for preview
- A compatibility checker that shows you how well your email is supported across popular email clients
- A spam scorer that analyses your email to determine if it's likely to be marked as spam
Tailwind integration: Tailwind is a popular CSS framework that provides classes for styling HTML and making it responsive. React Email integrates with Tailwind easily for creating beautiful emails.

All these features are free to use.

In this article, you'll learn how to generate an HTML file from a React Email template, convert it to a Go template, and inject data into the template for previewing.

Go Templates

The Go html/template package allows you to define reusable HTML templates that can be populated with dynamic data. These templates contain placeholders (called actions) that are evaluated by Go's templating engine and replaced with actual values during execution.

First, you give the package HTML content that contains Go-specific annotations. It converts the HTML content to a Go HTML template and the Go-specific annotations to actions in the template. The template is then executed with data to produce HTML output that contains the data.

package main

import (
	"html/template"
	"os"
)

func main() {
	tmpl := template.New("hello")
	tmpl, _ = tmpl.Parse(`Hello {{.}}`)
	tmpl.Execute(os.Stdout, "Gopher")
}

// Output: Hello Gopher
// Playground: https://goplay.tools/snippet/KxbkWPIArz5

In the code snippet above:

template.New creates an empty template object with the name "hello"
tmpl.Parse(`
Hello {{.}}
`) parses the HTML string Hello {{.}} to create a Go HTML template and saves it in tmpl . The {{.}} part of the HTML string is an action which acts as a placeholder for data. {{ and }} are called delimiters and . is the data access identifier.
tmpl.Execute(os.Stdout, "Gopher") populates the action with data - the "Gopher", string, and writes the resulting HTML output to the console.

Go Template Delimiters

In the previous code snippet, you used double curly braces ({{ and }}) as the delimiters in the Go template. Delimiters are symbols that Go uses to determine what parts of the input string represent an action – that is, a statement to be evaluated.

You can change the delimiters by using the Delims method on a template. An example is shown in the snippet below:

package main

import (
	"html/template"
	"os"
)

func main() {
	tmpl := template.New("hello")
	tmpl, _ = tmpl.Delims("((", "))").Parse(`Hello ((.))`)
	tmpl.Execute(os.Stdout, "Gopher")
}

// Output: Hello Gopher
// Playground: https://goplay.tools/snippet/00RuDzvZYwN

In the snippet above, (( and )) are used as the delimiters for the hello template.

This is important because you'll set your delimiters to prevent conflicts between Go's default delimiters and React's curly braces in React Email templates.

Create Dynamic Emails in Go with React Email

The image below summarizes how the sample application you'll build in this article works:

You'll create a React Email template that contains Go template annotations. Next, you'll use Node.js to create HTML files from it. Go will parse the HTML file to create a Go template, execute it, and send it.

Optionally, you'll use go-mail to send the email and MailHog, a local SMTP server, to preview it in your browser.

Set Up the Project

First, make sure that you have Go and Node.js installed on your computer already. Clone this freeCodeCamp-go-react-email repository and checkout the 01-setup branch using git checkout 01-setup.

The project contains a main.go file in the cmd directory and a go.mod file. It also contains a .gitignore file to instruct Git to ignore all node_modules directories.

Run go run cmd/main.go in the terminal of the project. If you see "It works!" logged to the console, you have set it up properly and you can continue to the next section.

Set Up React Email

In the project root directory, create a mailer directory which will serve as the mailer package. It will hold all functionality related to creating and sending mails.

In the mailer directory, you'll create the emails Node.js project that will handle React Email functionality. To create the project:

Create a directory called _emails in the mailer directory. The name of the directory starts with an underscore because it should be ignored when the go build command is run. So it won't be included in the Go compiled executable file.
Run npm init -y in the root terminal of the _emails directory to initialise the Node.js project in it. This will create a package.json file in the directory.
Update the value of the name field in package.json to "emails" to make the package name more conventional. This step is not compulsory.

Next, install the required React Email libraries by running the following commands in the root terminal of the _emails directory:

npm install @react-email/ui @types/react -D -E
npm install react-email react react-dom -E

After the installation is complete, replace the scripts field of the package.json file with the code snippet below:

  "scripts": {
    "dev": "email dev --dir ./src",
    "export": "email export --pretty --dir ./src --outDir ../templates"
  },

The dev script starts and runs the server for previewing the React Email templates in the browser. You will write the template with React and store it in the src directory under _emails. The export script transpiles the template files in the src directory from JSX (or TSX) to HTML and stores them in a directory called templates, a direct child of the mailer directory – not a child of the _emails directory.

The templates directory is stored as a child directory of the mailer directory because the Go project needs only the HTML output stored in the templates directory and not the contents of _emails.

If you've completed all these steps, you have set up React Email in the emails Node.js project. To view the current status of the project at this point, visit freeCodeCamp-go-react-email/02-setup-react-email.

In the next section, you'll create a React Email template and preview it in the browser.

Create a React Email Template

In this section, you'll create a React Email template and preview it in the browser. You'll also build and export the template to HTML files.

Create a directory called src inside the _emails directory. Inside the src directory, create a file called welcome.tsx. Copy and paste the content of the snippet below into welcome.tsx.

import {
  Body,
  Button,
  Container,
  Head,
  Heading,
  Html,
  Img,
  Preview,
  Section,
  Tailwind,
  Text,
} from "react-email";

interface WelcomeEmailProps {
  username?: string;
  company?: string;
  gophers?: string[];
}

const WelcomeEmail = ({
  username = "Nicole",
  company = "GoWorld",
  gophers = ["Tinky Winky", "Dipsy", "Laa-Laa", "Po"],
}: WelcomeEmailProps) => {
  const previewText = `Welcome to \({company}, \){username}!`;

  return (
    
      
      {previewText}
      
        
          
            
              
            
            
              Welcome to {company}, {username}!
            
            Hello {username},
            
              We're excited to have you onboard at {company}.
              We hope you enjoy your journey with us. If you have any questions
              or need assistance, feel free to reach out to any of the following
              gophers:
            
            
              
                {gophers.map((gopher) => (
                  {gopher}
                ))}
              
            
            
              
            
            
              Cheers,
              

              The {company} Team
            
          
        
      
    
  );
};

export default WelcomeEmail;

The code snippet above is the React Email template that you'll use in this article. To preview it, navigate to the terminal of the _emails root directory and run npm run dev . Use your web browser to visit the preview URL displayed on the terminal. Click on the "welcome" link on the left sidebar and you should see a UI similar to the one in the screenshot below:

In the UI above, React Email renders the email with the default values supplied to the welcome email template.

Stop the server by clicking on the terminal that runs it by pressing CTRL + C. Build the HTML output of the src directory and export it by running npm run export in the terminal of the _emails root directory. This creates a templates directory within the mailer directory where the exported HTML files are stored. In the templates directory, you'll see a welcome.html file – the HTML output from welcome.tsx.

To see the current status of the project, visit freeCodeCamp-go-react-email/03-create-react-email-template.

Set Up Go Templates from HTML Files

You have created a React Email template, previewed it, built it, and exported it as an HTML file. In this section, you'll update the React Email template to use the delimiters you set and not React's curly braces. You'll also create a Go template from the HTML file.

To get started, replace the content of welcome.tsx with the code snippet below to to use (( and )) as delimiters and remove TypeScript types:

import {
  Body,
  Button,
  Container,
  Head,
  Heading,
  Html,
  Img,
  Preview,
  Section,
  Tailwind,
  Text,
} from "react-email";

const WelcomeEmail = () => {
  const previewText = `Welcome to ((.Company)), ((.Username))!`;

  return (
    
      
      {previewText}
      
        
          
            
              
            
            
              Welcome to ((.Company)), ((.Username))!
            
            Hello ((.Username)),
            
              We're excited to have you onboard at ((.Company))
              . We hope you enjoy your journey with us. If you have any
              questions or need assistance, feel free to reach out to any of the
              following Gophers:
            
            
              
                ((range .Gophers))
                ((.))
                ((end))
              
            
            
              
            

            
              Cheers,
              

              The ((.Company)), Team
            
          
        
      
    
  );
};

export default WelcomeEmail;

Run npm run export in the root terminal of the _emails directory to build and export this version of the React Email template to HTML. The HTML generated will contain Go template annotations that will become actions when parsed by Go to form a Go HTML template.

In the mailer directory, create a file named fs.go. The code in the file will be used to embed the files in the templates directory for use in the Go application. Copy and paste the content of the snippet below into fs.go:

package mailer

import (
	"embed"
	"io/fs"
)

//go:embed templates/*
var embedded embed.FS
var templateFS, _ = fs.Sub(embedded, "templates")

//go:embed templates/* tells the Go compiler to embed files from the current directory (mailer) into the compiled binary of the Go application. You need this to access the HTML template files from the Go application. templateFS will be used to access the files in the templates subdirectory.

Create another file in the mailer directory and name it mailer.go. mailer.go will contain code used to parse HTML files to make Go HTML templates and also send emails. Copy the content of the code snippet below into mailer.go:

package mailer

import (
	"html/template"
	"io"
)

const (
	welcomeMailKey = "welcome_mail"
)

func setUpTemplates() (map[string]*template.Template, error) {
	templates := make(map[string]*template.Template)

	tmpl := template.New("welcome.html").Delims("((", "))")
	welcomeEmailTmpl, err := tmpl.ParseFS(templateFS, "welcome.html")
	if err != nil {
		return nil, err
	}

	templates[welcomeMailKey] = welcomeEmailTmpl

	return templates, nil
}

type Mailer struct {
	templates map[string]*template.Template
}

// NewMailer creates a new mailer
func NewMailer() (*Mailer, error) {
	tpls, err := setUpTemplates()
	if err != nil {
		return nil, err
	}

	return &Mailer{
		templates: tpls,
	}, nil
}

type WelcomEmailData struct {
	Username string
	Company  string
	Gophers  []string
}

func (mailer *Mailer) WriteWelcomeMail(w io.Writer, data WelcomEmailData) error {
	tmpl := mailer.templates[welcomeMailKey]
	err := tmpl.Execute(w, data)

	return err
}

In the code snippet above:

setUpTemplates creates a template object, tmpl, and sets its delimiters. tmpl parses welcome.html to convert it to a Go template and stores the template with welcomeEmailTmpl as its identifier. After that, welcomeEmailTmpl is added to the templates map with welcomeMailKey as its key and templates is returned.
NewMailer creates and returns a Mailer object which holds the templates map and methods to work with the mail templates.
WriteWelcomeMail is a method on Mailer that's used to execute the welcome email template with real data.

To view the current status of the codebase at this point, visit freeCodeCamp-go-react-email/04-create-golang-template.

Render the Dynamic Email in the Browser

In this section, you'll create a simple web server to view the rendered email template containing the dynamic values passed to it.

Replace the content of main.go with the code snippet below:

package main

import (
	"fmt"
	"net/http"
	"os"

	pkgMailer "github.com/orimdominic/freeCodeCamp-go-react-email/mailer"
)

func main() {
	mailer, err := pkgMailer.NewMailer()
	if err != nil {
		fmt.Fprint(os.Stderr, err)
		os.Exit(1)
	}

	http.HandleFunc("/mail", func(w http.ResponseWriter, r *http.Request) {
		username := r.URL.Query().Get("username")
		company := r.URL.Query().Get("company")
		gophers := []string{"Tinky Winky", "Dipsy", "Laa-Laa", "Po"}

		err := mailer.WriteWelcomeMail(w, pkgMailer.WelcomEmailData{
			Username: username,
			Company:  company,
			Gophers:  gophers,
		})
		if err != nil {
			http.Error(w, err.Error(), http.StatusInternalServerError)
			return
		}
	})

	port := ":8888"
	err = http.ListenAndServe(port, nil)
	if err != nil {
		fmt.Fprint(os.Stderr, err)
		os.Exit(1)
	}
}

The code snippet above first creates a mailer object using the NewMailer function from mailer.go. After the error handling, it creates a simple web server running on port 8888 with a GET /mail route.

The GET /mail route accepts two query parameters: username and company, which will be used as the dynamic data for the email. The result of executing the template with WriteWelcomeMail is written as an HTML response on the browser. You'll use this route to test the functionality of the mailer package.

Before you start the server, you should build and export the React Email templates so that your HTML files always have the most recent changes from React Email templates. Instead of navigating between different directories to build, export and run the server, you can use a Makefile.

Navigate to the terminal of the root directory of the project and create a file called Makefile. Copy and paste the content of the code snippet below into it:

run: email-build
	go run cmd/main.go

email-build: mailer/_emails
	npm --prefix mailer/_emails run export

The run script of the Makefile above builds and exports the React Email templates as HTML to the mailer/templates directory and then starts the Go application. Ensure that Makefile uses hard tabs, not spaces for indentation.

Run make run in the terminal of the root directory of the project and visit http://localhost:8888/mail?username=Nicole&company=GoWorld in the browser. You'll see the email rendered on the browser UI.

Replace the values of username and company in the URL to test the email with different values.

With this setup, you can integrate the result of executing the template with your mail client and the email recipient will see the email as it's displayed in the browser.

To view the current status of the codebase at this point, visit freeCodeCamp-go-react-email/05-render-dynamic-email.

Send and Test Email with go-mail and MailHog

In the previous section, you supplied data to execute your template, but it was rendered in the browser, not an email client. In this section, you'll use go-mail to send the email and MailHog to intercept and view it.

This section is optional. If you don't have MailHog installed locally, you'll need Docker Compose to set it up for this project. Make sure Docker Compose is installed on your computer before proceeding.

In your terminal, navigate to the root directory of the project and run go get github.com/wneessen/go-mail to install go-mail. Create a compose.yml file in the root directory of the project and paste the contents of the code snippet below into it:

services:
  mailhog:
    image: mailhog/mailhog
    restart: no
    logging:
      driver: "none" # disable saving logs
    ports:
      - 1025:1025 # smtp server
      - 8025:8025 # web ui

In your terminal, navigate to the project's root directory and run docker compose up to pull and start the MailHog SMTP server. MailHog listens for emails on port 1025 and exposes a web UI at http://localhost:8025 where you can view intercepted emails. Depending on your internet connection, the initial image pull may take a few minutes.

Replace mailer.go with the content of the code snippet below:

package mailer

import (
	"html/template"
	"io"

	"github.com/wneessen/go-mail"
)

const (
	welcomeMailKey = "welcome_mail"
    sender = "noreply@localhost.com"
)

func setUpTemplates() (map[string]*template.Template, error) {
	templates := make(map[string]*template.Template)

	tmpl := template.New("welcome.html").Delims("((", "))")
	welcomeEmailTmpl, err := tmpl.ParseFS(templateFS, "welcome.html")
	if err != nil {
		return nil, err
	}

	templates[welcomeMailKey] = welcomeEmailTmpl

	return templates, nil
}

type Mailer struct {
	client    *mail.Client
	templates map[string]*template.Template
}

// NewMailer creates a new mailer
func NewMailer() (*Mailer, error) {
	tpls, err := setUpTemplates()
	if err != nil {
		return nil, err
	}

	c, err := mail.NewClient(
		"localhost",
		mail.WithPort(1025),
		mail.WithTLSPolicy(mail.NoTLS),
	)

	if err != nil {
		return nil, err
	}

	return &Mailer{
		client:    c,
		templates: tpls,
	}, nil
}

type WelcomEmailData struct {
	Username string
	Company  string
	Gophers  []string
}

func (mailer *Mailer) WriteWelcomeMail(w io.Writer, data WelcomEmailData) error {
	tmpl := mailer.templates[welcomeMailKey]
	err := tmpl.Execute(w, data)

	return err
}

func (mailer *Mailer) SendWelcomeMail(to string, data WelcomEmailData) error {
	m := mail.NewMsg()
	m.From(sender)
	m.To(to)
	m.Subject("Welcome to " + data.Company)
	m.SetBodyHTMLTemplate(mailer.templates[welcomeMailKey], data)

	err := mailer.client.DialAndSend(m)
	return err
}

The new changes to mailer.go include:

An import of the go-mail
The creation of a sender constant which represents the email of the sender
The creation of a mail client with go-mail
The creation of a SendWelcomeMail method on the mailer struct which creates an email with welcomeEmailTmpl, executes it, and sends it to a receiver.

In main.go, update the GET /mail route to use SendWelcomeMail instead of WriteWelcomeMail. You can use any email address you want. The snippet below uses fcc@go.dev:

		err := mailer.SendWelcomeMail("fcc@go.dev", pkgMailer.WelcomEmailData{
			Username: username,
			Company:  company,
			Gophers:  gophers,
		})
		if err != nil {
			http.Error(w, err.Error(), http.StatusInternalServerError)
			return
		}

		fmt.Fprint(w, "Email sent")

Ensure that the mail server is running by visiting http://localhost:8025 in your web browser. In another terminal, from the root directory of the project, run make run to start the server. Test the functionality of the route by visiting http://localhost:8888/mail?username=Nicole&company=GoWorld once again. Next, check the email server by visiting http://localhost:8025. You should see a UI similar to the one in the screenshot below:

Click on "Welcome to Helix" to view the email.

To view the current status of the codebase at this point, visit freeCodeCamp-go-react-email/06-send-email.

Conclusion

By following along with this tutorial, you have:

Learned how to create Go email templates from React Email templates
Learned how to use Makefiles to run custom scripts
Previewed your email in the browser and tested it using MailHog

You can now skip the hassle of writing raw HTML email tables or learning a new templating language. With React Email and Go templates, you have a cleaner, more developer-friendly way to build and send beautiful emails.

How to Build a Bank Ledger in Golang with PostgreSQL using the Double-Entry Accounting Principle.

Paul Babatuyi — Wed, 25 Mar 2026 17:11:25 +0000

The Hidden Bugs in How Most Developers Store Money

Imagine you're building the backend for a million-dollar fintech app. You store each user's balance as a single number in the database. It feels simple: just update the number when money moves.

But with one line of code like UPDATE accounts SET balance = balance - 100, you've created a system that can silently lose millions. A server crash, a race condition, or a clever attack, and suddenly money vanishes or appears out of thin air.

There's no audit trail, no way to know what happened, and no way to prove it didn't happen on purpose.

This isn't just a theoretical risk. It's a trap that's caught even experienced developers. The world's most trusted financial systems avoid it by using double-entry accounting. Every transaction creates two records: a debit on one account, a credit on another. This lets you reconstruct every cent from history, catch inconsistencies, and audit every transaction.

There are no deletes, and no silent updates. Just an append-only trail that makes fraud and bugs much harder to hide.

In this guide, you'll build a robust backend in Go and PostgreSQL, using patterns inspired by real fintech companies. You'll learn how to design a double-entry ledger, generate type-safe SQL with sqlc, and write transactions that are safe even under heavy load.

By the end, you'll understand why these patterns matter – and how to use them to build software you can trust with real money.

Prerequisites and Project Overview
The Double-Entry Foundation
Type-Safe SQL with sqlc
The Store Layer: Transactions and Retries
The Service Layer: Business Logic
The API Layer
Running It Locally
Testing: Prove the System Works
Deployment
Conclusion

Project Resources:

Here's the project repository: https://github.com/PaulBabatuyi/double-entry-bank-Go

And here's the front-end repository: https://github.com/PaulBabatuyi/double-entry-bank

You can find the live frontend here: https://golangbank.app

You can find the live Swagger back-end API here: https://golangbank.app/swagger

Prerequisites and Project Overview

Before you dive in, make sure you have the following installed:

Go 1.23 or newer
Docker and Docker Compose
golang-migrate CLI: go install github.com/golang-migrate/migrate/v4/cmd/migrate@latest
sqlc CLI: go install github.com/sqlc-dev/sqlc/cmd/sqlc@latest

You'll also need a basic understanding of PostgreSQL and REST APIs to follow along.

If you've built a CRUD app before, you're ready for this. The project uses sqlc for type-safe queries, JWT for authentication, and a layered architecture that keeps business logic, persistence, and HTTP handling cleanly separated.

Here's how the project is organized:

.
├── cmd/                # Server entrypoint
│   └── main.go
├── internal/
│   ├── api/            # HTTP handlers & middleware
│   ├── db/             # Store layer (transactions, sqlc)
│   └── service/        # Business logic (ledger operations)
├── postgres/
│   ├── migrations/     # SQL migration files
│   └── queries/        # sqlc query files
├── docs/               # Swagger docs
├── Dockerfile, docker-compose.yml, Makefile
└── README.md

The architecture follows a clear three-layer pattern:

API Layer: Handles HTTP requests, authentication, and routing.
Service Layer: Contains the business logic. This is where double-entry rules are enforced.
Store Layer: Manages database transactions and persistence.

Every request flows from the handler, through the service, to the store, and finally to PostgreSQL. This separation makes the code easier to test, debug, and extend.

Backend Request Flow

graph TD
    A[HTTP Request] --> B[Handler - API Layer]
    B --> C[LedgerService - Business Logic]
    C --> D[Store - Persistence Layer]
    D --> E[(PostgreSQL)]
    E --> D
    D --> C
    C --> B
    B --> F[HTTP Response]

The Double-Entry Foundation: How Every Penny is Accounted For

Let's get to the heart of what makes this system bulletproof: double-entry accounting. Every operation – a deposit, withdrawal, or transfer – creates two entries that always balance. This is the secret sauce that keeps banks, payment apps, and even crypto exchanges from losing track of money.

Picture a simple deposit of $1,000:

| Account              | Debit   | Credit  |
|----------------------|---------|---------|
| User Account         |         | 1,000   |
| Settlement Account   | 1,000   |         |

Total debits always equal total credits. This is the fundamental rule. Every single operation in this system produces exactly this structure, with no exceptions.

Now picture a $200 transfer from User A to User B. Notice there are four entries, not two – both sides of both accounts are recorded:

| Account       | Debit   | Credit  | Description           |
|---------------|---------|---------|-----------------------|
| User A        | 200     |         | Transfer to User B    |
| User B        |         | 200     | Transfer from User A  |

Both entries share the same transaction_id, so you can always retrieve the complete picture of what happened with a single query. There's no guessing and no reconstructing, as the ledger tells the full story.

Why the Settlement Account Goes Negative

This trips up newcomers, so it's worth explaining explicitly. When a user deposits $1,000, the settlement account is debited $1,000. After several user deposits, the settlement balance will be negative. That's correct and expected: it represents the total amount of real-world money currently held inside the system on behalf of users. The invariant is:

SUM(all user account balances) + settlement balance = 0

If that ever doesn't hold, something is broken.

Enforcing the Rules in the Database

The database itself enforces these rules, not just the application code. Here's the core of the entries table migration:

CREATE TABLE IF NOT EXISTS entries (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    account_id UUID NOT NULL REFERENCES accounts(id) ON DELETE RESTRICT,
    debit NUMERIC(19,4) NOT NULL DEFAULT 0.0000 CHECK (debit >= 0),
    credit NUMERIC(19,4) NOT NULL DEFAULT 0.0000 CHECK (credit >= 0),
    transaction_id UUID NOT NULL,
    operation_type operation_type NOT NULL,
    description TEXT,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,

    CONSTRAINT check_single_side CHECK (
        (debit > 0 AND credit = 0) OR (debit = 0 AND credit > 0)
    )
);

Let's break down why each piece matters:

Single-sided entries are impossible. The check_single_side constraint means every entry must be either a debit or a credit, never both. If you try to insert an invalid row, the database rejects it – there's no way around it.
Every transaction is linked. Both the debit and credit entries share the same transaction_id (a UUID). This lets you fetch both sides of any operation instantly, making audits and debugging straightforward.
Operation types are explicit. The operation_type column is an enum at the database level, so only valid types like deposit, withdrawal, or transfer are allowed. There are no typos and no surprises.

The Settlement Account: The System's Anchor

Every real-world ledger needs a way to represent money entering or leaving the system. That's what the settlement account does. Here's how it's seeded in the database:

INSERT INTO accounts (id, name, balance, currency, is_system)
SELECT gen_random_uuid(), 'Settlement Account', 0.0000, 'USD', TRUE
WHERE NOT EXISTS (
    SELECT 1 FROM accounts WHERE is_system = TRUE AND name = 'Settlement Account'
);

The settlement account represents the "outside world." When a user deposits money, it comes from the settlement account. When they withdraw, it goes back. Using WHERE NOT EXISTS makes this migration idempotent – that is, safe to run multiple times without creating duplicates.

Type-Safe SQL with sqlc: No More Surprises

In financial systems, you can't afford surprises from your database layer. That's why this project uses sqlc, a tool that turns your SQL queries into type-safe Go code at compile time.

With sqlc, you see exactly what SQL runs, catch mistakes before they hit production, and avoid the "magic" (and hidden bugs) of most ORMs. Every query is explicit, every type is checked, and you get the best of both worlds: raw SQL power with Go's safety.

Why NUMERIC Becomes String (and Not float64)

Here's a subtle but critical detail from sqlc.yaml:

overrides:
    - db_type: "pg_catalog.numeric"
      go_type: "string"
    - column: "entries.debit"
      go_type: "string"
    - column: "entries.credit"
      go_type: "string"
    - column: "accounts.balance"
      go_type: "string"
    - db_type: "operation_type"
      go_type: "string"

Why string, not float64? Floating point arithmetic is imprecise. 0.1 + 0.2 in most programming languages does not equal exactly 0.3.

For money, you need exact decimal arithmetic. This project uses shopspring/decimal for all calculations and stores amounts as strings, converting at the service layer boundary. The database column itself is NUMERIC(19,4), which stores exact decimals – no float rounding ever touches your money.

Preventing Race Conditions: Locking with FOR UPDATE

One of the most important queries in the system is GetAccountForUpdate:

SELECT * FROM accounts
WHERE id = $1
LIMIT 1
FOR UPDATE; -- locks row for update, prevents TOCTOU races

This query uses FOR UPDATE to lock the account row during a transaction. Why? Imagine two requests both see a $500 balance and both try to withdraw $400. Without locking, both would succeed, and you'd end up with a negative balance. With FOR UPDATE, the second transaction waits until the first finishes, eliminating this classic race condition.

Calculating the True Balance: Always Trust the Entries

The real source of truth for any account is the sum of its entries, not the denormalized balance column. Here's the reconciliation query:

SELECT CAST(
    (COALESCE(SUM(credit), 0::NUMERIC) - COALESCE(SUM(debit), 0::NUMERIC))
    AS NUMERIC(19,4)
) AS calculated_balance
FROM entries
WHERE account_id = $1;

This computes the true balance from the ledger itself. It's how you catch bugs, audit the system, and prove that every penny is accounted for. The balance column on accounts is a denormalized cache for fast reads – and this query is the ground truth that validates it.

The Store Layer: Transactions and Automatic Retries

Every financial operation in this system runs inside a transaction – no exceptions. This is enforced by the ExecTx pattern in the store layer:

func (store *Store) ExecTx(ctx context.Context, fn func(q *sqlc.Queries) error) error {
    const maxAttempts = 10
    var lastErr error
    for attempt := 0; attempt < maxAttempts; attempt++ {
        lastErr = store.execTxOnce(ctx, fn)
        if lastErr == nil {
            return nil
        }
        if !isSerializationError(lastErr) {
            return lastErr
        }
        if attempt < maxAttempts-1 {
            if waitErr := sleepWithContext(ctx, retryWait(attempt)); waitErr != nil {
                return waitErr
            }
        }
    }
    return fmt.Errorf("transaction failed after %d attempts due to serialization conflicts: %w", maxAttempts, lastErr)
}

Why Serializable Isolation?

The transaction uses PostgreSQL's strictest isolation level: sql.LevelSerializable. This is like running transactions one at a time, eliminating entire classes of concurrency bugs. If two operations would conflict, PostgreSQL aborts one and returns a serialization error (SQLSTATE 40001).

Automatic Retries: Handling Real-World Concurrency

When a serialization error occurs, the code automatically retries with exponential backoff:

func retryWait(attempt int) time.Duration {
    base := 50 * time.Millisecond
    for i := 0; i < attempt; i++ {
        base *= 2
        if base >= time.Second {
            return time.Second
        }
    }
    return base
}

func sleepWithContext(ctx context.Context, d time.Duration) error {
    select {
    case <-ctx.Done():
        return ctx.Err()
    case <-time.After(d):
        return nil
    }
}

The backoff starts at 50ms and doubles each attempt, capping at 1 second. Up to 10 attempts are made. If the client disconnects mid-retry, sleepWithContext detects the cancelled context and returns immediately. This means no wasted resources.

The Service Layer: Where Business Logic Meets Double-Entry

The service layer is the heart of the system. Its job is to translate business operations – deposits, withdrawals, transfers – into double-entry journal entries that always balance.

Deposit: Crediting the User, Debiting the Settlement

Every deposit creates two entries: a credit to the user's account and a matching debit to the settlement account. Both entries share the same transaction ID.

func (s *LedgerService) Deposit(ctx context.Context, accountID uuid.UUID, amountStr string) error {
    amount, err := validatePositiveAmount(amountStr)
    if err != nil {
        return err
    }
    return s.store.ExecTx(ctx, func(q *sqlc.Queries) error {
        settlement, err := q.GetSettlementAccountForUpdate(ctx)
        if err != nil {
            return fmt.Errorf("settlement account not found: %w", err)
        }
        account, err := q.GetAccountForUpdate(ctx, accountID)
        if err != nil {
            return fmt.Errorf("account not found: %w", err)
        }
        if account.Currency != settlement.Currency {
            return ErrCurrencyMismatch
        }
        txID := uuid.New()
        // 1. Credit user account
        _, err = q.CreateEntry(ctx, sqlc.CreateEntryParams{
            AccountID:     accountID,
            Debit:         decimal.Zero.StringFixed(4),
            Credit:        amount.StringFixed(4),
            TransactionID: txID,
            OperationType: "deposit",
            Description:   sql.NullString{String: "External deposit", Valid: true},
        })
        if err != nil { return err }
        // 2. Debit settlement (opposing entry)
        _, err = q.CreateEntry(ctx, sqlc.CreateEntryParams{
            AccountID:     settlement.ID,
            Debit:         amount.StringFixed(4),
            Credit:        decimal.Zero.StringFixed(4),
            TransactionID: txID,
            OperationType: "deposit",
            Description:   sql.NullString{String: fmt.Sprintf("Deposit to account %s", accountID), Valid: true},
        })
        if err != nil { return err }
        // 3. Update both balances atomically
        if err = q.UpdateAccountBalance(ctx, sqlc.UpdateAccountBalanceParams{
            Balance: amount.StringFixed(4), ID: accountID,
        }); err != nil { return err }
        return q.UpdateAccountBalance(ctx, sqlc.UpdateAccountBalanceParams{
            Balance: amount.Neg().StringFixed(4), ID: settlement.ID,
        })
    })
}

Two things are worth highlighting. First, both accounts are locked with GetAccountForUpdate and GetSettlementAccountForUpdate before any entries are written. This prevents any other concurrent transaction from reading a stale balance and acting on it.

Second, amount.Neg() is used to debit the settlement. Its balance goes down, representing real money now held inside the system.

Withdraw: Debiting the User, Crediting the Settlement

Withdrawals are the mirror image of deposits. The key difference is the insufficient funds check, which must happen inside the transaction after the lock is acquired:

balanceDec, err := decimal.NewFromString(account.Balance)
if err != nil {
    return errors.New("invalid balance")
}
if balanceDec.LessThan(amount) {
    return ErrInsufficientFunds
}

Checking balance inside the transaction after FOR UPDATE is critical. Checking it before, outside the transaction, would create a classic time-of-check-to-time-of-use (TOCTOU) race. Two concurrent withdrawals could both pass the check, then both execute, overdrawing the account.

The entries for a $500 withdrawal look like this:

| Account              | Debit   | Credit  |
|----------------------|---------|---------|
| User Account         | 500     |         |
| Settlement Account   |         | 500     |

The settlement is credited because real money is leaving the system, and it's being "returned" to the outside world.

Transfer: User-to-User, No Settlement Involved

Transfers move money directly between two user accounts. The settlement account isn't involved. Both accounts are locked, currency is validated, and an insufficient funds check runs before any entries are created:

func (s *LedgerService) Transfer(ctx context.Context, fromID, toID uuid.UUID, amountStr string) error {
    amount, err := validatePositiveAmount(amountStr)
    if err != nil { return err }
    if fromID == toID {
        return ErrSameAccountTransfer
    }
    return s.store.ExecTx(ctx, func(q *sqlc.Queries) error {
        fromAcc, err := q.GetAccountForUpdate(ctx, fromID)
        if err != nil { return err }
        toAcc, err := q.GetAccountForUpdate(ctx, toID)
        if err != nil { return err }
        if fromAcc.Currency != toAcc.Currency {
            return ErrCurrencyMismatch
        }
        fromBalance, _ := decimal.NewFromString(fromAcc.Balance)
        if fromBalance.LessThan(amount) {
            return ErrInsufficientFunds
        }
        txID := uuid.New()
        // Debit sender, credit receiver — same transaction ID
        // ... CreateEntry calls + UpdateAccountBalance calls
    })
}

A $200 transfer creates exactly two entries under the same transaction_id:

| Account  | Debit   | Credit  |
|----------|---------|---------|
| Sender   | 200     |         |
| Receiver |         | 200     |

ReconcileAccount: Trust, But Verify

Reconciliation is how you prove the system is correct. The ReconcileAccount function compares the stored balance column against the sum of all credits minus debits in the entries table:

func (s *LedgerService) ReconcileAccount(ctx context.Context, accountID uuid.UUID) (bool, error) {
    account, err := s.store.GetAccount(ctx, accountID)
    if err != nil { return false, fmt.Errorf("account not found: %w", err) }

    calculatedStr, err := s.store.GetAccountBalance(ctx, accountID)
    if err != nil { return false, fmt.Errorf("failed to calculate balance: %w", err) }

    calculated, _ := decimal.NewFromString(calculatedStr)
    stored, _ := decimal.NewFromString(account.Balance)

    if !stored.Equal(calculated) {
        log.Error().
            Str("stored_balance", account.Balance).
            Str("calculated", calculated.StringFixed(4)).
            Msg("Balance mismatch detected")
        return false, fmt.Errorf("balance mismatch: stored %s, calculated %s",
            account.Balance, calculated.StringFixed(4))
    }
    return true, nil
}

If they don't match, something has gone wrong: a bug, a direct database modification, or a race condition that slipped through. In production, this check can run as a background job to catch issues before they become incidents.

The API Layer: Secure, Predictable, and Boring (By Design)

The API layer is where your business logic meets the outside world. Its job is to be secure, predictable, and, if you've done things right, a little bit boring.

JWT Authentication: Secrets Matter

Authentication is handled with JWTs. The secret used to sign tokens must be at least 32 characters long (as shorter secrets are insecure and can be brute-forced). This is enforced at startup:

// internal/api/middleware.go
func InitTokenAuth(secret string) error {
    if secret == "" {
        return errors.New("JWT_SECRET environment variable is required")
    }
    if len(secret) < 32 {
        return errors.New("JWT_SECRET must be at least 32 characters")
    }
    TokenAuth = jwtauth.New("HS256", []byte(secret), nil)
    return nil
}

The server will refuse to start if the secret is missing or too short. There's no fallback and no default: the system fails loudly rather than running insecurely.

The Handler Pattern: Parse, Authorize, Validate, Call, Respond

Every handler follows the same recipe: extract JWT claims, parse the account ID, fetch the account and verify ownership, decode the request body, call the service, and respond. Authorization always happens before calling the service layer. The service knows nothing about users, keeping business logic clean and testable.

// internal/api/handler.go
func (h *Handler) Register(w http.ResponseWriter, r *http.Request) {
    var input struct {
        Email    string `json:"email"`
        Password string `json:"password"`
    }
    if err := json.NewDecoder(r.Body).Decode(&input); err != nil {
        respondError(w, http.StatusBadRequest, "invalid input")
        return
    }
    // ... hash password, create user, generate JWT ...
}

Amount Normalization: Defensive by Default

API clients send amounts in different formats – sometimes as strings, sometimes as numbers. The normalization logic ensures all amounts are handled safely:

// internal/api/amount.go
func normalizeAmountInput(value interface{}) (string, error) {
    switch v := value.(type) {
    case string:
        return strings.TrimSpace(v), nil
    case json.Number:
        return strings.TrimSpace(v.String()), nil
    case float64:
        return strconv.FormatFloat(v, 'f', -1, 64), nil
    default:
        return "", errors.New("amount must be a number or string")
    }
}

The decoder uses dec.UseNumber() so JSON numbers arrive as json.Number rather than float64, preserving full precision. The float64 case exists as a safety fallback only.

Frontend Deployment Boundary

The backend no longer serves static frontend files. The frontend is deployed separately at https://golangbank.app from its own repository: https://github.com/PaulBabatuyi/double-entry-bank.

Running It Locally: Your First End-to-End Test

git clone https://github.com/PaulBabatuyi/double-entry-bank-Go.git
cd double-entry-bank-Go
cp .env.example .env
# Edit .env — set JWT_SECRET with: openssl rand -base64 32
make postgres
make migrate-up
make server

Once the server is running:

Frontend: https://golangbank.app
Swagger UI: http://localhost:8080/swagger/index.html (local dev) or https://golangbank.app/swagger (production)
Health check: http://localhost:8080/health

The Swagger UI lets you explore every endpoint, authorize with your JWT token, and test operations directly in the browser.

Testing: Prove the System Works

Testing financial systems is non-negotiable, and claims about correctness need to be backed by code. This project tests all three layers, each targeting a different kind of failure.

Service Layer: Core Financial Logic

The most important tests live in internal/service/ledger_test.go. They run against a real PostgreSQL database – not mocks – because mock-based tests can give a false sense of security. Real database tests catch issues that only appear in production-like environments.

func TestDeposit_Success(t *testing.T) {
    ledger := setupTestLedger(t)
    accountID := createTestAccount(t, ledger, "0.00")

    err := ledger.Deposit(context.Background(), accountID, "100.00")
    require.NoError(t, err)

    balance := getAccountBalance(t, ledger, accountID)
    assert.Equal(t, "100.0000", balance)
}

func TestWithdraw_InsufficientFunds(t *testing.T) {
    ledger := setupTestLedger(t)
    accountID := createTestAccount(t, ledger, "50.00")

    err := ledger.Withdraw(context.Background(), accountID, "100.00")
    assert.ErrorIs(t, err, ErrInsufficientFunds)
}

The createTestAccount helper uses the settlement account's currency automatically, which is important: all accounts must share a currency for transfers to work, and tests that silently use a different currency will fail in confusing ways.

Concurrency Test: Proving Serializable Isolation Works

This is the most important test in the suite:

func TestConcurrentDeposits(t *testing.T) {
    ledger := setupTestLedger(t)
    accountID := createTestAccount(t, ledger, "0.00")

    var wg sync.WaitGroup
    wg.Add(2)
    go func() {
        defer wg.Done()
        _ = ledger.Deposit(context.Background(), accountID, "100.00")
    }()
    go func() {
        defer wg.Done()
        _ = ledger.Deposit(context.Background(), accountID, "100.00")
    }()
    wg.Wait()

    balance := getAccountBalance(t, ledger, accountID)
    assert.Equal(t, "200.0000", balance)
}

Two goroutines deposit simultaneously. The serializable isolation level and retry logic ensure both operations succeed and neither overwrites the other. Without the FOR UPDATE locks and transaction retry logic, this test would fail non-deterministically – which is exactly the kind of bug that's impossible to reproduce in development but devastating in production.

Store Layer: Transaction Mechanics

Tests in internal/db/store_test.go verify the retry infrastructure itself, without needing a database connection:

func TestIsSerializationError(t *testing.T) {
    pqErr := &pq.Error{Code: "40001"}
    assert.True(t, isSerializationError(pqErr))
    assert.False(t, isSerializationError(errors.New("some other error")))
}

func TestRetryWait(t *testing.T) {
    assert.Equal(t, 50*time.Millisecond, retryWait(0))
    assert.Equal(t, 100*time.Millisecond, retryWait(1))
    assert.Equal(t, 200*time.Millisecond, retryWait(2))
    assert.Equal(t, time.Second, retryWait(5)) // capped
}

func TestSleepWithContext_Cancel(t *testing.T) {
    ctx, cancel := context.WithCancel(context.Background())
    cancel() // cancel immediately
    err := sleepWithContext(ctx, 50*time.Millisecond)
    assert.Error(t, err) // should return immediately, not wait
}

API Layer: Authentication and Input Handling

Handler tests in internal/api/handler_test.go verify that the HTTP layer behaves correctly at its boundaries:

func TestRegisterHandler_BadRequest(t *testing.T) {
    h := setupTestHandler(t)
    req := httptest.NewRequest(http.MethodPost, "/register", nil)
    rw := httptest.NewRecorder()
    h.Register(rw, req)
    assert.Equal(t, http.StatusBadRequest, rw.Code)
}

func TestRegisterHandler_Success(t *testing.T) {
    h := setupTestHandler(t)
    _ = InitTokenAuth("fV7sliKV3qn657I60wEFtw/Auk/0bNU9zdp30wFzfDg=")

    email := "testuser_" + uuid.New().String() + "@example.com"
    body, _ := json.Marshal(map[string]string{"email": email, "password": "testpassword123"})

    req := httptest.NewRequest(http.MethodPost, "/register", bytes.NewReader(body))
    rw := httptest.NewRecorder()
    h.Register(rw, req)
    assert.Equal(t, http.StatusCreated, rw.Code)
}

Using uuid.New().String() in the email ensures each test run creates a unique user, preventing conflicts on repeated runs against the same database.

Middleware tests verify the security boundary itself:

func TestInitTokenAuthFromEnv_MissingSecret(t *testing.T) {
    os.Unsetenv("JWT_SECRET")
    err := InitTokenAuthFromEnv()
    assert.Error(t, err) // must fail without a secret
}

Running the Tests

# Start the database
make postgres

# Run all tests with race detection
make test

# Run with coverage report
make coverage

# Run tests the same way CI does (includes migrations)
make ci-test

The -race flag is non-negotiable for financial code. It instruments the binary to detect data races at runtime – something static analysis can't catch. If a race exists, the race detector will find it.

Deployment: Engineering Decisions That Matter in Production

The deployment setup for this project reflects several engineering decisions worth understanding, regardless of what platform you deploy to.

Migrations on Container Start

The Docker entrypoint runs golang-migrate up before starting the Go binary:

# docker-entrypoint
migrate -path /app/postgres/migrations -database "$migrate_db_url" up
exec /usr/local/bin/ledger

Running migrations at startup rather than as a separate CI step has trade-offs. The upside is simplicity: the container is always self-consistent when it starts. The downside is that each deployment takes slightly longer. For a solo project or small team, this is the right call. At scale you'd separate migrations from deployment.

Startup Retry Logic

The entrypoint retries migrations up to 12 times with a 5-second sleep between attempts:

max_attempts=12
attempt=1
while [ "\(attempt" -le "\)max_attempts" ]; do
    migration_output=$(migrate ... up 2>&1)
    # If "connection refused" or "timeout", keep retrying
    # If any other error, fail immediately
    attempt=$((attempt + 1))
done

The critical distinction is which errors trigger a retry. Network-transient errors (connection refused, timeout) are retried. Everything else – a bad migration SQL, a missing tabl – fails immediately. This avoids waiting the full 60 seconds when a deployment has a real problem.

DB URL Fallback Chain

In cloud environments, the internal database URL is often a different variable than what you configure locally. The resolveDBURL function handles this transparently:

func resolveDBURL() string {
    connStr := strings.TrimSpace(os.Getenv("DB_URL"))
    fallbackVars := []string{"INTERNAL_DATABASE_URL", "RENDER_DATABASE_URL", "DATABASE_URL"}
    // Falls back through the chain if DB_URL is empty or resolves to localhost
    ...
}

This pattern means local developers set DB_URL in .env and don't need to think about it, while the deployed container automatically uses the internal database connection without any manual wiring.

HTTP Server Timeouts

The server is configured with explicit timeouts:

srv := &http.Server{
    Addr:              ":" + port,
    Handler:           r,
    ReadTimeout:       15 * time.Second,
    WriteTimeout:      15 * time.Second,
    IdleTimeout:       60 * time.Second,
    ReadHeaderTimeout: 5 * time.Second,
}

Without timeouts, a slow or malicious client can hold connections open indefinitely, eventually exhausting the server's resources. ReadHeaderTimeout is particularly important: it limits how long the server waits for the HTTP headers before closing the connection, protecting against Slowloris-style attacks.

Conclusion: Building for the Real World

You've just walked through the core patterns that power real fintech systems:

Double-entry ledger with database-enforced constraints
Settlement account for tracking external cash flows
Serializable transactions with exponential backoff retry
Reconciliation endpoint for verifying correctness
Type-safe queries with sqlc
Row-level locking to prevent race conditions
Tests that prove correctness under concurrency

These aren't just Go patterns. They're the same principles used at companies like Monzo, Stripe, and Nubank. The implementation details differ, but the underlying ideas are the same: every dollar is accounted for, every operation is atomic, and the system can always explain where every penny went.

What's next? Three concrete next steps:

Add idempotency keys to prevent duplicate transactions on retries. If a client retries a deposit because of a network timeout, you need to detect and reject the duplicate.
Add Prometheus metrics for transaction latency and failure rates. You want to know when your p99 latency spikes before your users do.
Add a scheduled reconciliation job that runs ReconcileAccount for every account on a schedule and alerts on mismatches. Catch bugs automatically, before they become customer complaints.

The developer who stores balance as a single number and updates it directly will eventually have an incident. The developer who builds a ledger has an audit trail, a reconciliation tool, and a system that can explain every penny.

That's the real reason fintech engineers build this way: not because it's more complex, but because it's more honest about what money actually is.

How to Get Started Coding in Golang

Njong Emy — Thu, 12 Mar 2026 15:56:21 +0000

In the world of Software Engineering, there are plenty of programming languages to learn. And there are both low-level and high-level options.

I’ve tried my hand at a few of them, and the one language that feels like it touches both worlds is Golang – popularly known as Go. Although Go is a high-level language, it has pretty amazing performance that brings it close to the low-level edge.

Go is a fast, statically typed programming language. Types are declared at compile time, and you don’t need to run the code before catching errors. It’s also a general-purpose language that you can use for backend, cloud, servers, and more.

Go has built-in testing support, so you don’t need to install extra testing libraries. Go also has some features of object-oriented languages, but in its own way. It mirrors some OOP concepts, but also uses concepts like interfaces, structs, and so on.

In this tutorial, we're going to be looking at a few basic concepts you need to know when getting started in any programming language. A few of them are generic to many programming languages, but some of them are concepts specific to the Go programming language. We'll take a look at:

Variables
Formatting strings
Arrays and slices
Loops
Functions
Maps
Structs, and
Package scope

By the end of the article, you'll be grounded in the pure basics of Go, and we'll run a few examples to see how these work on the command line.

What we'll cover:

Prerequisites
How to Install Go
How to Write Your First Go Program
How to Work with Variables and Numbers in Go
How to Format Strings in Go
How to Work with Arrays and Slices in Go
How to Work with Loops in Go
How to Work with Functions in Go
How to Work with Maps in Go
How to Work with Structs in Go
Package Scope in Go
Conclusion

Prerequisites

To follow along with this tutorial, it’ll be helpful if you know the basics of any programming language, such as variables, data types, and data structures. I’ll assume that you’ve worked with at least one programming language before.

How to Install Go

To install Go, head to golang.org. Depending on what OS you are running, the documentation provides different installation options.

In my case, I’m using WSL (Windows Subsystem for Linux) with Ubuntu, so I’ll install Go inside that environment.

First, update your package list:

sudo apt update
sudo apt upgrade -y

Next, install a tool to download files from the internet. We’ll use wget:

sudo apt install -y wget

Now download the Go binary package:

wget https://dl.google.com/go/go1.24.2.linux-amd64.tar.gz

Extract the archive to /usr/local:

sudo tar -C /usr/local -xzf go1.24.2.linux-amd64.tar.gz

After installing Go, add the Go binary directory to your PATH so the go command is available in the terminal:

export PATH=$PATH:/usr/local/go/bin

You can make this change permanent by adding the line above to your ~/.bashrc or ~/.profile.

To confirm Go was installed successfully, run:

go version

You should see the installed Go version printed in the terminal.

If you use VS Code, you can also install the Go extension for syntax highlighting.

How to Write Your First Go Program

Before we get into writing our first program, we need to establish how Go puts its code together. In Go, every file or code is part of a package.

In this example, we’ll create a file called main.go. The name isn’t special to Go, but it’s a common convention for the file that contains the program’s entry point.

package main

import "fmt"

func main() {
	fmt.Println("Hello, ninjas")
}

This is the most important file in our project. We'll begin by making the code we’ll write in this file part of a package we call main.

The fmt import is a package from the Go standard library. fmt is for formatting strings and outputting to the console. Notice that the Println function starts with a capital letter, because within the fmt package, the method is exported. Variables or methods that are exported in Go should begin with capital letters. The Println prints a line to the console.

The main function serves as the entry point of a Go program. A compiled Go program must contain exactly one main function inside a package main. (A larger codebase can still contain multiple programs, each in its own directory with its own package main.)

So when you run a Go program, the Go toolchain builds all files in the same package together and then starts execution from the main() function. The filename main.go does not determine execution order – it’s simply a common naming convention.

Unlike other custom packages, which are used to bundle application logic into libraries or reusable code, the main package is used to specify that a program is a standalone executable.

To run the code, you can simply type go run main.go, specifying the Go file that we want to execute.

Hello, ninjas

How to Work with Variables and Numbers in Go

How to Declare Variables

A variable is just a store for data. Go does not allow unused local variables. If you declare a variable inside a function and never use it, the compiler will raise an error. This helps prevent cluttered code and unused definitions.

Let’s see how to declare variables in Go by declaring a string

package main

import "fmt"

func main() {

	// strings
	var nameOne string = "emy"

	fmt.Println(nameOne)
}

Because Go is a statically typed language, every variable must have a type at compile time. Here, we have a variable nameOne, it’s type, and then an initial value assigned to it.

But what if we don't want to state the variable type ourselves? Fortunately, Go lets us define variables without a type stated at compile time

// strings
var nameOne string = "emy"
var nameTwo = "blessing"
var nameThree string

For nameTwo, we don’t need to explicitly state the type because Go directly infers it. If you hover above the variable, you can see that Go already tells you the type. The third variable, nameThree, has no value assigned to it just yet. We’ve only just declared it.

If you log the output to the console, you can see nameOne and nameTwo displayed, but you can’t see nameThree. It’s also logged, you just can’t see it because it has no value.

emy blessing

There’s an even shorter and simpler way to declare variables in Go:

nameFour := "peaches"

Here, Go also infers the variable type based on the value assigned to it. Notice how we also didn’t use the var keyword. You can use this method inside any function, not just in main. Just keep in mind that you can't use this method of declaring variables when reassigning or updating a variable, declaring constants, or declaring safe types.

How to Declare Integers

When it comes to declaring integers, we pretty much use the same technique as above:

var ageOne int = 20
var ageTwo = 30
ageThree := 40

fmt.Println(ageOne, ageTwo, ageThree)

When declaring integers, we can specify the amount of memory or the bit size we want an integer to have. We can declare the integer as int8, int16, or int64.

Each of these memory sizes can hold a specific range of numbers. int8, for example, can only hold integers between -128 and 127. Anything bigger than that would throw an error:

// bits and memory
	var numOne int8 = 25
	var numTwo int8 = -128

	var numThree int8 = 129 // will throw an error

Another type of integer in Go is the unsigned integer, declared with uint. You’ll use this to declare only positive integers – you can’t use it to store negative values.

uint, just like the int type, can also be associated with bit sizes. You could have a uint8, uint16, and so on. But the ranges for these are different. You can check the complete list of bit sizes for the integer type and their ranges on this Go page.

int is specifically used to declare whole integer numbers. To declare decimal numbers, you can use a float. The concept is the same with int, where you can have floats with different bit sizes. But you can only have float32 and float64, where the latter can store bigger numbers than the former.

How to Format Strings in Go

Printing Strings to the Console

We came across the fmt package at the beginning of the article. Let’s go through some of the methods this package exposes.

First, we have the Print() method which logs some output the console. This method is used to log simple strings to the command line when we just want to see the output of something, and don’t really care about the readability.

If we have two simple strings as shown below;

// Print
fmt.Print("Hello, ")
fmt.Print("world!")

// Output: Hello, world!

You can see that the code above is not on separate lines, and this is the downside of the Print function. If we had multiple things to log to the command line, this function would just lump them together and readability could suffer.

But we can enforce a new line with Print() by using the escape character \\n .

fmt.Print("hello! \\n")
fmt.Print("new line \\n")

/*
	hello! 
	new line 
*/

Using escape characters throughout our code can get tiring very quickly. Luckily for us, Go has a function that helps us achieve the perfect separate line formatting we are looking for. We don’t need to use escape characters when we use Println .

fmt.Println("Hello, friends.")
fmt.Println("How are you?")

/*
	Hello, friends.
	How are you?
*/

Format Specifiers

Sometimes, you may want to log a string to the console, but you may also want to have variables within that string. This is called a formatted string, and we can achieve this with the help of format specifiers. These format specifiers are reserved characters that Go uses at runtime to position variables within strings.

Let’s use a concrete example to see the concept better,

name := "Emy"
age := 27

fmt.Printf("my age is %v and my name is %v", age, name)

// Output: my age is Emy and my name is 27

The format specifier %v is the default format specifier that stands for variable. We can see that in the position of the format specifiers, Go has placed the values of the arguments we passed to the Printf (age and name). It's important to note that the order in which we pass these arguments is important.

As we just saw above, the output of this code:

name := "Emy"
age := 27

fmt.Printf("my age is %v and my name is %v", name, age)

Would be my age is Emy and my name is 27. The order in which we pass the arguments is the same way in which the compiler will replace the format specifier within our string.

Something new you might also notice is that we didn't use Print or Println in this code like we did before. When working with formatted directives, Go puts the functions Printf, Sprintf, and Appendf at our disposal.

Printf directs the output to the console, as we’ve seen.
With Sprintf, we don't direct the output to the command line, but we can store it in a variable and then use that variable somewhere else in our code.
It gets a little more complex when working with Appendf, which formats according to a format specifier and appends the result to a byte slice (bytes are a bit out of the scope of this article, so we won’t dwell on them).

Besides the %v format specifier, we can also use %q if we want the embedded variable to have quotes around it.

// strings
	name := "Emy"

	fmt.Printf("my name is %q", name)

// output: my name is "Emy"

This will work for the name variable, but not really for the age because it’s an integer.

name := "Emy"
age := 27

fmt.Printf("my age is %q and my name is %q", name, age)

//Output: my age is '\\\\x1b' and my name is "Emy"

We also have the %T specifier, which is used to output the type of a variable.

If we wanted to get the type of age:

fmt.Printf("This is a variable of type %T", age)

The output of this would be:

This is a variable of type int

You can check out other format specifiers on the official fmt package page.

How to Work with Arrays and Slices in Go

Arrays in Go

Arrays in Go are a little bit of a mouthful. Let’s take a look at how to define an array:

var ages = [3]int{20, 25, 30}

In this code, we have the variable name on the left side as is typical. On the right side, we have [3] to specify the size of the array, the type as int , and the values for the array inside the squiggly braces.

Once we declare an array, we can never change its size. This is kind of a bummer if you ask me, because during coding, you don’t always know the size of an array when you declare it (more on this below when we talk about slices).

Arrays in Go also can’t contain multiple types. The array we declared above can’t also contain a string, for example.

If we log the array to the console alongside its length with fmt.Println(ages, len(ages)), on the console we get:

[20 25 30] 3

Slices in Go

If you need to define an array where you don’t know or don’t want to specify the size during declaration, you can use a slice. Slices are abstractions of arrays, and are more flexible than arrays because they have dynamic sizing.

var scores = []int{100, 50, 60} // we don't specify the size
scores[2] = 25 // to update a value
scores = append(scores, 50) // returns a new scores slice with 50 appended to it  (you cannot do this with arrays)
fmt.Println(scores, len(scores)) // [100 50 60 50] 4

An important part of working with arrays, slices, or any data structures that store data is knowing how to get the elements we want. We may only want to grab data that belongs to a certain range, or data at certain positions, based on certain conditions, and so on.

When it comes to dealing with ranges, say you want to output the slice elements from position 1 (that is, index 0, the first element) right up to the third position (index 2).

rangeOne := scores[0:3] // [100 50 60]

The range scores[0:3] indicates that the code should list the scores from index 0 up to index 3 minus 1. So it doesn’t include the element at index 3.

If you wanted to log from index 2 in the slice right up to the end, for example, you’d write scores[2:] . Similarly, you could log from the beginning of the slice up to and excluding some position (in this case, index 3) like so, scores[:3] .

How to Work with Loops in Go

How to Iterate Loops

Loops in Go are similar to loops in other programming languages. The difference is that Go focuses on the for loop and doesn’t really dwell on while, do-while, or for-each.

x := 0
for x < 5 {
	fmt.Println("the value of x is", x)
	x++
}

The loop above is straightforward and just prints the value of x from 0 to 4.

To create a loop that uses an iterator, you can write this:

for i := 0; i < 5; i++ {
	fmt.Println("value of i is", i)
}
    /*
		value of i is 0
		value of i is 1
		value of i is 2
		value of i is 3
		value of i is 4
	*/

It does pretty much the same thing as the loop above it, but we declared the iterator, it’s range, and we iterate over it in the same expression.

What if you wanted to iterate or loop over a slice, for example? We can also use an iterator to achieve this as we have above.

names := []string{"emy", "ble", "winkii"}

for i := 0; i < len(names); i++ {
		fmt.Println(names[i])
}

How to Use the `range` Keyword

Another interesting keyword related to iterations is range. Using the range keyword, you can use loops to perform actions on elements in a slice. range provides the index and value of the element in a slice, and allows you to access those within the loop.

for index, value := range names {
		fmt.Printf("the position of %v is %v \\\\n", value, index)
	}

What if you didn’t want to use the index and only the value? Well, range requires that you define an index and a value. So if you rewrite your loop like this:

for index, value := range names {
		fmt.Printf("the value is %v \\\\n", value)
}

Go is going to throw an error because you’re declaring the index but not using it.

Luckily, there’s a way to bypass this, and Go does it neatly using the blank identifier, _. We use this identifier when a method or function expects that you define a return value that you don't need.

for _, value := range names {
		fmt.Printf("the value is %v \\\\n", value)
}

You could apply the same strategy if you wanted only the index, but not the value.

for index, _ := range names {
		fmt.Printf("the index is %v \\\\n", index)
}

How to Work with Functions in Go

A function is a reusable block of code. Functions in Go are mostly created outside the main function. This way, other files can access and use them.

package main

import (
	"fmt"
)

func sayGreeting(n string){
	fmt.Printf("Good morning")
}

func main() {
	sayGreeting("emy")
}

Go also allows you to pass functions as parameters to other functions.

func cycleNames(n []string, f func(string)) {
	for _, value := range n {
		f(value)
	}
}

The function above takes a slice and a function as parameters. The function passed as a parameter in turn takes a string. You could pass a slice of names alongside the greeting function, so that for each name in the slice, you run the greeting function to print a greeting for that name.

func main(){
	cycleNames([]string{"emy", "pearl"}, sayGreeting)
}

When passing the sayGreeting function as a parameter, you don’t invoke it immediately because that’s done within the cycleNames function already. You only pass its reference.

Functions with `return` Values

Functions may also need to have a return value. So, how does Go handle that? Let’s see a small example:

package main

import "fmt"

func sayHello(name string) string {
	fmt.Printf("Hello %v", name)
	return name
}

func main() {
	sayHello("Emy")
}

// Output: Hello Emy

Just like with variables, we must specify the data type for every function parameter, and also specify the data type of the expected return value. In the example above, our function sayHello takes a string and returns a string.

Functions can also have multiple return values, as we see in the example below:

package main

import "fmt"

func sayHello(name string, age int) (string, int) {
	fmt.Printf("Hello %v, you are %v years old", name, age)
	return name, age
}

func main() {
	sayHello("Emy", 27)
}

Just like the previous example, we have to specify data types for our function parameters and return values.

How to Work with Maps in Go

A map in Go is a built-in, unordered collection of unique key-value pairs, similar to dictionaries in Python or hash tables in other languages. Maps provide fast lookups, insertions, and deletions.

With maps, all the keys must be of the same data type, and so must the values. If one key is a string, then all the other keys must also be strings. The same concept applies for values.

scores := map[string]float64{
		"maths":           20,
		"english":         15,
		"french":          14,
		"spanish":         12,
	}

	fmt.Println(scores)
	fmt.Println(scores["maths"])

	/*
	map[english:15 french:14 maths:20 spanish:12]
	20
	*/

You can also loop through maps to get their keys and corresponding values:

// loops maps
	for key, value := range scores {
		fmt.Println(key, "-", value)
	}

	/*
	french - 14
	spanish - 12
	maths - 20
	english - 15
	*/

When it comes to mutating maps, it's important to note that maps are reference types. Reference types are types whose variables don’t store the actual data, but rather an internal pointer to the actual data. This means that if the same data is assigned to multiple variables, when one instance gets modified, the original gets modified as well.

Let’s understand this with an example;

package main

import "fmt"

func main() {
	scores := map[string]float64{
		"maths":   20,
		"english": 15,
		"french":  14,
		"spanish": 12,
	}

	scores2 := scores

	scores2["maths"] = 15
	fmt.Println(scores)
}

// Output: map[english:15 french:14 maths:15 spanish:12]

If you check out the output, you can see that we modified the math score for scores2, but that modification also affected the original map, scores .

How to Work with Structs in Go

A big downside of using maps is that we can only store one data type for keys and one data type for values. This feels restrictive. We need a data structure that lets us store a collection of data with different data types. This is where structs (or structures) come in.

Let’s look at an example:

type Book struct {
	ID     uint
	Title  string
	Author string
	Year   int
	UserID uint
}

Here we have a Book struct. Structs are defined by specifying the key of the data you want the struct to carry, and the data type associated with that key.

package main

import "fmt"

type Book struct {
	ID     uint
	Title  string
	Author string
	Year   int
    UserID uint
}

func main() {

	var book1 Book

	book1 = Book{1, "Jane Eyre", "Jane Austen", 1990, 6}

	fmt.Println(book1)
}

We initialised the Book struct by providing it with the data we stated in its definition. If you check the output of this (and ignore that Jane Austen didn’t write Jane Eyre, of course):

{1 Jane Eyre Jane Austen 1990 6}

If you look closely, structs resemble classes from the OOP languages. They define properties (just as with methods, properties starting with a capital letter are public and can be exported), and these properties can be used within methods or functions. The difference comes at the level of inheritance, because structs, unlike classes, support only composition and not inheritance.

We can also individually modify the properties of a struct using dot notation, like so:

	var book1 Book

	book1 = Book{1, "Jane Eyre", "Jane Austen", 1990, 6}

	book1.Title = "Things Fall Apart"
	book1.Author = "Chinua Achebe"

	fmt.Println(book1)

If you check the output again, you can see that what you initialised the struct to contain has been modified at the level of the Title and Author.

{1 Things Fall Apart Chinua Achebe 1990 6}

When working with real systems, structs come in very handy when creating models or designing the structure of data you want to store in your database.

Package Scope in Go

Importing Functions Within the Same Package

When coding real applications, it's uncommon to write all our application files in one file. This can get messy very quickly. Normally, we would write a function in one file and then call it from another file.

In this case, you can declare variables and functions in other files and still use them in your entry point file. Say you create another greetings.go file and declared some variable and method in it like this:

// greetings.go
package main

import "fmt"

var points = []int{20, 90, 100, 45, 70}

func sayHello(n string) {
	fmt.Println("Hello", n)
}

Notice that there is no func main() in this file (remember that we can have only one throughout our application).

Now, in your main.go file, you can reference the sayHello() function and points variable you created.

// main.go
func main() {
	sayHello("emy")

	for _, v := range points {
		fmt.Println(v)
	}
}

To run this, you’ll need to have both files running at the same time. That is:

go run main.go greetings.go

You can see that the output is the result of running the sayHello() function and looping over the points slice.

You could also define functions and variables in the main.go file and pass them to the greetings.go file. Remember that all these functions and variables passed between files must be declared outside the main() function.

Importing Functions Across Different Packages

What if your application has, say, 100 files? Will you run all those files simultaneously? No, you won't. Since our Go code can be organised in packages, we can import a function from one package within another package. Let’s rewrite our code above, but using two different packages.

In our greetings.go file, we now have our code under a new package, greetings , and our SayHello function begins with a capital letter so that other files can import and use it.

// greetings.go
package greetings

import "fmt"

var Points = []int{20, 90, 100, 45, 70}

func SayHello(n string) {
	fmt.Println("Hello", n)
}

In our main.go file,

// main.go
package main

import (
	"fmt"
	"greetings"
)

func main() {
	greetings.SayHello("emy")

	for _, v := range points {
		fmt.Println(v)
	}
}

As you can see, we’ve imported the greetings package we declared in another file, and we can access the SayHello() method from it with dot notation.

Conclusion

In this tutorial, you’ve learned some key Go concepts. You’re now familiar with how variables, strings, arrays and slices, loops, functions, maps, structs, and package scope works in Go.

At this point, you should try to take that further by building something a little bigger so you can see just how powerful and amazing Golang is.

The official Tour of Go website is also an amazing resource to reference if you want to explore these concepts a little deeper.

How to Find the Top-K Items: Heap and Streaming Approaches in Go

Gabor Koos — Tue, 10 Mar 2026 23:25:25 +0000

Finding the top K items in a dataset pops up everywhere: from highlighting the hottest posts in a social feed, to detecting the largest transactions in a financial system, or spotting the heaviest users in a telemetry stream.

At first glance, you might think sorting everything and picking the top K is fine, but that approach quickly becomes a bottleneck as data grows or flows in continuously.

In this article, we'll explore heap-based and streaming approaches in Go that keep only the elements that matter. You'll see how to process data efficiently, maintain bounded memory, and make performance predictable - even when datasets are massive or unending.

The theory is language-agnostic, but we'll use Go's container/heap package to demonstrate the implementation details.

By the end, you'll have a practical toolkit for top-K queries that works in real-world applications, from dashboards to analytics pipelines.

Prerequisites

To get the most out of this tutorial, you should be familiar with:

Basic data structures: especially heaps / priority queues. I'll explain the minimum you need, but prior exposure helps.
Big O notation: understanding time and space complexity will help you appreciate the efficiency of different approaches.
Basic Go programming: variables, slices, loops, and functions.
Arrays and slices: understanding how to traverse and manipulate collections.
Sorting concepts: knowing what it means to sort a collection, even if you don't implement it yourself.
Channels (optional): for the streaming example, a basic understanding of Go channels will make it easier to follow.

If some of these are unfamiliar, you can still follow the article, but you may need to refer to the Go documentation or basic tutorials to fill in the gaps. This tutorial is designed to be hands-on and practical, so I'll explain the key concepts as we go.

The Problem and the Naïve Approach

Let's start with a simple example. Suppose we have a list of user scores:

scores := []int{50, 20, 80, 70, 60}

And we want to find the top 3 scores. The naïve approach is to

Sort the entire slice in ascending or descending order.
Select the last K elements (or first K, depending on sort order).

package main

import (
    "fmt"
    "sort"
)

func main() {
    scores := []int{50, 20, 80, 70, 60}
    sort.Sort(sort.Reverse(sort.IntSlice(scores))) // sort descending
    top3 := scores[:3]
    fmt.Println(top3) // Output: [80 70 60]
}

Sorting the entire dataset takes O(n log n) time, even if we only need the top-K elements. For small datasets, this is fine, but for large datasets or streams, sorting everything is unnecessary and can waste both CPU and memory.

This motivates more efficient strategies, such as heap-based or streaming approaches, which maintain only the top-K elements as we traverse the data.

Efficient Top-K Algorithms in Go

Instead of sorting the entire dataset, we can use a min-heap to efficiently maintain the top-K elements.

What is a Min-Heap?

A min-heap is a complete binary tree where the value of each node is less than or equal to the values of its children. For example:

In a min-heap, the smallest element is always at the root. This property allows us to efficiently maintain the top-K largest elements by keeping a min-heap of size K.

Once the heap is full, if a new element is less than or equal to the root (the smallest top-K score), we reject it. If it's greater, we replace the root (which is no longer in the top-K) and re-heapify to restore the min-heap property.

Re-heapifying is the repair step after that replacement. Since we overwrite the root, the min-heap rule can be violated at the top of the tree.

To fix it, we inspect both children to find the smaller one, and only then decide whether to swap: if the parent is greater than that smaller child, we swap with that child and continue this sift-down process. We never swap with the larger child first, because that could still leave a smaller child above a larger parent and break the min-heap property.

The process stops when the parent is less than or equal to both children (or when we reach a leaf). Because each swap moves one level down, the repair touches at most the height of the heap, so it takes O(log K) time.

The following diagram illustrates the process: we insert 35 into a heap of size 5 containing 10, 15, 20, 25 and 30:

How it works for Top-K:

Maintain a min-heap of size K.
Fill the heap with the first K elements.
For each remaining element, compare it with the root (h[0], the smallest of the current top-K).
If it's not larger than the root, reject it in O(1). Otherwise replace the root and re-heapify in O(log K).
After processing all elements, the heap contains the top-K elements because smaller elements are continuously discarded.

Implementing a Min-Heap in Go

Go's standard library provides a container/heap package that makes it easy to implement a min-heap. Here's how we can use it to maintain the top-K scores:

package main

import (
    "container/heap"
    "fmt"
)

type IntHeap []int
func (h IntHeap) Len() int           { return len(h) }
func (h IntHeap) Less(i, j int) bool { return h[i] < h[j] } // min-heap
func (h IntHeap) Swap(i, j int)      { h[i], h[j] = h[j], h[i] }
func (h *IntHeap) Push(x interface{}) { *h = append(*h, x.(int)) }
func (h *IntHeap) Pop() interface{} {
    old := *h
    n := len(old)
    x := old[n-1]
    *h = old[0 : n-1]
    return x
}

func topK(nums []int, k int) []int {
    if k <= 0 {
        return []int{}
    }

    h := &IntHeap{}
    heap.Init(h)

    for _, num := range nums {
        if h.Len() < k {
            heap.Push(h, num)
            continue
        }

        // h[0] is the root of the min-heap (the smallest value in current top-K).
        // If num is not larger, we can reject it in O(1).
        if num <= (*h)[0] {
            continue
        }

        // Replace root and restore heap order in O(log K).
        (*h)[0] = num
        heap.Fix(h, 0)
    }

    result := make([]int, h.Len())
    for i := len(result)-1; i >= 0; i-- {
        result[i] = heap.Pop(h).(int)
    }
    return result
}

func main() {
    data := []int{50, 20, 80, 70, 60}
    fmt.Println(topK(data, 3)) // Output: [80 70 60]
}

In this implementation, we define an IntHeap type that implements the heap.Interface: the Len, Less, Swap, Push, and Pop methods.

IntHeap is a slice of integers that represents our min-heap. A slice can represent a binary tree in a compact form, where the parent-child relationships are determined by indices (for a node at index i, its left child is at 2*i + 1 and its right child is at 2*i + 2).
Len returns the number of elements in the heap.
Less defines the ordering for the heap, and since we want a min-heap, we return true if the element at index i is less than the element at index j.
Swap swaps the elements at the given indices.
Push adds a new element to the heap by appending it to the underlying slice.
heap.Pop(h) removes and returns the smallest element from this min-heap. Internally, Go's container/heap first swaps the root with the last element, restores heap order, and then calls our Pop method, which removes and returns the last element of the underlying slice.

The topK function first fills the heap to size K. After that, it uses h[0] as a fast threshold check, because h[0] is always the root of the min-heap (the smallest value currently in the top-K set).

If a new number is less than or equal to h[0], we reject it immediately in O(1). If it is larger, we replace the root and call heap.Fix(h, 0) to re-heapify in O(log K). heap.Fix restores heap order after a value change and may move the element down or up as needed. In this specific root-replacement case, it moves downward.

If K is larger than the number of input elements, topK simply returns all available elements, sorted in descending order.

Finally, we pop all elements while filling the result slice from right to left, so the returned top-K scores are in descending order.

Streaming / Online Top-K

The same min-heap logic can be applied to streaming data. Instead of waiting for all values first, we process each item as it arrives:

Read one value from the stream.
If the heap has fewer than K elements, push the value.
Otherwise, compare with the root (h[0]): reject it if it's not larger, or replace the root and re-heapify.

At any moment, the heap contains the best K values seen so far. This is ideal for logs, metrics, event pipelines, or any unbounded input where storing everything isn't practical.

Below is a simple implementation using a channel to simulate incoming values (reusing the IntHeap type from the previous section):

func topKFromStream(stream <-chan int, k int) []int {
    if k <= 0 {
        return []int{}
    }

    h := &IntHeap{}
    heap.Init(h)

    for value := range stream {
        if h.Len() < k {
            heap.Push(h, value)
            continue
        }

        if value <= (*h)[0] {
            continue
        }

        (*h)[0] = value
        heap.Fix(h, 0)
    }

    result := make([]int, h.Len())
    for i := len(result) - 1; i >= 0; i-- {
        result[i] = heap.Pop(h).(int)
    }
    return result
}

func main() {
    stream := make(chan int)

    go func() {
        defer close(stream)
        for _, v := range []int{50, 20, 80, 70, 60, 90, 10} {
            stream <- v
        }
    }()

    fmt.Println(topKFromStream(stream, 3)) // Output: [90 80 70]
}

This pattern gives you online top-K updates with bounded memory (you only need to store the top-K elements, not the whole dataset), while still returning a descending top-K result at the end.

If the stream ends before K elements arrive, topKFromStream returns all seen elements, sorted in descending order.

Of course, in a real application, you may want to handle edge cases (like empty streams, non-integer values (if you generalize the heap type), or concurrent access) and consider how to gracefully shut down the stream processing.

Note that although we chose Go for our examples, the same min-heap approach applies in any language with a priority queue or heap data structure, such as Python's heapq, Java's PriorityQueue, or C++'s std::priority_queue.

If your language doesn't have a built-in heap, you can implement one using an array and the standard heap operations (sift-up, sift-down). The core logic of maintaining a bounded heap of size K and using the root as a threshold remains consistent across implementations.

Trade-offs & Practical Considerations

When implementing top-K queries, choosing the right strategy depends on dataset size, latency requirements, and memory limits. The same algorithm can behave very differently depending on whether data is small and static or large and continuously arriving.

Memory Usage

With naive sorting, you must have the full dataset in memory before sorting. The min-heap approach uses O(K) working memory, since the heap never grows beyond K elements. This difference becomes especially important in streaming scenarios, where a heap lets you keep bounded memory no matter how many items eventually arrive.

Time Complexity

A heap-based top-K computation runs in O(N log K), which is typically much cheaper than full sorting when K is far smaller than N. Full sorting remains O(N log N), even if you only need a tiny top slice at the end. With the optimized root check (h[0]), rejected values are O(1) after warmup, while accepted updates still cost O(log K), making throughput easier to reason about under load.

Batch vs Streaming

In batch workloads, where all records are available up front, both full sorting and heap-based methods can work well. For smaller datasets, full sorting is often the simplest solution. As data grows larger - especially when K is much smaller than N - a heap-based approach reduces unnecessary work by avoiding a complete sort.

In streaming workloads, an online heap is typically the better fit. It processes each value as it arrives and maintains only the best K elements seen so far, which keeps memory usage bounded and avoids storing historical data that no longer matters.

In distributed systems, the pattern usually extends one step further: compute the top-K independently on each partition, then merge those partial results. With P partitions, this produces up to P × K candidate elements that need combining.

A straightforward approach is to push all candidates into another min-heap of size K, which takes O(PK log K) in the worst case. If each partition already returns its local top-K in sorted order, and the merge stage only needs to extract the final global top-K (not fully merge all P × K items), a heap-based k-way merge over the P "heads" can do this in O(K log P) time, which is much more efficient.

At scale, the merge phase can still become a bottleneck, so practical systems carefully balance partition size, fan-out, and merge strategy to keep pipelines performant.

Multi-dimensional Top-K

Some ranking problems are not one-dimensional. If ordering depends on multiple attributes, such as score, timestamp, and tie-breakers, you can still apply the same heap pattern by defining a custom comparator that captures your priority rules.

More advanced query types, such as skyline-style selection, may require different techniques, but for many practical cases a carefully defined heap ordering remains a robust and efficient foundation.

Conclusion

Top-K is a small problem that shows up in big systems. The difference between "sort everything" and "keep only what matters" may seem minor at first - until your dataset grows, your stream never ends, or your service needs to respond in milliseconds.

A min-heap turns the problem into a bounded one. Instead of scaling with the size of your data, your working set scales with K. That shift is what makes the approach powerful in practice, whether you're processing a batch job, consuming a live stream, or merging results across partitions.

Once you internalize this pattern, you'll start recognizing it everywhere: leaderboards, monitoring systems, analytics pipelines, ranking engines. The implementation is compact, the guarantees are clear, and the performance scales predictably - and that's usually what good engineering looks like.

From here, you can explore more advanced structures and techniques to tackle even bigger or more complex top-K challenges:

Max-heaps and dual heaps: for sliding-window top-K or tracking both largest and smallest elements.
Order-statistics trees / augmented BSTs: fast rank queries and updates in a sorted structure.
Skip lists with priorities: probabilistic balancing and efficient streaming updates.

These advanced patterns build on the min-heap foundation, letting you handle larger datasets, tighter latency constraints, or more sophisticated ranking criteria. Mastering them will put you in a position to design truly high-performance, real-world ranking and analytics systems.

Understanding Escape Analysis in Go – Explained with Example Code

Eti Ijeoma — Thu, 12 Feb 2026 18:31:12 +0000

In most languages, the stack and heap are two ways a program stores data in memory, managed by the language runtime. Each is optimized for different use cases, such as fast access or flexible lifetimes.

Go follows the same model, but you usually don’t decide between the stack and the heap directly. Instead, the Go compiler decides where values live. If the compiler can prove a value is only needed within the current function call, it can keep it on the stack. If it cannot prove that, the value “escapes” and is placed on the heap. This technique is called escape analysis.

This matters because heap allocations increase garbage collector work. In code that runs often, that extra work can show up as more CPU spent in GC, more allocations, and less predictable performance.

In this article, I’ll explain what escape analysis is, the common patterns that trigger heap allocation, and how to confirm and reduce avoidable allocations.

Prerequisites
Do You Really Need to Care About Escape Analysis?
Memory Layout and Lifecycle
Sharing Down and Sharing Up
Escape Analysis in Practice
How to Use Escape Analysis to Guide Performance
Conclusion
Further Reading

Prerequisites

Familiarity with Go fundamentals (functions, variables, structs, slices, maps)
Basic understanding of pointers in Go (& and *)
A general idea of how goroutines work

Do You Really Need to Care About Escape Analysis?

Before we go deeper, I want to call this out clearly. For the correctness of your program, it doesn’t matter whether a variable lives on the stack or on the heap, or whether you know that detail. The Go compiler is smart enough to place values where they need to be so that your program behaves correctly.

Most of the time, you don’t need to think about this at all. It only starts to matter when performance becomes a problem. If your program is already fast enough, you’re done, and there’s no point trying to squeeze out extra speed.

You should only start caring about stack vs heap when you have benchmarks that show your program is too slow, and those same benchmarks point to heavy heap allocation and garbage collection as part of the problem.

Memory Layout and Lifecycle

To get a better understanding of what escape analysis is, you first need a simple picture of how Go lays out memory while your program runs. At this level, it comes down to the stack each goroutine uses, how stack frames are carved out of that stack, and when values move to the heap where the garbage collector can see them.

Goroutine Stacks and Stack Frames

When a Go program starts, the runtime creates the main goroutine, and every go statement creates a new goroutine, each with its own stack.

There’s not a single global stack for the whole process. As of writing this article, with Go v1.25.7, each goroutine gets an initial contiguous block of 2,048 bytes of memory, which acts as its stack. The stack is where Go stores data that belongs to function calls. When a goroutine calls a function, Go reserves a chunk of that goroutine’s stack for the function’s local data. That chunk is called a stack frame.

It holds the function’s local variables and the call state needed to return and continue execution. If that function calls another function, a new frame is added on top. When the inner function returns, its frame becomes invalid, and the goroutine continues in the caller’s frame.

A stack frame only lives for as long as the function is active. Once the function returns, anything stored in its frame is considered invalid, even if the raw bytes are still in memory and will be reused later. Code must not rely on those values after the return

Go stacks can grow. A goroutine starts with a small stack and the runtime grows it when needed, but the lifetime rule stays the same. A value is safe in a stack frame only if nothing can still reference it after the function returns. If it might be referenced later, it can’t stay in that frame and must be placed somewhere safer.

Pointers and Lifetime

In Go, taking an address like p := &x means you now have a pointer in one stack frame that refers to a value which may have been created in another frame. When you pass that pointer into a function, Go still passes by value. The callee gets its own pointer variable on its own stack frame, but the address inside still points to the same underlying value. So pointers are how you share access to one value across several frames without copying the value itself.

Lifetime becomes important when a pointer can outlive the frame where the pointed value was created. As long as both the pointer and the value live inside frames that are still active in the current call stack, everything is safe.

Once a pointer might still exist after the original frame has returned, the value can no longer stay in that frame, because that frame will become invalid. At that point, the value has to be placed in a safer location so that no pointer ever points into dead stack memory.

Now that you have a picture of stacks, frames, and pointers, we can look at two common ways pointers move through your code. I’ll call them sharing down and sharing up. The names aren’t special Go terms. They’re just a simple way to describe how a pointer moves along the call stack.

Sharing down means a function passes a pointer or reference to functions it calls. The pointer moves deeper into the call stack, but the value it points to still belongs to a frame that is active.

Example code:

package main

import "fmt"

func main() {
    n := 10
    multiply(&n) 
}

func multiply(v *int) {
   *v = *v * 2
}

In main, you take the address of n and pass it into multiply. While multiply runs, both the main frame and the multiply frames are active. The pointer in multiply points to a value that still lives in an active frame, so this situation is safe from a lifetime point of view.

In the diagram below, after the multiply function runs and returns, the multiply frame becomes invalid, and we don’t need to do anything because the stack pointer is simply popped back to the previous frame's address. This action automatically reclaims all the memory used by that function in one step, so the garbage collector is not involved in cleaning up stack memory

Sharing up means a function returns a pointer, or stores it somewhere that will still be around after the function returns. The pointer moves back up the call stack or into some longer-lived state while the frame that created the value is about to end, so that value can no longer be tied to that one frame.

The same idea shows up when you share a value with another goroutine, because Go doesn’t let one goroutine hold pointers into another goroutine’s stack, so shared data needs a lifetime that is not tied to a single stack.

Heap, garbage collection, and lifetime

Values that might outlive a single stack frame can’t stay in that frame. The compiler places them on the heap instead. The heap is a separate region of memory that isn’t tied to one function call. Any goroutine can hold pointers to heap values, and those values stay valid as long as something in the program can still reach them. You can think of the heap as storage for “might live longer than this call”.

The garbage collector is what keeps this safe. Periodically, the runtime starts from a set of roots (global variables, active stack frames, some internal state) and follows all the pointers it can see. Any heap value that is still reachable is kept. Any heap value that is no longer reachable is treated as garbage and its memory is reclaimed.

This means a pointer in main will never legally point into dead stack memory. Either the value stayed in an active frame, or it was placed on the heap where the GC can track its lifetime. The tradeoff is that more heap allocations and longer-lived objects require the GC to do more work.

Here’s an example:

package main

import "fmt"

type Car struct {
    Brand string
    Model string
}

func main() {
    // main receives a pointer from a function it called and this is sharing up
    carPtr := makeCar("Volkswagen", "Golf") 

    fmt.Printf("I received a car: %s %s\n", carPtr.Brand, carPtr.Model)
}

func makeCar(b, m string) *Car {
    myCar := Car{
        Brand: b,
        Model: m,
    }
    return &myCar
}

In the above code:

In makeCar (the callee frame), Go creates a local variable myCar. Because you return &myCar, the compiler allocates the Car value on the heap, and let’s myCar hold the heap address 0xc00029fa0.
When makeCar returns, that address is copied into carPtr in main (the top frame). carPtr is just another stack variable, but its value is still 0xc00029fa0, so now main also points to the same heap Car.
On the right, the heap bubble shows the actual Car value at 0xc00029fa0. Both car (while makeCar is running) and carPtr (after it returns) reach that same value through their pointers.
Once makeCar is done, its frame drops into the “invalid memory” region, but the Car stays alive on the heap because main still holds carPtr. That’s the escape: the value stops being tied to the callee frame and gets heap lifetime instead.

Escape Analysis in Practice

Escape analysis is how the Go compiler decides whether a value lives on the stack or on the heap. It’s not only about returning pointers – it follows how addresses move through your code. If a value might outlive the current function, the compiler can’t keep it in that stack frame and moves it to the heap. Since only the compiler sees the full picture, the useful thing is to ask it to show these decisions and then link them back to your code.

To do that, we can pass compiler flags using -gcflags when running go build or go run. If you want to see the available options, you can check go tool compile -h. In that list, -m prints the compiler’s optimisation decisions, including escape analysis output. If you want more details, you can use -m=2 or -m=3 for a more verbose output. The -l flag disables inlining, so the report is easier to read because the compiler is not merging small functions into their callers.

So, the command will look like this:

go run -gcflags='all=-m -l' .

Or for a build:

go build -gcflags='all=-m -l' .

How to Use Escape Analysis to Guide Performance

You can think of escape analysis as the thing that turns your code choices into GC work. When a value escapes, it gets heap lifetime, and the garbage collector has to visit it. In hot paths, lots of small escaping values show up as extra GC time and jitter in latency. When a value stays in a stack frame, it becomes invalid and dies with the frame and the GC does not care about it.

Here are five simple practices that help performance without making

Prefer values for small data: If the function doesn’t need to mutate the caller’s data, use value types for small structs and basic types when passing arguments and returning results. It’s cheap to copy an int or a small struct, and it often keeps lifetimes local to a single call.
Use pointers when sharing or mutation is part of the design: opt for pointers when you genuinely need shared mutable state or want to avoid copying large structs.
Avoid creating long-lived references by accident: Be careful when returning pointers to locals, capturing variables in closures, or storing addresses in long-lived structs, maps, or interfaces. These patterns are the ones most likely to push values out of a stack frame.
Pass in reusable buffers on hot paths: On code paths that run very often, the problem is usually not one big allocation, but many small ones happening in a loop. A common cause is functions that always create a new buffer inside, even when the caller could have passed one in.

A simple way to cut those extra allocations is to let the caller own the buffer. The caller allocates a []byte once, then passes it into the function each time. The function only fills the buffer instead of creating a new one.

Here’s an example of how a bad function allocates a new buffer every call:
```
 package main

 // Bad: helper allocates every call.
 func fillBad() []byte {
     buf := make([]byte, 4096)
     // pretend we read into it
     buf[0] = 1
     return buf
 }

 func hotPathBad() {
     for i := 0; i < 1_000_000; i++ {
         b := fillBad() // allocates 1,000,000 times
         _ = b
     }
 }

 func main() {
     hotPathBad()
 }
```
When we run escape analysis with this:
```
 go run -gcflags='-m -l' .
```
We see the following:
```
 ./main.go:5:13: make([]byte, 4096) escapes to heap
```
If we were only allocating a few times, we could choose not to worry – but the real problem is how this looks inside the loop. hotPathBad calls fillBad on every iteration, so each call allocates a new 4 KB slice on the heap. If this loop runs many times, you end up creating a lot of short-lived heap objects. The garbage collector then has to find and clean up all those buffers, which adds extra work that you could have avoided by reusing a single buffer.

Here’s an example of a better version where the caller allocates once and reuses:
```
 package main

 func fill(buf []byte) int {
     // pretend we read into it
     buf[0] = 1
     return 1
 }

 func hotPath() {
     buf := make([]byte, 4096) 

     for i := 0; i < 1_000_000; i++ {
         n := fill(buf) 
         _ = buf[:n]
     }
 }

 func main() {
     hotPath()
 }
```
In this version, hotPath controls the buffer. It allocates buf once, then passes it into fill on every loop. You still read the same data, but you avoid creating a new slice on each call. That reduces avoidable allocations in the hot path.

Conclusion

In Go, where a value ends up is not decided by how you create it. It’s decided by how long that value must remain valid and how it is referenced as your code runs.

The practical takeaway is not to avoid pointers. It’s to be deliberate about lifetime. Value semantics can keep lifetimes tight and reduce GC work, while pointers can be the right choice when you need shared state or in-place updates. The balance is to write the clear version first, then look at your benchmarks and profiles to see if anything actually really needs to change.

Unit Testing in Go - A Beginner's Guide

Gabor Koos — Mon, 12 Jan 2026 17:55:57 +0000

If you're learning Go and you’re already familiar with the idea of unit testing, the main challenge is usually not why to test, but how to test in Go.

Go takes a deliberately minimal approach to testing. There are no built-in assertions, no annotations, and no special syntax. Instead, tests are written as regular Go code using a small standard library package, and run with a single command. This can feel unusual at first if you're coming from ecosystems with richer testing frameworks, but it quickly becomes predictable and easy to reason about.

In this article, we'll look at how unit testing works in Go in practice. We'll write a few small tests, run them from the command line, and cover the most common patterns you'll see in real Go codebases, such as table-driven tests and testing functions that return errors. We'll focus on the essentials and won't cover more advanced topics like mocks or external frameworks.

The goal is to show how familiar testing concepts translate into idiomatic Go. By the end, you should feel comfortable reading and writing basic unit tests and integrating them into your regular Go workflow.

What We'll Cover:

Prerequisites
Writing Your First Test
Table-Driven Tests
Testing Functions That Return Errors
Best Practices and Tips
Conclusion
Solutions to Exercises
- Subtract Function and Tests
- SafeSubtract Function and Tests

Prerequisites

Before you start, you should be comfortable with:

Writing and running basic Go programs
Defining and calling functions in Go
Understanding basic Go types (int, string, bool, and so on)
Using the Go command-line tool (go run, go build)
Basic understanding of unit tests: what a test is and why it's useful
Familiarity with Test-Driven Development concepts like testing before or alongside writing code
Awareness of common testing ideas such as assertions, test coverage, and checking error conditions

You don't need prior experience with Go's testing package or Go-specific test patterns, as this guide will cover all of that.

Writing Your First Test

Let's start with a simple function to test. Imagine you have a small calc package with an Add function:

// calc.go
package calc

// Add returns the sum of two integers
func Add(a, b int) int {
    return a + b
}

To test this function, create a new file named calc_test.go in the same package. In Go, test files must end with _test.go to be recognized by the testing tool.

Inside calc_test.go, you write a test function:

// calc_test.go
package calc

import "testing"

func TestAdd(t *testing.T) {
    got := Add(2, 3)
    want := 5
    if got != want {
        t.Errorf("Add(2, 3) = %d; want %d", got, want)
    }
}

Here's what's happening:

The function name starts with Test and takes a single *testing.T parameter. Go automatically discovers and runs any function that follows this convention.
The t.Errorf call reports a test failure. Unlike some frameworks, Go doesn't provide special assertions – you simply check a condition and call t.Errorf or t.Fatalf if it fails.
Each test is a standalone function. You can write as many as you like, and Go will run them all.

Running Your Test

Once the file is saved, you can run your test with:

go test

This runs tests for the current package (files ending with _test.go). If you want to run tests recursively in all subdirectories of your project, use:

go test ./...

The ./... pattern is shorthand for "run tests in this directory and all subdirectories". This is especially useful in larger projects where your code is spread across multiple packages.

If everything is working, you should see output indicating that the test passed:

$ go test
PASS
ok      _/C_/projects/Articles/Go_Testing       0.334s

You can add the -v flag for verbose output:

go test -v

This will show you the names of the tests as they run:

$ go test -v
=== RUN   TestAdd
--- PASS: TestAdd (0.00s)
PASS
ok      _/C_/projects/Articles/Go_Testing       0.356s

Not much difference for a single test, but it becomes useful as you add more tests.

Now let's see what happens if the test fails. Change the expected value in calc_test.go to an incorrect one:

  ...
    want := 6 // Incorrect expected value
  ...

Run the tests again:

$ go test
--- FAIL: TestAdd (0.00s)
    calc_test.go:9: Add(2, 3) = 5; want 6
FAIL
exit status 1
FAIL    _/C_/projects/Articles/Go_Testing       0.340s

or with verbose output:

$ go test -v
=== RUN   TestAdd
    calc_test.go:9: Add(2, 3) = 5; want 6
--- FAIL: TestAdd (0.00s)
FAIL
exit status 1
FAIL    _/C_/projects/Articles/Go_Testing       0.337s

Of course, your tests should always check for the correct expected values! A failing (but correct) test is a sign that your code needs to be fixed.

We only created one test file and one test function with one assertion here, but Go's testing tool can handle many files and functions at once. Behind the scenes, Go will automatically:

Find all _test.go files in the specified packages (for example, current directory for go test, or recursively in all subdirectories with go test ./...).
Identify functions that start with Test and have the correct signature.
Compile them together with your package into a temporary test binary.
Execute each test function and report the results.

To prove this, let's quickly add a Divide function to our package:

// calc.go
...
// Divide returns the result of dividing a by b
func Divide(a, b int) int {
    return a / b
}

(Note that this is an integer division, so fractional parts are discarded. Divide(5, 2) would return 2.)

And another test file with a corresponding test:

// calc_2_test.go
package calc

import "testing"

func TestDivide(t *testing.T) {
    got := Divide(10, 2)
    want := 5    
    if got != want {
        t.Errorf("Divide(10, 2) = %d; want %d", got, want)
    }    
}

Now when you run go test, both TestAdd and TestDivide will be executed:

$ go test
PASS
ok      _/C_/projects/Articles/Go_Testing       0.325s

Or:

$ go test -v
=== RUN   TestAdd
--- PASS: TestAdd (0.00s)
=== RUN   TestDivide
--- PASS: TestDivide (0.00s)
PASS
ok      _/C_/projects/Articles/Go_Testing       0.323s

Divide by Zero

What happens if we try to Divide by zero? Let's add another test case for that:

// calc_test.go
...
func TestDivideByZero(t *testing.T) {
    defer func() {
        if r := recover(); r == nil { // Check if a panic occurred
            t.Errorf("Divide did not panic on division by zero")
        }
    }()
    Divide(10, 0) // This should cause a panic
}

This test checks that the Divide function panics when dividing by zero. When you run the tests again, you'll see that this new test also passes:

$ go test -v
=== RUN   TestAdd
--- PASS: TestAdd (0.00s)
=== RUN   TestDivide
--- PASS: TestDivide (0.00s)
=== RUN   TestDivideByZero
--- PASS: TestDivideByZero (0.00s)
PASS
ok      _/C_/projects/Articles/Go_Testing       0.312s

(Note that in real-world Go code, it's better to return (int, error) for unsafe operations instead of panicking.)

Feel free to experiment by adding more test cases, changing expected values, and exploring how Go's testing framework handles different scenarios.

`t.Errorf` vs `t.Fatalf`

In the examples above, we used t.Errorf to report test failures. This function logs the error but allows the test to continue running. This is useful when you want to check multiple conditions in a single test function.

In contrast, t.Fatalf logs the error and immediately stops the execution of the current test. Use t.Fatalf when continuing the test after a failure doesn't make sense or could cause misleading results.

For example, in the TestDivideByZero test, if the Divide function does not panic, we use t.Errorf to report the failure but continue to the end of the test. But if we had additional checks after the division, we might want to use t.Fatalf to stop execution immediately upon failure.

While t.Errorf and t.Fatalf use fmt-style formatting, for simple messages without formatting, you can also use t.Error and t.Fatal, respectively.

In the next section, we'll look at table-driven tests, a common Go pattern for testing multiple cases efficiently.

Table-Driven Tests

In Go, it's common to want to run the same test logic for multiple inputs and expected outputs. Rather than writing a separate test function for each case, Go developers often use table-driven tests. This pattern keeps your tests concise, readable, and easy to extend.

Table-Driven `Add` Test

Let's rewrite our Add test using a table-driven approach (and delete calc_2_test.go for clarity):

// calc_test.go
package calc

import "testing"

func TestAddTableDriven(t *testing.T) {
    tests := []struct {// Define a struct for each test case and create a slice of them
        name string
        a, b int
        want int
    }{
        {"both positive", 2, 3, 5},
        {"positive + zero", 5, 0, 5},
        {"negative + positive", -1, 4, 3},
        {"both negative", -2, -3, -5},
    }

    for _, tt := range tests {// Loop over each test case
        t.Run(tt.name, func(t *testing.T) {// Run each case as a subtest
            got := Add(tt.a, tt.b)
            if got != tt.want {// Check the result
                t.Errorf("Add(%d, %d) = %d; want %d", tt.a, tt.b, got, tt.want) // Report failure if it doesn't match
            }
        })
    }
}

Here's how it works:

We define a slice of structs, each representing a test case.
Each struct contains the test name, input values, and the expected result.
We loop over the slice and call t.Run(tt.name, func(t *testing.T) { ... }) to run each test as a subtest.
If a subtest fails, you can see which one by its name in the output.

$ go test
PASS
ok      _/C_/projects/Articles/Go_Testing       0.452s

Or to see detailed output:

$ go test -v
=== RUN   TestAddTableDriven
=== RUN   TestAddTableDriven/both_positive
=== RUN   TestAddTableDriven/positive_+_zero
=== RUN   TestAddTableDriven/negative_+_positive
=== RUN   TestAddTableDriven/both_negative
--- PASS: TestAddTableDriven (0.00s)
    --- PASS: TestAddTableDriven/both_positive (0.00s)
    --- PASS: TestAddTableDriven/positive_+_zero (0.00s)
    --- PASS: TestAddTableDriven/negative_+_positive (0.00s)
    --- PASS: TestAddTableDriven/both_negative (0.00s)
PASS
ok      _/C_/projects/Articles/Go_Testing       0.385s

Table-Driven Divide Test

We can apply the same pattern to Divide, including checking for divide-by-zero:

// calc_test.go
...
func TestDivideTableDriven(t *testing.T) {
    tests := []struct { // Define test cases
        name     string
        a, b     int
        want     int
        wantPanic bool
    }{
        {"normal division", 10, 2, 5, false},
        {"division by zero", 10, 0, 0, true},
    }

    for _, tt := range tests { // Loop over
        t.Run(tt.name, func(t *testing.T) { // Run subtest
            if tt.wantPanic { // Check for expected panic
                defer func() { // Recover from panic
                    if r := recover(); r == nil {
                        t.Errorf("Divide(%d, %d) did not panic", tt.a, tt.b)
                    }
                }()
            }
            got := Divide(tt.a, tt.b) // Tests that do not panic
            if !tt.wantPanic && got != tt.want {
                t.Errorf("Divide(%d, %d) = %d; want %d", tt.a, tt.b, got, tt.want)
            }
        })
    }
}

This example shows how to handle both normal and panic cases in a single table-driven test:

The wantPanic field tells the test whether we expect a panic.
We use defer and recover to check for a panic when needed.
Normal test cases still check the result as usual.

Run all tests as before:

$ go test -v
=== RUN   TestAddTableDriven
=== RUN   TestAddTableDriven/both_positive
=== RUN   TestAddTableDriven/positive_+_zero
=== RUN   TestAddTableDriven/negative_+_positive
=== RUN   TestAddTableDriven/both_negative
--- PASS: TestAddTableDriven (0.00s)
    --- PASS: TestAddTableDriven/both_positive (0.00s)
    --- PASS: TestAddTableDriven/positive_+_zero (0.00s)
    --- PASS: TestAddTableDriven/negative_+_positive (0.00s)
    --- PASS: TestAddTableDriven/both_negative (0.00s)
=== RUN   TestDivideTableDriven
=== RUN   TestDivideTableDriven/normal_division
=== RUN   TestDivideTableDriven/division_by_zero
--- PASS: TestDivideTableDriven (0.00s)
    --- PASS: TestDivideTableDriven/normal_division (0.00s)
    --- PASS: TestDivideTableDriven/division_by_zero (0.00s)
PASS
ok      _/C_/projects/Articles/Go_Testing       0.321s

Subtest names make it easy to see which case passed or failed.

Exercise

Try creating your own table-driven test for a new function, Subtract(a, b int) int. Include at least four test cases:

Both positive numbers
Positive minus zero
Negative minus positive
Both negative

Then run your tests and verify the output.

Testing Functions That Return Errors

Many Go functions return an error as the last return value. Writing tests for these functions is slightly different from testing pure functions like our Add or Divide, because you need to check both the result and whether an error occurred.

Safe Divide Function

Let's add a SafeDivide function to return an error instead of panicking:

// calc.go
...
import "fmt"
...
// SafeDivide returns the result of dividing a by b.
// It returns an error if b is zero.
func SafeDivide(a, b int) (int, error) {
    if b == 0 {
        return 0, fmt.Errorf("cannot divide by zero")
    }
    return a / b, nil
}

Writing Tests for `SafeDivide()`

We can use a table-driven test again:

// calc_test.go
func TestSafeDivide(t *testing.T) {
    tests := []struct {
        name      string
        a, b      int
        want      int
        wantError bool
    }{
        {"normal division", 10, 2, 5, false},
        {"division by zero", 10, 0, 0, true},
    }

    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            got, err := SafeDivide(tt.a, tt.b)
            if tt.wantError {
                if err == nil {
                    t.Errorf("SafeDivide(%d, %d) expected error, got nil", tt.a, tt.b)
                }
                return // stop here, no need to check `got`
            }
            if err != nil {
                t.Errorf("SafeDivide(%d, %d) unexpected error: %v", tt.a, tt.b, err)
            }
            if got != tt.want {
                t.Errorf("SafeDivide(%d, %d) = %d; want %d", tt.a, tt.b, got, tt.want)
            }
        })
    }
}

What's happening here:

We added a wantError field to indicate whether the test expects an error.
If an error is expected, we check that err != nil. If not (that is, err == nil), we fail the test.
If no error is expected, we check both the returned value (got) and that err == nil.
Using t.Run subtests keeps everything organized and readable.

Running the tests again:

$ go test -v
...
=== RUN   TestSafeDivide
=== RUN   TestSafeDivide/normal_division
=== RUN   TestSafeDivide/division_by_zero
--- PASS: TestSafeDivide (0.00s)
    --- PASS: TestSafeDivide/normal_division (0.00s)
    --- PASS: TestSafeDivide/division_by_zero (0.00s)
PASS
ok      _/C_/projects/Articles/Go_Testing       0.323s

Showing that both normal and error cases are handled correctly.

Exercise

Update your Subtract(a, b int) int function to a SafeSubtract(a, b int) (int, error) variant that returns an error if the result would be negative. Then write a table-driven test that covers:

A positive result
Zero result
A negative result (should return an error)

Best Practices and Tips

Writing tests in Go is straightforward, but there are a few conventions and tips that make your tests more readable, maintainable, and idiomatic:

Name Tests Clearly

First, make sure you use descriptive names for test functions and subtests. A good name explains what you're testing and under what conditions.

Here’s an example:

t.Run("Divide positive numbers", func(t *testing.T) { ... })
t.Run("Divide by zero returns error", func(t *testing.T) { ... })

Keep Tests Small and Focused

Each subtest should verify one thing, and each test function should cover a single function or method.

Try to avoid combining multiple unrelated checks in the same test function, and use table-driven tests help keep multiple similar checks concise without losing clarity.

Use Table-Driven Tests for Repetitive Cases

If you find yourself writing multiple similar test functions, switch to a table-driven pattern. It makes it easier to add new cases, reduces duplicated code, and keeps output organized with t.Run.

Check Errors Explicitly

In Go, functions often return error. So make sure you always check for errors in tests, even if you expect nil.

You can use the wantError pattern in table-driven tests for clarity.

if tt.wantError {
    if err == nil {
        t.Errorf("expected error, got nil")
    }
}

Avoid Panics When Possible

Panics are fine for some internal checks, but in production code, prefer returning an error.

Your tests can check for panics using defer and recover, but this should be the exception rather than the norm.

Run Tests Frequently

Try to make running tests a habit: go test -v ./.... Frequent testing helps catch mistakes early and reinforces TDD practices.

Keep Tests in the Same Package

By convention, tests live in the same package as the code they test. You can create _test.go files for testing, and Go automatically recognizes them.

Only use a separate package calc_test if you want to test your code from the outside, like a consumer. External test packages (just like every other external package) cannot access unexported identifiers.

Use t.Fatalf vs t.Errorf Appropriately

t.Errorf reports a failure but continues running the test.
t.Fatalf stops the test immediately, which is useful if subsequent code depends on successful setup.

These tips will help you write clean, maintainable, and idiomatic Go tests that are easy to read and extend. Following these practices early in your Go journey will make testing less intimidating and more effective.

Conclusion

Unit testing in Go may feel different at first, especially if you're coming from ecosystems with heavy frameworks and assertions. But the simplicity of Go's testing tools is one of its strengths: once you understand the conventions, writing, running, and organizing tests becomes predictable and intuitive.

In this guide, you've seen how to:

Write basic test functions with the testing package
Run tests from the command line and interpret the results
Use table-driven tests to cover multiple cases efficiently
Handle functions that return errors and check for expected failures

Beyond these fundamentals, testing is not just about verifying correctness, it's also about confidence. Well-tested code allows you to refactor, experiment, and add new features with less fear of breaking existing functionality.

As you continue writing Go code, try to integrate testing early, follow the idiomatic patterns you've learned, and explore more advanced topics such as:

Using mocks or interfaces to isolate dependencies
Benchmark tests with testing.B
Coverage analysis with go test -cover

The key takeaway is that testing in Go is accessible, flexible, and powerful, even without fancy frameworks. By building these habits now, you'll write code that's more reliable, maintainable, and enjoyable to work with.

Solutions to Exercises

Subtract Function and Tests

// calc.go
package calc

func Subtract(a, b int) int {
    return a - b
}

// calc_test.go
package calc

import "testing"

func TestSubtractTableDriven(t *testing.T) {
    tests := []struct {
        name string
        a, b int
        want int
    }{
        {"both positive", 5, 3, 2},
        {"positive minus zero", 5, 0, 5},
        {"negative minus positive", -1, 4, -5},
        {"both negative", -3, -2, -1},
    }

    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            got := Subtract(tt.a, tt.b)
            if got != tt.want {
                t.Errorf("Subtract(%d, %d) = %d; want %d", tt.a, tt.b, got, tt.want)
            }
        })
    }
}

SafeSubtract Function and Tests

// calc.go
package calc

import "fmt"

func SafeSubtract(a, b int) (int, error) {
    result := a - b
    if result < 0 {
        return 0, fmt.Errorf("result would be negative")
    }
    return result, nil
}

// calc_test.go
package calc

import "testing"

func TestSafeSubtract(t *testing.T) {
    tests := []struct {
        name      string
        a, b      int
        want      int
        wantError bool
    }{
        {"positive result", 5, 3, 2, false},
        {"zero result", 3, 3, 0, false},
        {"negative result", 2, 5, 0, true},
    }

    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            got, err := SafeSubtract(tt.a, tt.b)
            if tt.wantError {
                if err == nil {
                    t.Errorf("SafeSubtract(%d, %d) expected error, got nil", tt.a, tt.b)
                }
                return
            }
            if err != nil {
                t.Errorf("SafeSubtract(%d, %d) unexpected error: %v", tt.a, tt.b, err)
            }
            if got != tt.want {
                t.Errorf("SafeSubtract(%d, %d) = %d; want %d", tt.a, tt.b, got, tt.want)
            }
        })
    }
}

Real-Time Systems for Web Developers: From Theory to a Live Go + React App

Emmanuel Etukudo — Wed, 07 Jan 2026 17:24:03 +0000

Many developers think that “real-time” is about Websockets, Live data, or instant refreshes on web application dashboards.

And although these concepts are closely related to what real-time means, the systems engineering definition is a bit different. A real-time system is not defined by how fast it is, but how predictable it is.

In this tutorial, you’ll learn about what real-time systems are, why most web applications are not real-time, and how to build a soft real-time system with tools you’re likely already familiar with: Go, React, and TypeScript.

At the end of this tutorial, we’ll build a live application that:

processes time-sensitive events
enforces deadlines
drops work when it’s too late
and visualises latency and missed deadlines in real-time

This article will help shape your mind the next time you’re building a real-time system.

Prerequisites
What a Real-Time System Really Means
- Types of Real-Time Systems
- Why Most Web Apps Are Not Real-Time
What We Are Going to Build
System Architecture
Why Go is a Good Fit for Our Use Case
Final Thoughts

Prerequisites

This tutorial assumes that you have basic knowledge of Go, React, and WebSockets, particularly working with go-routines and consuming WebSockets events in React. If you don’t, I strongly recommend reviewing introductory tutorials before continuing so you can get the most out of this guide.

Helpful references include:

Go Concurrency and Goroutines: The official Go blog covers concurrency patterns, including goroutines and channels, which are fundamental for writing concurrent Go code.
WebSockets with Go: A thorough WebSocket tutorial using the popular gorilla/websocket package that shows how to set up a WebSocket server in Go and manage connections and messages. Here’s a Go WebSocket tutorial (with gorilla/websocket).
WebSockets in React: A complete guide to WebSockets in React, explaining how to open and manage a WebSocket connection and handle incoming messages in components. Here’s a complete guide to WebSockets with React.

Technologies we’ll work with include:

Go, for building our backend system and enforcing real-time guarantees
React, for building a responsive frontend UI that displays streamed events
Websocket, for low-latency delivery of data from the backend to the client

What a Real-Time System Really Means

In a traditional web application, correctness is measured by whether the system produced the right result. In a real-time system, correctness is measured by whether the system produced the right result before the deadline. If the result from the test is “No”, then the system has failed – even though the result is correct.

Types of Real-Time Systems

There are a few different types of real-time systems that you should be aware of, each with varying levels of strictness:

Realtime System	Niche applicable
Hard Real-time: Missing a deadline is catastrophic here.	Flight Control & Pacemakers
Soft Real-time: Missing deadline degrades quality but does not crash the system.	Video Streaming & Trading Dashboards
Firm Real-time: Late results are useless and should be discarded.	Car Auction Web/Mobile Apps

The majority of web-based real-time systems fall under the soft real-time category, and that’s exactly what we’ll build here.

Why Most Web Apps Are Not Real-Time

There are many reasons a system may not be real-time, and typically, even those marketed as real-time applications lack this guarantee.

Here’s why:

Websockets guarantee delivery, not timeliness
Massage queues are optimized for durability and throughput
Infinite buffering hides deadlines
User Interfaces (UIs) render when they can, not when they should

In other words, data will arrive eventually, but nothing enforces when it must be processed. That’s exactly the gap we’ll address in this tutorial.

What We Are Going to Build

In this tutorial, we’ll build a Deadline-Aware Live Event Monitor. You can think of it as a simplified real-time system for sensor data, trading events, alerts, or live telemetry.

Our app will have these features and constraints:

Events are generated at a fixed rate
Each event has a deadline
The backend processes events only if they can be completed on time
Late events are marked or dropped
The frontend visualizes:
- processing latency
- missed deadlines
- and system health

This will give us the required metrics to measure the real-time behavior of the system instead of guessing.

System Architecture

The high-level system architecture looks like this:


+-------------+     +------------------+     +----------------+
| Event       | --> | Deadline-Aware   | --> | WebSocket      |
| Generator   |     | Go Processor     |     | Server         |
+-------------+     +------------------+     +----------------+
                                                     |
                                                     v
                                           +----------------+
                                           | React Dashboard|
                                           +----------------+

Let’s break down the responsibilities:

Backend:

Generates time-sensitive events
Enforces deadline
Applies back pressure
Streams result to client (frontend)

Front End:

Consumes real-time events
Renders live metrics
Remains responsive under load

Time is part of your Data Model

In a real-time system, time is explicit, not implicit. This means that each event processed includes:

when it was created
how long is it allowed to live, and
when it was processed

Conceptually, a typical data model for an event looks like this:

{
  id: string
  createdAt: number
  deadlineMs: number
  processedAt?: number
  status: "on-time" | "late" | "dropped"
}

This is the mindset shift we hope to establish: in a real-time system, time is an essential component for your system that guarantees accuracy.

Why Go is a Good Fit for Our Use Case

Go is not a hard real-time language, but it’s excellent for soft real-time workloads. This is because of its:

Cheap goroutines
Structured concurrency with channels
Deadline propagation via context.Context
Simple runtime behavior

Most importantly, Go makes it easy to fail fast, which is essential for real-time systems.

Generating Events with Go

We’ll begin the development of our backend system by first defining the Event struct and creating a fixed-rate event generator function:

type Event struct {
        ID         string
        CreatedAt time.Time
        DeadlineMs  time.Duration
}

Here we’ve created an Event struct with the following properties:

ID a unique identifier that helps in the management of each event processed by the system
CreatedAt to track the time the event was created
Deadline to help evaluate if the event met the assigned deadline or if it failed

Next, we’ll create the event generator startGenerator function:

func startGenerator(out chan<- Event) {
        ticker := time.NewTicker(50 * time.Millisecond)
        defer ticker.Stop()

        for range ticker.C {
                event := Event{
                        ID:         uuid.New().String(),
                        CreatedAt:  time.Now(),
                        DeadlineMs: 100,
                }


                select {
                case out <- event:
                default:
                  // Drop event when load peaks on the goroutine
                }
        }
}

Here, the event generator function accepts a Go channel as a parameter and uses a time.Ticker channel that fires every 50 milliseconds. On each tick, it creates a new Event with a uniqueID, a creation timestamp, and a deadline of 100 milliseconds (DeadlineMs: 100).

The generator then attempts to send the event into the output channel using a non-blocking send. If the channel is ready, the event is delivered immediately. If the channel is not ready (for example, because downstream consumers are slow or overloaded), the default case is executed, and the event is dropped.

Why do we have to drop the event here? Well, because hiding overload drops real-time guarantees. In short, dropping events is a deliberate backpressure strategy: it prevents overload from cascading through the system and protects latency bounds, which is often more important than completeness in real-time streaming systems.

Deadline-aware Processing

Next, we’ll create the processEvent function to handle the processing of the event. In Go, you can enforce deadlines by using context.WithTimeout like this:

func processEvent(event Event) string {
        ctx, cancel := context.WithTimeout(
                context.Background(),
                event.Deadline,
        )

        defer cancel()
        workDone := make(chan struct{})

        go func() {
                time.Sleep(50 * time.Millisecond)
                close(workDone)

        }()

        select {
        case <-workDone:
                return "on-time"
        case <-ctx.Done():
                return "late"
        }
}

Here we’ve intentionally ensured that work finishes before the deadline or it fails immediately.

In the processEvent function, each event is processed under a hard deadline enforced by a context with a timeout. The timeout duration is derived directly from the event’s deadline, meaning the event is only considered valid within the specified time window.

The actual work is executed in a separate goroutine, which simulates processing by sleeping for 50 milliseconds and then signaling completion by closing the workDone channel. We’ve intentionally structured this so that work either completes within the deadline or is treated as a failure immediately.

Applying Back-Pressure

In real-time systems, queues do not solve overload – they merely postpone it. When incoming events arrive faster than they can be processed, a queue continues to grow, increasing the time each event spends waiting.

Buffers can also hide failure. By absorbing excess load, they create the illusion that the system is healthy, even as processing delays grow beyond acceptable limits. This hidden degradation is dangerous because the system continues operating in a compromised state, producing results that are technically correct but operationally useless due to lateness.

As queues and buffers grow, latency increases silently. There is often no explicit error or signal that deadlines are being missed – the system simply becomes slower over time. In real-time systems, this silent latency growth is especially harmful because it violates the assumption that results are delivered within a known and bounded time window.

For these reasons, I strongly recommend that you use bounded channels. When the system becomes overwhelmed, bounded channels enforce back-pressure by refusing additional work. Instead of blocking indefinitely or growing unbounded queues, the system drops events when it cannot keep up.

This behavior makes failures visible. Dropped events are an explicit signal that the system is operating beyond its capacity. Rather than degrading unpredictably, the system degrades in a controlled and observable way. In this context, dropping events is a feature, not a bug, because it preserves latency guarantees for the events that do get processed and allows operators to detect, reason about, and respond to overload conditions immediately.

Streaming Events to the Browser

Next, we’ll build the Websocket broadcast system to push processed events to the frontend using the Gorilla Go websocket package (but feel free to use any package of your choice).

package main

import (
        "encoding/json"
        "net/http"
        "github.com/gorilla/websocket"
)

var upgrader = websocket.Upgrader{
        CheckOrigin: func(r *http.Request) bool {
                return true
        },
}

func wsHandler(out <-chan Event) http.HandlerFunc {
        return func(w http.ResponseWriter, r *http.Request) {
                conn, _ := upgrader.Upgrade(w, r, nil)
                defer conn.Close()
                for event := range out {
                        data, _ := json.Marshal(event)
                        conn.WriteMessage(websocket.TextMessage, data)
                }
        }
}

Here, we simply upgrade an incoming HTTP request to a WebSocket connection and continuously reads events from our Event channel we created earlier, serializing each event to JSON and broadcasting it to the connected client. It acts purely as a transport layer, pushing already-processed events to clients with low latency.

It’s important to note that WebSockets do not make a system real-time. WebSockets merely provide low-latency delivery from the backend to the client. The real-time guarantees are established earlier in the backend pipeline through deliberate design choices: fixed-rate event generation, explicit per-event deadlines, bounded queues, non-blocking sends, deadline-aware processing using contexts, and fail-fast behavior when deadlines are exceeded.

By the time an event is sent over a WebSocket, it has already either met its real-time constraints or been discarded. The WebSocket layer simply transports the result – it doesn’t enforce or create real-time behavior.

Consuming a WebSocket Event (React + TypeScript)

Up until now, we’ve been building the backend of our real-time event generator and broadcast system. In the next sections, we’ll build the frontend of the system using React and TypeScript.

We’ll start by initializing a WebSocket client to consume incoming events from the backend using the traditional WebSocket browser interface.

const socket = new WebSocket("ws://localhost:8080/ws");
socket.onmessage = (event) => {
  const data = JSON.parse(event.data);
  buffer.push(data);
};

Here we simply initialize a new WebSocket, passing along the WebSocket URL from the backend. You have to reference the same URL. Instead of rendering every message immediately, it’s recommended that you always batch updates.

Making React Real-Time Friendly

Next, let’s create a React useRealTimeEvents hook to handle streaming and processing of event broadcasts from the backend. Rendering on every message causes render storms, UI lag, and misleading dashboards. Instead, we render on animation frames.

import { useEffect, useRef, useState } from 'react';
import type { RealTimeEvent } from 'types/types';

function useRealTimeEvents() {
  const [events, setEvents] = useState([]);
  const buffer = useRef([]);

  useEffect(() => {
    const ws = new WebSocket('ws://localhost:8080/ws');
    ws.onmessage = (msg) => {
      buffer.current.push(JSON.parse(msg.data));
    };

    let raf: number;

    const flush = () => {
      if (buffer.current.length > 0) {
        const pendingEvents = buffer.current.slice(0);

        setEvents((prev) => {
          const next = [...prev, ...pendingEvents];
          return next.slice(-50);
        });
      }
      raf = requestAnimationFrame(flush);
    };

    raf = requestAnimationFrame(flush);

    return () => {
      ws.close();
      cancelAnimationFrame(raf);
    };
  }, []);
  return events;
}

export default useRealTimeEvents;

The UI is part of the real-time system. It’s important to note that the system broadcasts messages to the frontend in milliseconds. This behavior already crosses the web browser refresh rate threshold.

In certain situations, you might even guess that the use of setTime should come in handy here. And that’s a good alternative – but there’s a better solution: using requestAnimationFrame().

The requestAnimationFrame() takes in a callback function flush which is regulated by the animation frame, ensuring that we don’t cross the refresh rate threshold before the next repaint. You can learn more about it requestAnimationFrame() here.

Creating the StatsBar Component

Next, let’s create a little statusBar component to show events that arrived within the deadline and those that came in late.

Create a new component StatsBar and add the code below:

import { type FC } from 'react';
import type { RealTimeEvent } from 'types/types';

const StatsBar: FC<{ events: RealTimeEvent[] }> = ({ events }) => {
  const late = events.filter((e) => e.status === 'late').length;
  return (
    "flex flex-row gap-2 bg-gray-500 w-full py-2.5 px-2">
      Events: {events.length} | 
      Late: {late}
    
  );
};

export default StatsBar;

Here we are creating a minimal stats component to show the total number of events and those that arrived late by looping through the incoming event list whose statuses are late.

Creating the Events Table

Next, we’ll create a EventTable component to display the events.

import { type FC } from 'react';
import type { RealTimeEvent } from 'types/types';

export const EventsTable: FC<{ events: RealTimeEvent[] }> = ({ events }) => {
  const formatDate = (date: string) => new Date(date).toLocaleString();
  return (
    "w-full">
      "relative overflow-x-auto bg-neutral-primary-soft shadow-xs rounded-base border border-default">
        "w-full text-sm text-left rtl:text-right text-body">
          "text-sm text-body bg-neutral-secondary-soft border-b rounded-base border-default">
            
            {events.map((e, i) => (
              "bg-neutral-primary border-b border-default"
              >
                
            ))}
          
              "col" className="px-6 py-3 font-medium">
                ID
              
              "col" className="px-6 py-3 font-medium">
                Satus
              
              "col" className="px-6 py-3 font-medium">
                Created At
              
              "col" className="px-6 py-3 font-medium">
                Processed At
              
              "col" className="px-6 py-3 font-medium">
                Deadline
              
            
          
          "px-6 py-4">{e.id.slice(0, 6)}
                "px-6 py-4">{e.status}
                "px-6 py-4">{formatDate(e.createdAt)}
                "px-6 py-4">{formatDate(e.processedAt)}
                "px-6 py-4">{e.deadlineMs}
              
        
      
    
  );
};

"col" className="px-6 py-3 font-medium"> ID	"col" className="px-6 py-3 font-medium"> Satus	"col" className="px-6 py-3 font-medium"> Created At	"col" className="px-6 py-3 font-medium"> Processed At	"col" className="px-6 py-3 font-medium"> Deadline
"px-6 py-4">{e.id.slice(0, 6)}	"px-6 py-4">{e.status}	"px-6 py-4">{formatDate(e.createdAt)}	"px-6 py-4">{formatDate(e.processedAt)}	"px-6 py-4">{e.deadlineMs}

Here we loop through all incoming events and display the event’s id, status, and deadline. These metrics will help us gain insight into the performance of our real-time events broadcast system.

Putting it All Together

At this point, we have implemented a complete, end-to-end real-time application that connects a deadline-aware Go backend to a lightweight React frontend. On the backend, events are generated at a fixed rate, processed under explicit deadlines, and dropped when the system is under load to preserve real-time guarantees. Only events that meet these guarantees are forwarded to connected clients via WebSocket broadcast.

On the frontend, we’ve built the useRealtimeEvents hook to establish a persistent WebSocket connection and continuously stream events from the backend as they arrive. The StatsBar component provides immediate visibility into the system’s behavior by summarizing key characteristics of the event stream, while the EventTable component renders individual events in the order they are received. Together, these components clearly mirror the behavior of the system under normal conditions in real-time.

With both the frontend and backend components in place, the application now functions as a real-time monitor. The backend enforces timelines and correctness, and the frontend simply reflects the outcome of those decisions in real-time. There is no buffering or replay logic on the client side – what’s displayed on the UI is exactly what the system was able to process within the specified deadline.

Finally, replace your Welcome component with the code below to display the StatusBar and EventsTable component.

import { useRealtimeEvents } from "./hooks/useRealtimeEvents";
import { StatsBar } from "./components/StatsBar";
import { EventTable } from "./components/EventTable";

export default function App() {
  const events = useRealtimeEvents();

  return (
    
      Real-Time Event Monitor
      
      
    
  );
}

The React frontend is scaffolded using create-react-app, but the same approach can be used for other frameworks, such as Next.js or Vite. The complete source code, including the frontend and backend, is available in the repository here. You can reach out to me on the X platform if you need my assistance.

Final Thoughts

If you’ve followed this tutorial up to this point, congratulations! You’ve learnt the most critical part of building resilient deadline-aware real-time systems.

Remember, real-time systems are not about being fast, but about how predictable they are. You don’t need a Real-Time Operating System (RTOS), a PhD, or specialized hardware to start learning real-time design.

All you need to excel is to respect time, bound your resources, and accept that sometimes, dropping data is the correct behavior. If you understand that, you’re already thinking like a real-time systems engineer.

How to Build a Smart HomeKit Virtual Light in Go

Rez Moss — Fri, 19 Dec 2025 00:41:41 +0000

Recently, I wanted to understand how smart home devices actually work. When you scan a QR code and a light appears in your Home app, what's really happening? When you tap "on", what bytes travel across your network?

The best way I know to understand something is to build it, so I created a virtual HomeKit light in Go. And in this tutorial, I’ll walk you through how I went about it. We’ll pull back the curtain on smart home protocols so you understand how they work, in depth. Let’s dive in.

What we’ll cover:

What HomeKit Actually Is
The Smart Home Protocol Landscape
How HomeKit Discovery Works
The Pairing Process: What Happens When You Scan the QR Code
The Setup URI: What's in That QR Code?
What Happens When You Toggle the Light
The Accessory Database Model
Persisting Pairing Data
Event Notifications
The Complete Implementation
What I Learned

What You'll Need

Before we start building, let's make sure you have the right setup. This project requires two things:

Go 1.21 or later: We're using some modern Go features, and the brutella/HAP library works best with recent versions. You can check your version with go version. If you need to upgrade, grab the latest from go.dev
An Apple HomeKit environment: This means an iPhone or iPad running iOS 15+ with the Home app. You'll also want to be on the same WiFi network as the machine running your virtual light. HomeKit is entirely local, so your phone needs to be able to reach your development machine directly.

One thing that tripped me up initially is that if you’re running this on a Linux server or inside a container, make sure mDNS traffic isn’t being blocked. Your firewall needs to allow UDP port 5353 (for mDNS discovery) and whatever port your accessory runs on (we'll use 51826). On a Mac this usually just works.

What HomeKit Actually Is

HomeKit is Apple's smart home framework. It's comprised of three things:

a protocol (HAP) that defines how devices talk to each other,
a security model that encrypts and authenticates everything,
and an ecosystem (the Home app, Siri, automations)

Here, we’ll be focused on the protocol layer. We're building something that speaks HAP well enough that Apple's ecosystem accepts it as a real accessory.

The Smart Home Protocol Landscape

Before getting started, let's understand what we're dealing with. There are two protocols at play here:

HomeKit Accessory Protocol (HAP): Apple's original smart home protocol from 2014. It runs over your local WiFi network, uses mDNS for discovery, and encrypts everything with Curve25519 and ChaCha20-Poly1305. Every HomeKit device you've ever used speaks HAP.
Matter: The new industry standard (2022) backed by Apple, Google, Amazon, and others. Matter is actually built on many of the same cryptographic primitives as HAP. When Apple added Matter support, they essentially made HomeKit bilingual, as it can speak both protocols.

Here's what's interesting: Matter devices that connect to Apple Home still end up being controlled through HomeKit's infrastructure. Matter is the pairing and discovery layer, but once a device is in your Home, Apple's ecosystem takes over.

For this project, I'm using the HAP protocol directly via the brutella/hap library. This lets us see exactly what's happening without Matter's additional abstraction layer.

How HomeKit Discovery Works

When you run a HomeKit accessory on your network, it doesn't just sit there waiting. It actively announces itself using mDNS (multicast DNS), also called Bonjour on Apple platforms.

The accessory broadcasts a service record that looks like this:

_hap._tcp.local.
  name: Virtual Light._hap._tcp.local.
  port: 51826
  txt: 
    c#=1          // config number (changes trigger rediscovery)
    ff=0          // feature flags
    id=XX:XX:XX   // device ID (like a MAC address)
    md=Virtual Light  // model name
    pv=1.1        // protocol version
    s#=1          // state number
    sf=1          // status flag (1=not paired, 0=paired)
    ci=5          // category (5=lightbulb)
    sh=XXXXXX     // setup hash

Your iPhone is constantly listening for _hap._tcp.local. broadcasts. When it sees one with sf=1 (unpaired), it shows up in "Add Accessory" as available.

Let's see this in code. Here's the minimal server setup:

package main

import (
    "context"
    "fmt"
    "log"

    "github.com/brutella/hap"
    "github.com/brutella/hap/accessory"
)

func main() {
    light := accessory.NewLightbulb(accessory.Info{
        Name:         "Virtual Light",
        Manufacturer: "My Smart Home",
    })

    server, err := hap.NewServer(hap.NewFsStore("./data"), light.A)
    if err != nil {
        log.Fatal(err)
    }

    server.Pin = "00102003"
    server.Addr = ":51826"

    server.ListenAndServe(context.Background())
}

When ListenAndServe runs, it:

Generates a unique device ID if one doesn't exist
Starts listening on port 51826
Registers the mDNS service record
Waits for connections

At this point, your iPhone can discover it. But what happens when you try to pair it?

The Pairing Process: What Happens When You Scan the QR Code

This is where it gets interesting. HomeKit uses the SRP (Secure Remote Password) protocol for pairing. It’s the same protocol used in things like 1Passwords authentication.

When you scan the QR code or enter the PIN, here's the actual sequence:

Step 1: Pair Setup M1 (iOS → Accessory)

iOS sends: { method: "pair-setup", state: 1 }

Your phone initiates pairing, telling the accessory "I want to pair with you."

Step 2: Pair Setup M2 (Accessory → iOS)

Accessory sends: { 
  state: 2,
  salt: <16 random bytes>,
  public_key: 
}

The accessory generates an SRP salt and public key. The PIN code you entered isn't sent over the network – instead, it's used to derive a verifier locally.

Step 3: Pair Setup M3 (iOS → Accessory)

iOS sends: {
  state: 3,
  public_key: ,
  proof: 
}

Your iPhone uses the PIN to compute its own SRP values and sends a proof that it knows the PIN.

Step 4: Pair Setup M4 (Accessory → iOS)

Accessory sends: {
  state: 4,
  proof: 
}

The accessory verifies the proof. If the PIN was wrong, pairing fails here. If correct, it sends its own proof back.

Step 5-6: Key Exchange

Now both sides have a shared secret derived from SRP. They use this to establish an encrypted channel and exchange long term Ed25519 public keys. These keys are stored permanently. This is why your lights still work after rebooting your router.

The whole dance takes about 2 seconds. After this, sf in the mDNS record changes from 1 to 0 and the accessory disappears from "Add Accessory".

The Setup URI: What's in That QR Code?

The QR code contains a URI that encodes everything needed for pairing:

X-HM://0ABCDEFGH1234
        ^^^^^^^^^^^^
        |       |
        |       +-- Setup ID (4 chars)
        +---------- Encoded payload (9 chars, base-36)

The payload packs three things into 45 bits:

Category: what type of accessory this is (5 = lightbulb, 6 = outlet, 10 = thermostat, and so on)
Flags: how the accessory can pair (2 = supports IP ,wifi pairing , 4 = supports BLE pairing , 6 = supports both)
PIN code as integer

This lets your iPhone know what icon to show and the PIN to use, all from scanning a single QR code.

func generateSetupURI(pin, setupID string, category int) string {
    // PIN "00102003" becomes integer 102003
    var pinInt uint64
    for _, c := range pin {
        if c >= '0' && c <= '9' {
            pinInt = pinInt*10 + uint64(c-'0')
        }
    }

    // Bit layout:
    // [39:32] = category (5 = lightbulb)
    // [31:28] = flags (2 = IP pairing supported)
    // [26:0]  = PIN code
    payload := (uint64(category) << 32) | (2 << 28) | (pinInt & 0x7FFFFFF)

    // Encode as base-36 (0-9, A-Z)
    const chars = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ"
    encoded := ""
    for payload > 0 {
        encoded = string(chars[payload%36]) + encoded
        payload /= 36
    }

    for len(encoded) < 9 {
        encoded = "0" + encoded
    }

    return "X-HM://" + encoded + setupID
}

When your iPhone camera sees X-HM://, it knows this is a HomeKit code. It decodes the payload to extract the category (so it can show the right icon) and the PIN (so you don't have to type it). The setup ID helps with identification when multiple unpaired accessories are on the network.

What Happens When You Toggle the Light

Now for the part I was most curious about. When you tap the light button in the Home app, what actually travels across your network?

Step 1: Encrypted Session

Your iPhone doesn't just send commands in plaintext. Every paired session uses the longterm keys exchanged during pairing to establish a session key. All communication is encrypted with ChaCha20Poly1305.

Step 2: HAP Request

Inside the encrypted channel, HomeKit uses a simple HTTP like protocol. A "turn on" command looks like this:

PUT /characteristics HTTP/1.1
Host: Virtual Light._hap._tcp.local
Content-Type: application/hap+json

{
  "characteristics": [{
    "aid": 1,        // accessory ID
    "iid": 10,       // instance ID (the "On" characteristic)
    "value": true    // new state
  }]
}

Step 3: Accessory Response

The accessory processes the request and responds like this:

HTTP/1.1 204 No Content

If something went wrong, it'll return a status object with an error code.

In our Go code, we hook into this with a callback:

light.Lightbulb.On.OnValueRemoteUpdate(func(on bool) {
    if on {
        fmt.Println("💡 Light ON")
    } else {
        fmt.Println("💡 Light OFF")
    }
})

This callback fires when the value in that PUT request changes. The brutella/hap library handles all the decryption, parsing, and response generation.

The Accessory Database Model

HomeKit organizes everything into a hierarchy:

Accessory (aid=1)
└── Services
    ├── AccessoryInformation (iid=1)
    │   ├── Name (iid=2)
    │   ├── Manufacturer (iid=3)
    │   ├── Model (iid=4)
    │   └── SerialNumber (iid=5)
    │
    └── Lightbulb (iid=9)
        ├── On (iid=10)           ← boolean
        ├── Brightness (iid=11)   ← int 0-100
        └── Hue (iid=12)          ← float 0-360

Each characteristic has an iid (instance ID). When you change brightness to 75%, the PUT request targets aid=1, iid=11, value=75.

This model is why HomeKit accessories are interoperable. Every lightbulb, regardless of manufacturer, has the same characteristic structure.

Persisting Pairing Data

When your accessory pairs with a controller (iPhone), it stores:

The controller's Ed25519 public key
A controller ID (36chars UUID)
Permission level (admin or regular user)

The accessory also has its own keypairs that must persist across restarts. If you lose this, all paired controllers become orphaned – that is, they think they’re paired, but the accessory doesn't recognize them.

As mentioned earlier, we need to save pairing info so that if the app/device restarts, it can communicate with Homekit again. You could use a database, but for a single accessory, a JSON file works fine. If the process crashes mid-session, you won’t lose pairing data.

I wrote a simple JSON store to keep everything in one file:

type JSONStore struct {
    path string
    data map[string][]byte
    mu   sync.RWMutex
}

func (s *JSONStore) Set(key string, value []byte) error {
    s.mu.Lock()
    defer s.mu.Unlock()
    s.data[key] = value
    return s.save()
}

func (s *JSONStore) Get(key string) ([]byte, error) {
    s.mu.RLock()
    defer s.mu.RUnlock()
    if v, ok := s.data[key]; ok {
        return v, nil
    }
    return nil, fmt.Errorf("key not found: %s", key)
}

The HAP library stores several keys:

uuid – accessory's unique identifier
public / private – Ed25519 keypair
*-pairings – paired controller keys

If you delete this JSON file, the accessory (our virtual-light) forgets all its paired controllers. Your iPhone still thinks it's paired, but the accessory doesn't recognize it anymore – you'll see "No Response" in the Home app. The fix removes the accessory from the Home app and pairs it fresh using the QR code again.

Event Notifications

One thing I didn't expect is that HomeKit supports push notifications from accessories. When our light state changes (maybe from a physical switch), we can notify all connected controllers:

light.Lightbulb.On.SetValue(true)  // This triggers notifications

Under the hood, the accessory maintains persistent connections with controllers. When a characteristic changes, it sends an EVENT message:

EVENT/1.0 200 OK
Content-Type: application/hap+json

{
  "characteristics": [{
    "aid": 1,
    "iid": 10,
    "value": true
  }]
}

This is how your Home app updates in realtime when someone else turns on a light.

The Complete Implementation

Here's everything together:

package main

import (
    "context"
    "encoding/json"
    "fmt"
    "log"
    "os"
    "os/signal"
    "sync"
    "syscall"

    "github.com/brutella/hap"
    "github.com/brutella/hap/accessory"
    "github.com/skip2/go-qrcode"
)

const (
    pinCode  = "00102003"
    setupID  = "VLTX"
    category = 5
    dbFile   = "data.json"
)

type JSONStore struct {
    path string
    data map[string][]byte
    mu   sync.RWMutex
}

func NewJSONStore(path string) *JSONStore {
    s := &JSONStore{
        path: path,
        data: make(map[string][]byte),
    }
    s.load()
    return s
}

func (s *JSONStore) load() {
    file, err := os.ReadFile(s.path)
    if err != nil {
        return
    }
    json.Unmarshal(file, &s.data)
}

func (s *JSONStore) save() error {
    file, err := json.MarshalIndent(s.data, "", "  ")
    if err != nil {
        return err
    }
    return os.WriteFile(s.path, file, 0644)
}

func (s *JSONStore) Set(key string, value []byte) error {
    s.mu.Lock()
    defer s.mu.Unlock()
    s.data[key] = value
    return s.save()
}

func (s *JSONStore) Get(key string) ([]byte, error) {
    s.mu.RLock()
    defer s.mu.RUnlock()
    if v, ok := s.data[key]; ok {
        return v, nil
    }
    return nil, fmt.Errorf("key not found: %s", key)
}

func (s *JSONStore) Delete(key string) error {
    s.mu.Lock()
    defer s.mu.Unlock()
    delete(s.data, key)
    return s.save()
}

func (s *JSONStore) KeysWithSuffix(suffix string) ([]string, error) {
    s.mu.RLock()
    defer s.mu.RUnlock()
    var keys []string
    for k := range s.data {
        if len(k) >= len(suffix) && k[len(k)-len(suffix):] == suffix {
            keys = append(keys, k)
        }
    }
    return keys, nil
}

func generateSetupURI(pin, setupID string, category int) string {
    var pinInt uint64
    for _, c := range pin {
        if c >= '0' && c <= '9' {
            pinInt = pinInt*10 + uint64(c-'0')
        }
    }

    payload := (uint64(category) << 32) | (2 << 28) | (pinInt & 0x7FFFFFF)

    const chars = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ"
    encoded := ""
    for payload > 0 {
        encoded = string(chars[payload%36]) + encoded
        payload /= 36
    }

    for len(encoded) < 9 {
        encoded = "0" + encoded
    }

    return "X-HM://" + encoded + setupID
}

func main() {
    light := accessory.NewLightbulb(accessory.Info{
        Name:         "Virtual Light",
        Manufacturer: "My Smart Home",
    })

    light.Lightbulb.On.OnValueRemoteUpdate(func(on bool) {
        if on {
            fmt.Println("💡 Light ON")
        } else {
            fmt.Println("💡 Light OFF")
        }
    })

    store := NewJSONStore(dbFile)

    server, err := hap.NewServer(store, light.A)
    if err != nil {
        log.Fatal(err)
    }

    server.Pin = pinCode
    server.SetupId = setupID
    server.Addr = ":51826"

    fmt.Println("==============================================")
    fmt.Println("       Virtual HomeKit Light")
    fmt.Println("==============================================")
    fmt.Println("PIN: 001-02-003")
    fmt.Println()

    setupURI := generateSetupURI(pinCode, setupID, category)
    if qr, err := qrcode.New(setupURI, qrcode.Medium); err == nil {
        fmt.Println(qr.ToSmallString(false))
    }

    fmt.Println("Manual: Home app → + → More Options → Virtual Light")
    fmt.Printf("Data stored in: %s\n", dbFile)
    fmt.Println("==============================================")

    ctx, cancel := context.WithCancel(context.Background())
    go func() {
        c := make(chan os.Signal, 1)
        signal.Notify(c, os.Interrupt, syscall.SIGTERM)
        <-c
        cancel()
    }()

    fmt.Println("Running... (Ctrl+C to stop)")
    server.ListenAndServe(ctx)
}

Run it, pair it, and watch the terminal as you toggle from your phone. Each "💡 Light ON" is the end of an encrypted request that traveled from your phone, through your router, to this Go process.

What I Learned

Building this cleared up several things I'd been fuzzy on:

HomeKit is entirely local. There are no cloud servers involved in controlling devices – your commands go directly from phone to device over your LAN. This is why HomeKit devices work when your internet is down.
The security model is solid. SRP for pairing means the PIN never crosses the network. Ed25519 + ChaCha20 for sessions means that even someone sniffing your WiFi sees only encrypted blobs.
Matter doesn't replace HAP. At least not in Apple's ecosystem. Matter handles discovery and pairing across ecosystems, but Apple Home still uses HAP concepts internally.
The protocol is HTTPish. Once you get past the encryption, it’s just PUT/GET requests with JSON bodies – surprisingly approachable.

Thanks for reading!

The code is here if you want to experiment yourself. You could try adding brightness control, or create a switch instead of a light. The best way to understand a protocol is to speak it ;)

How to Build an LSM Tree Storage Engine from Scratch – Full Handbook

Ramesh Sinha — Thu, 18 Dec 2025 20:25:02 +0000

Databases are one of the most important parts of a software system. They allow us to store huge amounts of data in an organized way and retrieve it efficiently when we need it.

In the early days, when the volume of data was relatively small, engineers prioritized fast data retrieval and stored data in B-tree structures that made searching efficient.

But over time, we started building systems that needed to ingest massive amounts of data like logs, metrics, likes, chats and tweets. This made it necessary to design a storage system that would make writing faster.

One such storage system is the LSM-tree (Log-Structured Merge tree).

In this tutorial, rather than immediately diving into the theoretical concepts of an LSM-Tree Storage system, I’ll take a practical, problem-driven approach. I believe that learning through solving problems is far more effective and engaging than simple memorization of concepts.

By approaching these ideas progressively, my goal is to guide you step by step through real-world engineering challenges and solutions, giving you a front-row seat to the intricacies of building a robust storage system from scratch.

We’ll begin by identifying real-world challenges that arise in database design – like handling write-heavy workloads, ensuring data durability, or managing efficient storage. These challenges will set the stage for each feature and component of LSM-Trees.

Through this method, we’ll explore the foundations of LSM-Tree storage systems and dive deeper into their key components: MemTable, SSTable, Write-Ahead Log (WAL), and Manifest File.

We’ll also examine the Write and Read paths, explore Durability and Crash-Recovery mechanisms, and conclude with one of the most critical processes: Compaction.

By the end of this handbook, you’ll understand not just what these components are but also why they are designed the way they are and how they solve the unique challenges of building modern, high-performance databases.

What We’ll Cover:

Prerequisites
What is an LSM Tree?
Preface: Setting up to Build an LSM-Tree Database
Conclusion
- Complete Code

Prerequisites

While this tutorial is designed to be comprehensive and approachable, it’ll be helpful if you come in with some foundational knowledge in the following areas:

Programming in Golang: Familiarity with Go syntax, error handling, and standard libraries (example os, encoding/gob, container/heap) will make it easier to work through the implementation examples.
Basic data structures and algorithms: Concepts such as maps, heaps, some sorting algorithms, and early termination are leveraged throughout the tutorial.
Understanding persistent storage: Awareness of the differences between in-memory and disk-based storage, as well as sequential versus random read/write operations will be helpful in grasping performance-related trade-offs.
General database knowledge: If you're familiar with key-value databases or CRUD operations (Create, Read, Update, Delete), you’ll have a head start.
Concurrency: Basic understanding of threads and concurrency.

While having experience in these areas will deepen your understanding of the concepts and reduce the learning curve, I will provide sufficient detail and practical explanations at every step ensuring you gain the insights necessary to follow along and build your own LSM-tree-based storage engine.

What is an LSM Tree?

A log-structured merge-tree (or LSM tree) is a data structure that makes database writes super fast by recording new data in memory first, then periodically sorting and merging it into larger files on disk.

The “log” in its name refers to the fact that it saves data in a log-structured format (rather than simply storing it). We will come to what those logs are in a little bit.

LSM trees keep appending new data to the existing data, instead of looking for something that exists and updating it. In other words, you don't have to spend any CPU cycles thinking about where to store data – just append it at the end.

An LSM tree also has "tree" in its name, but does it actually store data in a tree? Not really. The “tree” here is mostly an abstract concept. It refers to the hierarchical organization of levels (L0, L1, L2, and so on), not a tree data structure with nodes and pointers. Again, we will come to those levels in a little bit, but for now, let’s just say it makes sense to call it a tree given that it stores data in a leveled fashion.

Just note that there isn't a tangible tree structure in play (like a binary trees or graph) – it’s not node-based storage.

Finally, there is the "merge" part of the name. For now, suffice it to say that you’ll soon see how this storage engine merges data to save storage by avoiding duplication.

Personally, I think that "Log-Structured Merge System" would be clearer than "tree," but "LSM tree" is the established term in the industry, so that's what we'll use.

Preface: Setting up to Build an LSM-Tree Database

Now that we have set the context, lets put this theory into practice and start building our own LSM-tree-based database storage engine from scratch.

To follow along with this tutorial:

Make sure you have Golang installed on your system. If not, you can download and install it from the official Go website.
Set up your development environment and create a new Go module for this project by running: go mod init lsm-db
Keep a code editor or IDE ready to try out the examples.

Initial Feature Set: Laying the Foundation of the Database System

When I’m designing or building a system, I like to think that the system already exists, and I assume that I can just start calling functions that support the features of the system. I’ll follow that pattern here and assume that the following functions of the LSM tree exist and we can invoke those functions from main.go.

db, err := NewDB[string, string](3, 3) // there is a feature to create new with some parameters we will get to
db.Put("a", "apple") // a feature to add key value
db.Delete("a") // a feature to delete a key
val, _ := db.Get("a") // a feature to get value given a key

As we progress through this journey, I’ll introduce essential features such as in-memory storage, flushing data to disk, and handling duplicate keys. We’ll also explore more advanced components, including a Write-Ahead Log (WAL) to ensure crash tolerance, a Manifest file to maintain the database state across application restarts, and a Compaction process to clean up redundant or stale data by merging older SSTables.

By the end of this tutorial, you will gain a clear understanding of how all these components work together to form a robust and efficient LSM-tree-based storage system.

MemTable: In-Memory Data Storage

We’re building a database storage system, so of course you’ll need a way to store data. This means you need some kind of backing storage. This backing storage in an LSM tree is called a MemTable. The "Mem" refers to its in-memory storage. The benefit of in-memory storage is that it’s orders of magnitude faster than storing on disk.

For simplicity, at the core of the MemTable you can use a map (or dictionary depending on the programming language) as the underlying data structure to store key-value pairs. The map allows for fast lookups, insertions, and deletions, making it ideal for in-memory storage where performance is crucial. So the structure for MemTable will look like:

type MemTable[K comparable, V any] struct {
    data map[K]V // this is primary storage map. It's generic so that you
                  // can store any kind of data
}

Above code defines a MemTable struct, where data is a map that acts as the main storage for our key-value pairs. Since the data field is a map, you’ll be able to quickly add, retrieve, or delete values associated with a given key.

You must have noticed something new in the code. The use of . This syntax is Go’s generic types feature, which allows us to write flexible code that can handle different data types.

Generics are a way to write code that is independent of any specific data type. They allow you to write functions and data structures that can work with a string, int, float, or any custom type you define, without sacrificing type safety.

In the above code, K and V are type parameters. They say: "This MemTable can work with any Key type K that is comparable, and any value type V."

Now that you have the MemTable, think of what functions it should provide to its clients. Well, the clients need to be able to save and retrieve values associated with a key, so the following functions would fit naturally:

func (m *MemTable[K, V]) Put(key K, value V) {
    m.data[key] = value
}

func (m *MemTable[K, V]) Get(key K) (V, bool) {
    value, ok := m.data[key]
    var zero V
    if !ok {
        return zero, false
    }
    return value, true
}

The above code has Put and a Get functions – let’s break them down:

Put: This function allows the client to insert a key-value pair into the MemTable. If the key already exists in the map, its value will be updated with the new value provided as an argument. This is effectively the write operation of our key-value store.
Get: This function is responsible for retrieving a value associated with a give key from the MemTable. It returns two values, the value itself (of type V) and a boolean (true or false). The boolean indicates whether the key was found in the map. If the key does not exist, the function return a zero value (more on that below) along with false.

Did you notice var zero V?

It's pretty interesting. Think of a situation where we don't get a value from the map – say the key is not there, or something else is wrong. What should the function Get return in that case? Can it return an int (0), or a string "Not found", or some random object (foo)? You don't know anything about the type yet (Generics), so you can't tell it what to return.

In this case, the compiler comes to the rescue. Go has this zero value concept: everything should have a zero value. An int has 0, string has "", bool has false, and pointer, slice, and map have nil. By saying var zero V you are telling the compiler, "I don't know the type yet, you figure out the type while compiling and put that type here as the return type." Neat!

I missed one thing though: how would a client invoke these functions? Right, we need a way to build the MemTable type.

To construct and initialize a MemTable, we can use a factory function: a common programming pattern for creating and returning new objects or instances without directly exposing the underlying implementation details.

func NewMemTable[K comparable, V any]() *MemTable[K, V] {
    return &MemTable[K, V]{
        data: make(map[K]V),
    }
}

Notice how we’ve initialized the data field using the built-in make function. Here’s why we do this:

Go has a built-in function called make, which is used to allocate and initialize slices, maps, and channels. This allocation ensures that they are ready for use without the risk of runtime panics.

You might wonder, why not use the new function to allocate the map? After all, developers coming from other programming backgrounds (like C++ or Java) might expect to use new for all types of memory allocation. But Go differentiates how it manages memory for composite types versus basic/numeric types, and that’s where make comes in.

This distinction matters because the new function only allocates memory for an object and returns a pointer to that memory. The object itself is not initialized, meaning that while the memory is allocated, the map isn’t ready to use. If we try to perform operations (like adding a key-value pair) on a map only allocated using new, it will cause a runtime panic because the map wasn’t correctly initialized.

For example:

m := new(map[string]int) // Allocates a pointer to an uninitialized map
(*m)["a"] = 1            // This will panic because the map is not initialized

On the other hand, make both allocates and initializes the map, ensuring it’s fully functional right away. That’s why the correct way to create a map is:

m1 := make(map[string]int) // Initializes the map properly
m1["a"] = 1                // This works as expected

Now that you have the MemTable which can store data in memory, let's hook it up and use it.

But before that, do you remember at the beginning that I used functional invocations like db.Put and db.Get? Well, what is db? Because we are building a database storage system, it makes more sense to name the interface db instead of MemTable, right? And to be honest, it seems like MemTable is going to be part of the database system, not the whole system, doesn't it?

Even if it's not intuitive at the moment to define something like a DB type, let's just do it. Trust me, it will start to get clearer as we move along. This db type will wrap adding and retrieving data from MemTable.

type DB[K comparable, V any] struct {
    memtable *MemTable[K, V]
}

// factory function for DB type
func NewDB[K comparable, V any]() (*DB[K, V], error) {
    memtable := NewMemTable[K, V]()
    return &DB[K, V]{
        memtable: memtable,
    }, nil
}

Let's just define the Put and Get functions which will invoke corresponding functions in MemTable:

func (db *DB[K, V]) Put(key K, value V) error {
    db.memtable.Put(key, value)
    return nil
}

func (db *DB[K, V]) Get(key K) (V, error) {
    if val, ok := db.memtable.Get(key); ok {
        return val, nil
    }
    var zero V
    return zero, errors.New("key not found")
}

Let's integrate whatever we’ve built so far and run it. To run add below code in main.go and run using go run main.go

db, err := NewDB[string, string]()
if err != nil {
    log.Fatalf("Failed to create DB: %v", err)
}
db.Put("a", "apple")
val, _ := db.Get("a")
log.Printf("Get('a') = %s (should be 'apple')", val)

Look at that, you have built an in-memory database where your clients can store and fetch data from. It’s using generics so you can store any kind of values (int, string, objects).

Now say you shipped this solution and then it crashes. Your clients will lose all the data. Why will this crash? For one thing, memory is limited and at some point you’re going to run out of it. So there are two major problems with just in-memory storage:

It's not durable.
Unbounded memory usage is going to crash the system.

How do we solve these problems?

Here's a thought: what if we flush the MemTable data to disk at some regular interval? That way we can ensure that MemTable doesn't grow out of bounds. Also if the db crashes, we won’t lose all the data. We’ll still lose the data that hasn’t been flushed yet, but that's way better than losing all of it.

SSTable: Persisting Data for Durability

An SSTable is a sorted string table. I wish they’d called it a "Secondary Storage" table, but historically keys and values were strings – hence sorted string table. An SSTable is a persistent, ordered, and immutable file that stores key-value pairs. It’s a file stored on disk, so it's pretty clear that it’s persistent (durable).

Let’s discuss a couple key features of the SSTable:

It’s ordered: There is an incentive to store the keys in a sorted order, and it makes searching keys faster and efficient. If not for that, you'd have to scan the whole file to be able to find a key. Later, I will point out some code that leverages sorted storage.
It’s immutable: Once an SSTable file is written, it can’t be modified. To update or delete a key, you must write a new record in a newer SSTable. This simplifies the design and makes reads and writes very predictable.

But wait, how does that simplify the design?

One of the most complex things in software engineering is dealing with concurrency. Let’s say you’re writing to a file and another thread updates it underneath. How do you know you have the correct data?

With immutable design, you don't have to worry about this at all. You are 100% confident that the data you are reading has not been altered by anybody else. I’ll take that as a massive simplification: you don't have to deal with locks, starvation, staleness, and so on.

How does it make the write path predictable?

I will answer this partially here and come back to it when we have completed some more implementation. You’ll see that every write in our code follows the exact same steps. There is not a single different condition or edge case.

In a traditional database (using a B-Tree), a typical write involves:

Finding the data on disk.
Reading the block of data from disk into memory.
Modifying the data in memory.
Writing the entire block back to disk.

The more steps, the more unpredictable performance can get, because the write can be fast if data is already in the memory cache or slow if there are multiple disk seeks needed.

Granted, our code is an overly simplified version, but the extension of this concept still stands true in real LSM implementations.

How does it make the read path predictable?

Read is predictable because any number of threads can read the same SSTable file at the same time without any problem, with full confidence that data has not been updated.

In contrast, when reading from a mutable data structure, you have to worry that another thread might be in the process of changing the data you are trying to read.

To prevent this, B-Tree-based databases use complex locking mechanisms, and that adds overhead and unpredictability.

I should raise a caution here: the Read in LSM tree storage is not always predictable. It can be faster if data is read from memory and it can be very slow if multiple SSTables need to be looked up to find the key.

Having said that, you don't have to worry about other performance bottlenecks because of locks. Meaning, in B-Tree storage, your read query can be slower because another write query is holding a lock. In simple low-concurrency use cases, you will mostly get amazing read performance from a B-Tree structure, but this advantage wears off as concurrency increases.

LSM tree was built for highly concurrent, write-heavy use cases, and at times slower reads are a trade-off.

The takeaway that gives you ammunition to design better is that B-trees are better for read-heavy workloads. Reads are generally faster and more consistent, but performance can have unpredictable outliers under high write concurrency due to locking.

An LSM tree is better for write-heavy workloads. Writes are much faster. Reads are generally slower and more variable, but their performance profile is more predictable under high write concurrency because there is no read-write locking.

Let's implement an SSTable to see how it works.

The write path:

func writeSSTable[K comparable, V any](memtable *MemTable[K, V], path string) (*SSTable[K, V], error) {
    file, err := os.Create(path)
    if err != nil {
        return nil, err
    }
    defer file.Close()

    pairs := make([]Pair[K, V], 0, len(memtable.data))
    for k, v := range memtable.data {
        pairs = append(pairs, Pair[K, V]{Key: k, Value: v})
    }

    sort.Slice(pairs, func(i, j int) bool {
        return any(pairs[i].Key).(string) < any(pairs[j].Key).(string)
    })

    encoder := gob.NewEncoder(file)
    for _, pair := range pairs {
        if err := encoder.Encode(pair); err != nil {
            return nil, err
        }
    }

    return &SSTable[K, V]{path: path}, nil
}

The following things are important to note from the above code:

sort.Slice: Remember I spoke about order earlier? So we store data in the SSTable in a sorted fashion, and we will see how we leverage it in the read path.
I have used the gob encoding package. An encoder makes life simpler for you because it streams the data to and from Go data structures to binary streams that can be stored on disk. It handles all the complexity of representing types, field names, and values in a standardized binary format, so that you don't have to.

The read path:

func (s *SSTable[K, V]) Get(key K) (V, error) {
    file, err := os.Open(s.path)
    if err != nil {
        var zero V
        return zero, err
    }
    defer file.Close()

    decoder := gob.NewDecoder(file)

    for {
        var pair Pair[K, V]
        if err := decoder.Decode(&pair); err != nil {
            if err == io.EOF {
                break
            }
            var zero V
            return zero, err
        }

        // for simple comparison we are assuming key is just string
        keyInDB := any(pair.Key).(string)
        if keyInDB == any(key).(string) {
            if any(pair.Value).(string) == TOMBSTONE {
                var zero V
                return zero, ErrDeleted
            }
            return pair.Value, nil
        }

        if keyInDB > any(key).(string) {
            var zero V
            return zero, ErrNotFound
        }
    }

    var zero V
    return zero, ErrNotFound
}

On the read path, look at keyInDB > any(key).(string). This is one of the examples of how we took advantage of storing data in a sorted key order. The moment we find a key in the SSTable that is greater than the key we are looking for, we stop looking because it’s obvious all other keys will be greater than this, so we won't find our key anymore.

Now that you have implemented the SSTable, you just have to decide when to flush data from the MemTable to the SSTable. You can just define max size for MemTable and flush it to disk on the write path when the max size is reached.

I am skipping some variables, boilerplate code, and simplifying things for brevity. I will post a GitHub link with the complete implementation later.

type DB[K comparable, V any] struct {
    memtable        *MemTable[K, V]
    maxMemtableSize int
    memtableSize    int
    sstables        []*SSTable[K, V]
    sstableCounter  int
}

func NewDB[K comparable, V any](maxMemtableSize int) (*DB[K, V], error) {
    sstables := make([]*SSTable[K, V], 0)
    memtable := NewMemTable[K, V]()

    return &DB[K, V]{
        memtable:        memtable,
        maxMemtableSize: maxMemtableSize,
        sstables:        sstables,
    }, nil
}

func (db *DB[K, V]) Put(key K, value V) error {
    db.memtable.Put(key, value)
    db.memtableSize++

    if db.memtableSize >= db.maxMemtableSize {
        if err := db.flushMemtable(); err != nil {
            return err
        }
    }

    return nil
}

func (db *DB[K, V]) flushMemtable() error {
    sstablePath := fmt.Sprintf("data-%d.sstable", db.sstableCounter)
    sstable, err := writeSSTable(db.memtable, sstablePath)
    if err != nil {
        return err
    }

    db.sstables = append(db.sstables, sstable)
    db.sstableCounter++
    db.memtable = NewMemTable[K, V]()
    db.memtableSize = 0

    return nil
}

You'll notice that every time we flush to disk, we write to a new SSTable versus using a single SSTable for the whole database. This is the immutability aspect we discussed earlier.

func (db *DB[K, V]) Get(key K) (V, error) {
    if val, ok := db.memtable.Get(key); ok {
        return val, nil
    }

    for i := len(db.sstables) - 1; i >= 0; i-- {
        sstable := db.sstables[i]
        val, err := sstable.Get(key)

        if err != nil {
            var zero V

            if err == ErrDeleted {
                return zero, ErrNotFound
            }
            if err == ErrNotFound {
                continue
            }
            return zero, err
        }

        return val, nil
    }

    var zero V
    return zero, ErrNotFound
}

One important aspect to note on the read path is that we are reading the newest SSTable first. This is because the newest SSTable has the most updated value for the key.

So, say you have a key "a" with value "apple", and along the way you update that value for "a" to be "apricot". You'd have flushed it to a new SSTable (for immutability), and so if you were to read an older SSTable, first you'd get the older value. So by reading the newer SSTable first, we get the correct value and we don't have to worry about updating older SSTables.

The WAL (Write Ahead Log ): Crash Recovery Made Simple

Now that we have an SSTable, our data is durable and we are safe from losing data upon crashes. Are we really safe, though? Think of a scenario where a crash happens before we flush to the SSTable. We know MemTable has a max threshold, and until then, data lives in memory. So we’re still prone to losing data if a crash happens before the flush.

This is where the WAL (Write Ahead Log) comes into the picture. It’s the single most important aspect of the LSM tree.

We’ll follow a simple rule: "Before we write a piece of data to the in-memory MemTable, we first write it to a log file on disk."

If a crash happens and the database starts again, the first thing it does is look for a WAL, read it if one is found, and replay all the data into MemTable. This process reconstructs the MemTable to the exact state it was in right before the crash.

It's natural to think that if all of your writes are first written to disk it will impact performance. You aren’t wrong, but at the same time there are nuances.

The writes to WAL are different in that they are append-only sequential writes, meaning random disk seeks are not required. On a traditional spinning hard drive (HDD), this is fast because the disk's read/write head does not have to move to a new location. On a modern solid-state drive (SSD), sequential writes are also much faster than random writes.

Whatever small performance impact we accept is a trade-off for durability.

Now that we know what WAL does, let's implement it. Two key functions of WAL are to write to a file on disk and replay MemTable upon start.

Note that in the factory function below (NewWAL), the file has been opened in append mode.

func NewWAL[K comparable, V any](path string) (*WAL[K, V], error) {
    file, err := os.OpenFile(path, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644)
    if err != nil {
        return nil, err
    }
    return &WAL[K, V]{
        file:    file,
        encoder: gob.NewEncoder(file),
    }, nil
}

func (wal *WAL[K, V]) Write(key K, value V) error {
    entry := WALEntry[K, V]{Key: key, Value: value}
    return wal.encoder.Encode(&entry)
}

func ReplayWAL[K comparable, V any](path string) (*MemTable[K, V], error) {
    memtable := NewMemTable[K, V]()
    file, err := os.Open(path)
    if err != nil {
        if os.IsNotExist(err) {
            // If the file doesn't exist, that's fine. Return an empty memtable.
            return memtable, nil
        }
        return nil, err
    }
    defer file.Close()

    decoder := gob.NewDecoder(file)
    for {
        var entry WALEntry[K, V]
        if err := decoder.Decode(&entry); err != nil {
            if err == io.EOF {
                break // We've reached the end of the file.
            }
            return nil, err
        }
        memtable.Put(entry.Key, entry.Value)
    }

    return memtable, nil
}

A couple notes about the above code:

NewWAL: This function creates an instance of the WAL for our database. It takes in the file path where the WAL data should be stored and opens the file using Go’s os.OpenFile function. Also, a gob.Encoder is initialized to simplify the encoding of Go data structures into binary format for efficient storage in the WAL file.
Write: The Write function appends a new key-value pair to the WAL file. Every write operation to the MemTable first calls this function to ensure the update is durably recorded:
ReplayWAL: This is the most important function. In the event of crash, this function comes to our rescue by reconstructing the MemTable from the WAL file. It replays the entries stored in the WAL file and writes it into MemeTable. Following it how it works:
1. The function begins by creating a new empty MemTable instance that will be populated with key-value pairs.
2. It then attempts to open the WAL file. If the file does not exist (example – if this is the first startup), the function assumes there’s nothing to recover and simply returns the empty MemTable.
3. A gob.Decoder is used to read the WAL file, which helps to deserialize the saved binary-encoded WALEntry data back into key-value pairs.
4. For each successfully decoded WALEntry, the key-value pair is added back into the MemTable using the Put function.

With this, the database can fully recover its state by replaying all the operations recorded in the WAL.

As far as integration is concerned, every time you create a new DB, you should think of replaying from an existing WAL and opening the WAL in append mode. Also, Put should first write to WAL.

func NewDB[K comparable, V any](maxMemtableSize int) (*DB[K, V], error) {
    walPath := "db.wal"
    memtable, err := ReplayWAL[K, V](walPath) // this is the replay
    if err != nil {
        return nil, err
    }
    //open WAL in append mode
    wal, err := NewWAL[K, V](walPath)
    if err != nil {
        return nil, err
    }

    return &DB[K, V]{
        memtable:        memtable,
        maxMemtableSize: maxMemtableSize,
        memtableSize:    len(memtable.data),
        wal:             wal,
        walPath:         walPath,
        sstables:        make([]*SSTable[K, V], 0),
    }, nil
}

func (db *DB[K, V]) Put(key K, value V) error {
//first write to WAL
    if err := db.wal.Write(key, value); err != nil {
        return err
    }

    db.memtable.Put(key, value)
    db.memtableSize++

    if db.memtableSize >= db.maxMemtableSize {
        if err := db.flushMemtable(); err != nil {
            return err
        }
    }

    return nil
}

Manifest File: Tracking the State of the Database

By this point, the database is pretty robust and durable, but an important question lingers: upon restarts, how does our database know about SSTables? Knowing about all SSTables is important for fetching data.

So say our database crashed after writing several SSTables. Without knowing about these SSTables, the database will create a new slice of SSTables and all of our old data is gone – queries won't read those files.

To solve this problem, we introduce an inventory of SSTables called MANIFEST. Every time we successfully create a new SSTable in flushMemtable, we add its path to the MANIFEST and save the MANIFEST to disk.

The very first thing NewDB does on startup is read the MANIFEST. This gives it the list of all the file paths, and it uses this list to perfectly reconstruct its SSTables slice.

In short, MANIFEST determines the state of the DB.

Manifest contains a slice of SSTablePaths. The Read function will read the MANIFEST file to restore the knowledge of the SSTables. The Write function will write a new manifest file.

type Manifest struct {
    SSTablePaths []string
}

func ReadManifest(path string) (*Manifest, error) {
    file, err := os.Open(path)
    if err != nil {
        if os.IsNotExist(err) {
            // If manifest doesn't exist, return empty manifest
            return &Manifest{SSTablePaths: []string{}}, nil
        }
        return nil, err
    }
    defer file.Close()

    var manifest Manifest
    decoder := gob.NewDecoder(file)
    err = decoder.Decode(&manifest)
    if err != nil {
        return nil, err
    }

    return &manifest, nil
}

func WriteManifest(path string, manifest *Manifest) error {
    tmpPath := path + ".tmp"
    file, err := os.Create(tmpPath)
    if err != nil {
        return err
    }

    encoder := gob.NewEncoder(file)
    if err := encoder.Encode(manifest); err != nil {
        file.Close()
        os.Remove(tmpPath)
        return err
    }

    if err := file.Close(); err != nil {
        return err
    }
    // Atomic Rename
    return os.Rename(tmpPath, path)
}

You'll notice that we aren’t modifying the existing MANIFEST file directly. Instead, we’re creating a temporary file, writing all the data to it, closing it, and then atomically renaming it to replace the old MANIFEST.

The os.Rename() operation is atomic on most filesystems, meaning it either completely succeeds or completely fails – there's no in-between state. This is crucial because if the system crashes while updating the MANIFEST, we need to ensure we don't end up with a corrupted file. We’ll discuss this again below when we’re talking about compaction.

With this approach, we either have the old valid MANIFEST or the new valid MANIFEST, never a partially written corrupted file.

From an integration standpoint, NewDB will read the manifest and set its SSTable slice based on that. The flush method, given that it writes to SSTable, will also write SSTable info to manifest to keep the db updated about new SSTables.

type DB[K comparable, V any] struct {
    memtable        *MemTable[K, V]
    maxMemtableSize int
    memtableSize    int
    sstables        []*SSTable[K, V]
    sstableCounter  int
    wal             *WAL[K, V]
    walPath         string
    manifest        *Manifest
    manifestPath    string
}

func NewDB[K comparable, V any](maxMemtableSize int) (*DB[K, V], error) {
    walPath := "db.wal"
    memtable, err := ReplayWAL[K, V](walPath)
    if err != nil {
        return nil, err
    }

    wal, err := NewWAL[K, V](walPath)
    if err != nil {
        return nil, err
    }

    manifestPath := "MANIFEST"
    manifest, err := ReadManifest(manifestPath)
    if err != nil {
        return nil, err
    }

    sstables := make([]*SSTable[K, V], len(manifest.SSTablePaths))
    for i, path := range manifest.SSTablePaths {
        sstables[i] = &SSTable[K, V]{path: path}
    }

    return &DB[K, V]{
        memtable:        memtable,
        maxMemtableSize: maxMemtableSize,
        memtableSize:    len(memtable.data),
        wal:             wal,
        walPath:         walPath,
        manifest:        manifest,
        manifestPath:    manifestPath,
        sstables:        sstables,
    }, nil
}

func (db *DB[K, V]) flushMemtable() error {
    sstablePath := fmt.Sprintf("data-%d.sstable", db.sstableCounter)
    sstable, err := writeSSTable(db.memtable, sstablePath)
    if err != nil {
        return err
    }

    db.sstables = append(db.sstables, sstable)
    db.sstableCounter++

    db.manifest.SSTablePaths = append(db.manifest.SSTablePaths, sstablePath)
    if err := WriteManifest(db.manifestPath, db.manifest); err != nil {
        return err
    }

    db.memtable = NewMemTable[K, V]()
    db.memtableSize = 0

    return nil
}

At this point, our DB has almost everything. It can write to memory (MemTable), persist to disk (sstable), and recover from crashes (WAL and manifest). You should include the update and delete feature for completeness – so let’s look at those next.

Update and Delete: Handling Mutability in an Immutable System

By this time, you should know that in an LSM storage system, data is never updated – rather, new data is written. For example, if you have a data pair ("a": "apple") and over time this has to change to the pair ("a": "apricot"), a new pair will be written to a different SSTable without any change to the existing pair. And yes, this leads to duplicates.

Also, interestingly, data isn't even deleted during write operations. The reason for that is, in a traditional sense, if you have to delete ("a":"apple"), you will have to find where it lives on disk and remove it. This makes writes slow. So instead, a clever mechanism is used: instead of removing the data directly, you can mark the key as deleted by writing a special TOMBSTONE value.

So, in the case of deleting (a : apple), you wouldn't remove the key from any SSTable. Instead, you’d write a new key-value pair such as ("a": "TOMBSTONE"). Here’s what this achieves:

The "TOMBSTONE" serves as a marker within the SSTable, telling the system that the key "a" has been logically deleted, even though it still physically exists in older SSTables.
During future reads, any value associated with "TOMBSTONE" will be treated as deleted, ensuring that the entry no longer shows up in query results.
This mechanism avoids the need for immediate deletions or expensive in-place updates, making write operations faster and simpler.

But this also raises the following questions:

How do you accurately read when there are duplicates? Meaning, how do users get ("a": "apricot") instead of ("a": "apple") because the former is latest and accurate?
How do you handle deletes to ensure deleted keys are not returned (and instead, a proper error message is returned)?
These stale and deleted data are garbage. How do you get rid of them to save on storage space?

As long as data is in MemTable (in-memory map), the duplicates are easy to handle: new values will just replace the old values.

But it gets tricky when data is in multiple SSTables. There is a very simple solution to this problem, and that is to just read the newer SSTable before older ones. That way, you will always read the latest value for a given key and exit early.

The following code in the read path ensures reading from newer SSTables before moving to older ones (note the loop starts from len(db.sstables) - 1):

func (db *DB[K, V]) Get(key K) (V, error) {
    // Check memtable first
    if val, ok := db.memtable.Get(key); ok {
        if any(val).(string) == TOMBSTONE {
            var zero V
            return zero, ErrNotFound
        }
        return val, nil
    }

    // Then check sstables from newest to oldest
    for i := len(db.sstables) - 1; i >= 0; i-- {
        sstable := db.sstables[i]
        val, err := sstable.Get(key)

        if err == nil {
            return val, nil
        }

        var zero V
        if err == ErrDeleted {
            return zero, ErrNotFound
        }
        if err == ErrNotFound {
            continue
        }
        return zero, err
    }

    var zero V
    return zero, ErrNotFound
}

And for delete, you could just add a new value "TOMBSTONE":

func (db *DB[K, V]) Delete(key K) error {
    return db.Put(key, any(TOMBSTONE).(V))
}

Note: This implementation assumes V is a string type. In a production system, you would need a more robust way to handle tombstones that works with any value type.

Handling deleted keys becomes simple now. You can check for the value (in MemTable and SSTable) and return an error if the value is "TOMBSTONE":

// db.go
func (db *DB[K, V]) Get(key K) (V, error) {
    if val, ok := db.memtable.Get(key); ok {
        if any(val).(string) == TOMBSTONE { //got TOMBSTONE, return zero
            var zero V
            return zero, ErrNotFound
        }
        return val, nil
    }
    // ... rest of function
}

// sstable.go
func (s *SSTable[K, V]) Get(key K) (V, error) {
    // ... earlier code

    keyInDB := any(pair.Key).(string)
    if keyInDB == any(key).(string) {
        if any(pair.Value).(string) == TOMBSTONE {
            var zero V
            return zero, ErrDeleted
        }
        return pair.Value, nil
    }

    // ... rest of function
}

Compaction: Cleaning Up Stale and Deleted Data

We have handled all the scenarios so far except for one. It’s not a concern of serving read/write traffic but something that’s important for the health of the storage system.

Over time, the system has developed a lot of garbage (stale, deleted data) and needs a garbage collection mechanism. Compaction is a background maintenance process that cleans up and reorganizes data in an LSM storage system.

As the system grows, multiple SSTables have been created. This leads to reads needing multiple file operations to get values. By compacting (or merging) multiple SSTables into a single one, you avoid disk operation overhead. Along the way, you should also permanently delete data that has been TOMBSTONED.

Note: Compaction is the only time data is permanently deleted from an LSM storage system.

To grasp the concept of compaction, we are going to implement something called Full Compaction where you will merge all the existing SSTables into one larger SSTable. In real-world database implementations, the strategy is more complex, there are multi level compaction involved.

Compaction Algorithm

We’re going to implement K-way merge to perform compaction. It’s a general algorithm that takes K sorted lists and merges them into a single, combined sorted list. In this case, the K sorted lists are the SSTables, and you are going to merge all of them into a single SSTable.

Our SSTables are already sorted, so the idea of merging them involves:

Taking the smallest (first) keys from each SSTable
Finding the smallest among those keys
Storing the found smallest key into new SSTable file
Fetching next key from the SSTable the smallest key belongs to
Repeating this process for all SSTables

Here’s a simple example with numbers:

Assume we have 3 sorted lists:
List A : [4, 8, 12]
List B : [3, 9]
List C : [7, 10, 11]

In the first iteration, we will take (4, 3, 7) because those are the smallest keys for individual lists. 
We find the smallest among those, which is 3, and store 3 in the result list.

In the second iteration, we will take (4, 9, 7). Note that 3 has already been accounted for. 
We pick 4 and store it to the result list.

Repeating this until all lists are empty, we get:
Result List : [3, 4, 7, 8, 9, 10, 11, 12]

The core part of this algorithm is to find the smallest key among the smallest keys from the individual SSTables. Fortunately, we have a data structure called Min-Heap that does this for us. So, you’re going to take the smallest key from each SSTable and put them all onto a Min Heap for it to return the smallest among those. We’re going to leverage go’s container/heap package to get the Min-Heap data structure and corresponding algorithm to find minimum value and put it at the top of heap.

Min Heap needs you to provide a function for it to determine what is the smaller key between two keys, as it uses that logic to determine global minimum. The following function is implemented for that:

func (h MinHeap[K, V]) Less(i, j int) bool {
    // again for simple comparison assume string key
    keyI := any(h[i].Pair.Key).(string)
    keyJ := any(h[j].Pair.Key).(string)
    if keyI != keyJ {
        return keyI < keyJ
    }
    // this is needed for the case when you have duplicate keys,
    // you will want to pick the one that is in newer sstable because that is latest
    return h[i].SSTableIndex > h[j].SSTableIndex
}

One important aspect about the above shown Less function is how it handles ties. So if we have two pairs with same key, which is lesser? Let’s assume two pairs as (a: apple) and (a: apricot), where (a: apple) is the older value (written to an older SSTable), which pair should the Less function return as the lesser value?

The answer is the one which is in the newer SSTable (see h[i].SSTableIndex > h[j].SSTableIndex). It ensures that the SSTable with higher index (that is, latest) becomes the lesser value, so (a: apricot) wins. It’s is important to always get the newer value of a given key.

The code for compaction looks something like the following. Note that we’re discarding deleted values (TOMBSTONE) and the older values.

// put this in a new file compaction.go
func MergeSSTables[K comparable, V any](sstables []*SSTable[K, V], newPath string) (*SSTable[K, V], error) {
    newFile, err := os.Create(newPath) // create a new sstable file
    if err != nil {
        return nil, err
    }
    defer newFile.Close() // prevent memory leak by ensuring file is closed

    newEncoder := gob.NewEncoder(newFile) // initialize encoder for new SSTable file


    files := make([]*os.File, len(sstables)) // open all the sstables
    decoders := make([]*gob.Decoder, len(sstables)) // initialize one decoder per ssltable file
    for i, sstable := range sstables {
        files[i], err = os.Open(sstable.path)
        if err != nil {
            return nil, err
        }
        defer files[i].Close() // prevent memory leak by ensuring file is closed
        decoders[i] = gob.NewDecoder(files[i])
    }

    // read first pair from each sstable and store in a pair array
    pairs := make([]Pair[K, V], len(decoders))
    emptySSTables := make([]bool, len(decoders)) // track empty sstables
    for i, decoder := range decoders {
        if err := decoder.Decode(&pairs[i]); err != nil {
            if err == io.EOF {
                emptySSTables[i] = true
                continue
            }
            return nil, err
        }
    }

    // push those pairs onto heap
    h := &MinHeap[K, V]{}
    for i, pair := range pairs {
        if !emptySSTables[i] {
            heap.Push(h, &HeapItem[K, V]{Pair: pair, SSTableIndex: i})
        }
    }

    // init the min-heap calculation algorithm from container/heap package
    heap.Init(h)

    var lastKey K
    firstKey := true

    // pop the min item from heap and store it into new sstable
    for h.Len() > 0 {
        item := heap.Pop(h).(*HeapItem[K, V])

        // If this key is a duplicate of the last one we saw, skip it
        if !firstKey && item.Pair.Key == lastKey {
            // We only care about the version from the newest SSTable,
            // which we have already processed
        } else {
            if any(item.Pair.Value).(string) != TOMBSTONE {
                // discard deleted
                if err := newEncoder.Encode(item.Pair); err != nil {
                    return nil, err
                }
            }
        }

        lastKey = item.Pair.Key
        firstKey = false

        // Push the next item from the same SSTable into the heap
        var nextPair Pair[K, V]
        if err := decoders[item.SSTableIndex].Decode(&nextPair); err == nil {
            heap.Push(h, &HeapItem[K, V]{Pair: nextPair, SSTableIndex: item.SSTableIndex})
        } else if err != io.EOF {
            return nil, err
        }
    }

    return &SSTable[K, V]{path: newPath}, nil
}

All the compaction magic has been packed in one function, MergeSSTables. The function has the following logical steps (and you can check the inline comments in the code to follow along):

We create a new destination SSTable file and initialize corresponding gob.Encoder
We open all the existing SSTable files, and store their references to files array. Also, we initialize one gob.Decoder per exiting SSTable file. To prevent memory leak, a defer statement ensures that each file will be closed once the function completes its work.
Each decoder reads the first key-value pair from its corresponding SSTable and stores it in the pairs array.
SSTables that are already exhausted (for example, are empty or have hit the end of the file) are marked as such in the emptySSTables slice, and we skip pushing them onto the heap.
We push each pair from the pairs array to Min-Heap and then initialize the Min-Heap calculation algorithm. This algorithm is present in Go’s container/heap package.
Each time the smallest key-value pair is popped from the min-heap, it’s compared with the previously processed key (lastKey). Duplicate keys (those whose values are already written) are skipped.
Values marked with a "TOMBSTONE" (logically deleted entries) are ignored and not written to the new SSTable, effectively cleaning up deleted data.
To continue the merge, the next key-value pair from the same SSTable (as the one we just processed) is read and pushed onto the heap, unless the end of the SSTable (io.EOF) has been reached.

To integrate this with the DB, you could use a compaction threshold and trigger compaction as part of the flush when this threshold is reached:

type DB[K comparable, V any] struct {
    memtable            *MemTable[K, V]
    maxMemtableSize     int
    memtableSize        int
    sstables            []*SSTable[K, V]
    sstableCounter      int
    wal                 *WAL[K, V]
    walPath             string
    manifest            *Manifest
    manifestPath        string
    compactionThreshold int
}

func NewDB[K comparable, V any](maxMemtableSize int, compactionThreshold int) (*DB[K, V], error) {
    walPath := "db.wal"
    memtable, err := ReplayWAL[K, V](walPath)
    if err != nil {
        return nil, err
    }

    wal, err := NewWAL[K, V](walPath)
    if err != nil {
        return nil, err
    }

    manifestPath := "MANIFEST"
    manifest, err := ReadManifest(manifestPath)
    if err != nil {
        return nil, err
    }

    sstables := make([]*SSTable[K, V], len(manifest.SSTablePaths))
    for i, path := range manifest.SSTablePaths {
        sstables[i] = &SSTable[K, V]{path: path}
    }

    return &DB[K, V]{
        wal:                 wal,
        walPath:             walPath,
        memtable:            memtable,
        memtableSize:        len(memtable.data),
        maxMemtableSize:     maxMemtableSize,
        manifestPath:        manifestPath,
        manifest:            manifest,
        sstables:            sstables,
        compactionThreshold: compactionThreshold,
    }, nil
}

// a new compact function
func (db *DB[K, V]) Compact() error {
    compactedSSTablePath := fmt.Sprintf("data-compacted-%d.sstable", db.sstableCounter)
    compactedSSTable, err := MergeSSTables(db.sstables, compactedSSTablePath)
    if err != nil {
        return err
    }
    // write new SSTable to MANIFEST file
    db.manifest.SSTablePaths = []string{compactedSSTablePath}
    if err := WriteManifest(db.manifestPath, db.manifest); err != nil {
        return err
    }
    //note delete only after writing manifest
    for _, sstable := range db.sstables {
        if err := os.Remove(sstable.path); err != nil {
            log.Printf("Failed to remove old sstable %s: %v", sstable.path, err)
        }
    }

    db.sstables = []*SSTable[K, V]{compactedSSTable}
    db.sstableCounter++

    return nil
}

func (db *DB[K, V]) flushMemtable() error {
    sstablePath := fmt.Sprintf("data-%d.sstable", db.sstableCounter)
    sstable, err := writeSSTable(db.memtable, sstablePath)
    if err != nil {
        return err
    }

    db.sstables = append(db.sstables, sstable)
    db.sstableCounter++

    db.manifest.SSTablePaths = append(db.manifest.SSTablePaths, sstablePath)
    if err := WriteManifest(db.manifestPath, db.manifest); err != nil {
        return err
    }

    db.memtable = NewMemTable[K, V]()
    db.memtableSize = 0

    // trigger compaction
    if len(db.sstables) >= db.compactionThreshold {
        if err := db.Compact(); err != nil {
            log.Printf("Compaction failed: %v", err)
            return err
        }
    }

    return nil
}

Notice the Compact() function in the integrated DB code? This is where we invoke previously defined the MergeSSTables function to trigger the compaction process. After invoking MergeSSTables, we write a new SSTable to the MANIFEST file and then delete the older SSTables.

Previously, in the Manifest File: Tracking the State of the Database, I spoke about atomic renaming os.Rename(tmpPath, path). Let’s talk about why the atomic renaming of MANIFEST matters for compaction.

During compaction, we're making a major change to the database state: replacing multiple SSTables with a single compacted one. The MANIFEST update is critical here because it's the source of truth for which SSTables exist.

Let’s think about what could go wrong without atomic renaming:

You start writing the new MANIFEST (which points to the compacted SSTable)
System crashes mid-write
MANIFEST is corrupted and unreadable
On restart, the database has no idea which SSTables exist
All data is effectively lost

With atomic renaming:

We write the new MANIFEST to MANIFEST.tmp
We fully close and sync it to disk
We atomically rename MANIFEST.tmp to MANIFEST using os.Rename(tmpPath, path)
If crash happens before step 3: old MANIFEST is intact, we retry compaction
If crash happens during step 3: atomic operation either completes or doesn't – no corruption
If crash happens after step 3: new MANIFEST is in place, we're good

This is also why we delete the old SSTables only after successfully updating the MANIFEST. If we deleted them before updating MANIFEST and then crashed, the MANIFEST would still point to files that no longer exist.

Complete Picture:

Conclusion

Congratulations! You've built a working LSM tree storage engine from scratch. By following the problem-driven approach – discovering issues and implementing solutions as they arose – you've experienced how engineers think about building robust storage systems. I hope this is better than just memorizing the concepts.

Key Takeaways

Append-only writes make LSM-trees fast for write-heavy workloads
Immutability eliminates complex concurrency issues
Trade-off is that LSM-tree favor writes over reads (opposite of B-trees)
Durability requires multiple mechanism working together (WAL, MANIFEST, atomic operations)
Background maintenance (compaction) is essential for long-term health and cost.

Important note: This is a learning implementation. This means that I intentionally simplified the code, so it’s not production-ready. Key limitations include:

No concurrency control (missing mutexes/locks)
No bloom filters for efficient lookups
Simplified compaction strategy
Type safety issues with generic tombstones
Missing robust error recovery

Complete Code:

Like I’ve mentioned before, I've omitted boilerplate code and helper functions for brevity. The complete, runnable implementation is available at this GitHub repo.

To learn more about production LSM implementations, study RocksDB, LevelDB, or read the original LSM tree paper by O'Neil et al: https://www.cs.umb.edu/~poneil/lsmtree.pdf

How to Get Started with PocketBase: Build a Lightweight Backend in Minutes

Manish Shivanandhan — Fri, 14 Nov 2025 18:11:55 +0000

If you’re a developer looking for a simple, fast, and self-hosted backend, PocketBase might be exactly what you need.

It’s an open-source backend written in Go that lets you set up a complete backend with database, authentication, file storage, and real-time updates, all in a single executable file.

In this guide, we’ll explore what makes PocketBase special, how to set it up, and how you can deploy it to the cloud.

What We’ll Cover:

What is PocketBase?
Why Developers Love PocketBase
How to Install Pocketbase
Extending PocketBase with JavaScript
Using SDKs to Interact with PocketBase
Self-Hosting PocketBase using Sevalla
Security and Open Source Nature
When to Use PocketBase
Conclusion

What is PocketBase?

PocketBase is an all-in-one backend that provides everything you need to power a modern web or mobile app, eliminating the need for large infrastructure.

It includes an embedded SQLite database, real-time subscriptions, file and user management, a clean admin dashboard, and a REST-style API.

Since it runs from a single file, you can deploy it almost anywhere, from a VPS to your local machine or even a Raspberry Pi.

It’s designed for developers who want control and simplicity at the same time. You don’t need to manage separate servers for authentication, storage, and API endpoints. PocketBase handles all of this out of the box. You can use it as a standalone backend or embed it in your Go application to create a custom solution.

Why Developers Love PocketBase

PocketBase focuses on speed and simplicity. You don’t need to install multiple packages or services.

Once downloaded, you can start it with a single command. Then it’ll launch a web-based admin dashboard.

The database is built using SQLite, which means data is stored locally by default, but you can use extensions to connect it with your existing workflows or cloud storage.

Another major advantage is its real-time capabilities. Every change in the database can be broadcast instantly to connected clients through WebSocket subscriptions. This makes it perfect for building apps like chat systems, dashboards, and collaboration tools that require instant updates.

How to Install PocketBase

Getting PocketBase running takes less than a minute. You can download a prebuilt executable from the official releases page.

It supports all major platforms, including Windows, macOS, and Linux.

Once downloaded, extract the archive and navigate to the folder in your terminal. Run the following command:

./pocketbase serve

This command starts a local server and launches the admin dashboard at http://127.0.0.1:8090/_/. From there, you can create collections, add users, upload files, and manage data. There’s no setup wizard or dependency installation – everything is self-contained inside that one binary.

If you’re a Go developer, you can also install PocketBase as a Go module and use it directly in your project to build custom logic or extend the existing API.

Using PocketBase as a Go Framework

PocketBase can act as a Go framework, letting you build your own backend logic while keeping everything in one file. Here’s a simple example that shows how you can extend it with a custom route.

package main

import (
    "log"
    "github.com/pocketbase/pocketbase"
    "github.com/pocketbase/pocketbase/core"
)
func main() {
    app := pocketbase.New()
    app.OnServe().BindFunc(func(se *core.ServeEvent) error {
        se.Router.GET("/hello", func(re *core.RequestEvent) error {
            return re.String(200, "Hello world!")
        })
        return se.Next()
    })
    if err := app.Start(); err != nil {
        log.Fatal(err)
    }
}

Once the file is ready, run these commands:

go mod init myapp && go mod tidy
go run main.go serve

You’ll now have a working backend with both the PocketBase dashboard and your custom /hello endpoint available at the same time.

This makes PocketBase flexible, you can use it as a ready-to-run backend or as part of a more complex Go project.

Extending PocketBase with JavaScript

PocketBase includes a built-in JavaScript engine that lets you extend its behavior without modifying or recompiling the Go code. This makes it easy to add custom logic, validation, automation, and event-driven workflows directly inside your backend.

You can create JavaScript files inside the pb_hooks folder, and PocketBase will automatically load and run them. These scripts can listen to events like record creation, updates, authentication, and more.

Here’s a simple example that sends a welcome email whenever a new user signs up.

Create a file named pb_hooks/user_email.js inside your PocketBase directory:

/// 

onRecordAfterCreate("users", async (e) => {
    const user = e.record;

    console.log("New user registered:", user.email);

    // Example: send a welcome email using a third-party API
    const emailResponse = await $http.send({
        url: "https://api.example.com/send-email",
        method: "POST",
        body: JSON.stringify({
            to: user.email,
            subject: "Welcome to our app!",
            message: `Hi ${user.username}, thanks for signing up!`
        }),
        headers: {
            "Content-Type": "application/json"
        }
    });

    console.log("Email status:", emailResponse.status);
});

This script runs automatically whenever a new record is created in the users collection. It picks up the user’s email, logs it, and uses PocketBase’s built-in HTTP client ($http) to call an external email service.

You can use the same pattern to validate data before saving, trigger webhooks, block actions, or update related records. Since everything runs inside PocketBase, you don’t need extra servers or functions to automate backend logic.

This makes it friendly for teams who may not be comfortable with Go but still want to add dynamic logic to their backend. You can find more details in the official documentation under “Extend with JavaScript.”

Using SDKs to Interact with PocketBase

To make it easier to communicate with your backend, PocketBase provides official SDKs for JavaScript and Dart.

The JavaScript SDK works well for browser-based or Node.js projects, while the Dart SDK is ideal for mobile apps built with Flutter. Both SDKs provide an easy way to connect, authenticate users, and perform CRUD operations without manually writing HTTP requests.

For example, in JavaScript you can connect and fetch data like this:

import PocketBase from 'pocketbase'

const pb = new PocketBase('http://127.0.0.1:8090')
const records = await pb.collection('posts').getList(1, 20)
console.log(records)

This simplicity allows you to focus on building your frontend while PocketBase handles authentication, database operations, and real-time updates.

Self-Hosting PocketBase using Sevalla

When you are ready to move beyond testing, PocketBase gives you two options. You can self-host it using your own infrastructure or use their managed cloud version at Pocketbase.io.

Self-hosting gives you full control and is usually preferred by technical teams who want to keep sensitive data in-house.

You can choose any cloud provider, like AWS, DigitalOcean, or others to set up Pocketbase. But I will be using Sevalla.

Sevalla is a PaaS provider designed for developers and dev teams shipping features and updates constantly in the most efficient way. It offers application hosting, database, object storage, and static site hosting for your projects.

I am using Sevalla for two reasons:

Every platform will charge you for creating a cloud resource. Sevalla comes with a $50 credit for us to use, so we won’t incur any costs for this example.
Sevalla has a template for PocketBase, so it simplifies the manual installation and setup for each resource you will need for installation.

Click on the “PocketBase” template. You will see the resources needed to provision the application. Click on “Deploy Template”:

You can see the resource being provisioned. Once the deployment is complete, go to the PocketBase application and click on “Visit app”

You will see a 404 message. Add _ to the url and you will see the login dashboard.

To login for the first time, you will need a superuser login. To create that, go back to the application and click “Logs”. You will see a url starting with https://0.0.0.0.

Replace the 0.0.0.0 with your new cloud URL and copy paste the full path into the browser. You will see the option to create a super user for your PocketBase deployment.

Once you have created the super user, you can again go to /_ and login using the username and password. You should now see the PocketBase dashboard.

You now have a production-grade Pocketbase server running on the cloud. You can use this to set up tables for your database and use the JavaScript or other SDKs to interact with Pocketbase.

Security and Open Source Nature

PocketBase is open source and licensed under MIT, which means you’re free to use it in personal or commercial projects. If you find a bug or security issue, you can report it to the maintainers, and they’ll address it promptly.

The project’s transparent development and active community make it a solid choice for startups, indie developers, and hobbyists who prefer to own their infrastructure.

Because it’s still in active development, backward compatibility isn’t guaranteed before version 1.0. But it’s already stable enough for small to medium-scale applications.

When to Use PocketBase

PocketBase is perfect for projects that need a simple backend with low maintenance. It’s ideal for prototypes, small SaaS products, indie apps, internal tools, and educational projects.

Instead of setting up a complex stack with PostgreSQL, Node.js, and nginx, you can get your backend running instantly and focus on your product.

Suppose your project later grows into something bigger. In that case, you can migrate to a more complex setup or continue using PocketBase as a lightweight service for specific features like authentication or real-time data sync.

Conclusion

PocketBase brings back the joy of fast development without complicated setups. With just one executable, you get a backend that supports authentication, real-time updates, file uploads, and an admin dashboard. It’s open source, fast, and customizable, making it a great choice for developers who want to move quickly without giving up control.

Whether you’re building a personal app, a startup MVP, or an internal dashboard, PocketBase gives you the power to set up a full backend in minutes. You can start small, extend it as needed, and deploy it anywhere – all while keeping your workflow simple and efficient.

Hope you enjoyed this article. Find me on Linkedin or visit my website.

How to Use Closures in Go

Gabor Koos — Mon, 27 Oct 2025 20:35:29 +0000

If you've written code in JavaScript, Python, or Rust, you've probably heard the word closure before. The concept has subtle differences in each language, but the core idea is the same: a closure is a function that captures variables from its surrounding scope. This allows the function to "remember" the environment in which it was created, even when it's executed outside that environment, which has powerful implications for how we write and structure our code.

In this article, we'll explore how closures work in Go, a statically typed language known for its simplicity and efficiency. We'll look at how to create closures, how they capture variables, and some practical use cases.

What We'll Cover

Prerequisites
What Closures Really Are in Go
- How Go Makes This Work
- Multiple Independent Closures
The Classic Loop Trap
How to Create Closures in Go
Closures and Concurrency
- Independent State Across Goroutines
- Capturing Shared Variables Safely
Practical Patterns with Closures
How Closures Affect Memory and Performance
- Variables May Live Longer Than Expected
- Many Closures Can Add Overhead
How to Test and Debug Closures
Best Practices and Takeaways for Using Closures in Go
Conclusion

Prerequisites

To follow along with this article, you should have a basic understanding of Go programming, including functions and variable scope. If you're new to Go, consider checking out the official Go tour to get up to speed.

What Closures Really Are in Go

At its simplest, a closure in Go is a function that references variables defined outside of it. That may sound abstract, so let's start with an example you can run right away:

package main

import "fmt"

func counter() func() int {
    n := 0
    return func() int {
        n++
        return n
    }
}

func main() {
    next := counter()
    fmt.Println(next()) // 1
    fmt.Println(next()) // 2
    fmt.Println(next()) // 3
}

When you call counter(), it returns another function, but that function keeps access to the variable n that lived inside counter.

Even though counter() has already finished running, n hasn't disappeared. Each time you call next(), it updates the same n that was created during the original counter() call.

This is the defining property of a closure:

A closure "closes over" its environment, keeping the variables it needs alive for as long as the closure itself exists.

How Go Makes This Work

Normally, local variables in Go live on the stack, which is cleared when a function returns.

But if a nested function needs to keep using one of those variables, Go's compiler performs what's called escape analysis: it sees that the variable will outlive the function call, so it moves that variable to the heap, where it can stay alive as long as something references it - in this case, the closure.

You can actually ask the compiler to show you this process:

go build -gcflags="-m" main.go

You might see output like:

./main.go:6:6: moved to heap: n

That tells you the variable n "escaped" the stack so the closure could use it safely later.

Multiple Independent Closures

Each call to a function that returns a closure creates a new, independent environment:

a := counter()
b := counter()
fmt.Println(a()) // 1
fmt.Println(a()) // 2
fmt.Println(b()) // 1

Here, a and b are two separate closures, each with its own n. Calling a() increments its own n, while calling b() starts from its own separate n.

The Classic Loop Trap

One of the most common surprises for Go developers comes when closures are used inside a loop. Even experienced programmers often fall into this trap.

Consider this example:

package main

import "fmt"

func main() {
    funcs := make([]func(), 0)
    for i := 0; i < 3; i++ {
        funcs = append(funcs, func() {
            fmt.Println(i)
        })
    }
    for _, f := range funcs {
        f()
    }
}

You might expect this to print 0, 1, and 2, but it actually prints

3
3
3

Why Does this Happen?

Inside the loop, each function literal captures the variable i itself, not its value at that moment.

The loop reuses the same i variable for all iterations. By the time the loop finishes, i equals 3, and all the closures see this same i when they run later.

How to Fix It

There are two common idiomatic fixes:

Shadow the loop variable:

for i := 0; i < 3; i++ {
    i := i // new variable shadows the loop variable
    funcs = append(funcs, func() {
        fmt.Println(i)
    })
}

Pass the variable as a parameter to an inner function:

for i := 0; i < 3; i++ {
    funcs = append(funcs, func(x int) func() {
        return func() { fmt.Println(x) }
    }(i))
}

Both approaches create a new variable for each iteration, so each closure captures its own independent value.

How to Create Closures in Go

There are a few different ways to create closures in Go. Let's explore some common patterns.

Returning Closures From Functions

The most common pattern is having a function return a closure that keeps its own state:

func makeCounter() func() int {
    n := 0
    return func() int {
        n++
        return n
    }
}

c1 := makeCounter()
fmt.Println(c1()) // 1
fmt.Println(c1()) // 2

Each call to makeCounter creates a new closure with its own n, as we saw earlier.

Named Inner Functions

You can also give a name to a function literal for readability or debugging:

func makeCounter() func() int {
    n := 0
    next := func incr() int {
        n++
        return n
    }
    return next
}

This works the same way but gives the inner function a name (incr), which can be helpful in stack traces. Other than that, it behaves just like an anonymous function.

Inline Closures in Loops or Goroutines

Closures are often defined inline, especially for loops or goroutines:

for i := 0; i < 3; i++ {
    go func(x int) {
        fmt.Println(x)
    }(i)
}

Here, we pass i as a parameter to the closure, ensuring each goroutine gets its own copy of the value, avoiding the loop variable trap.

Closures With Parameters

Closures can accept their own arguments:

func adder(base int) func(int) int {
    return func(x int) int {
        return base + x
    }
}

add5 := adder(5)
fmt.Println(add5(10)) // 15

Here, adder returns a closure that adds a fixed base value to whatever argument it receives.

Capturing Multiple Variables

Closures can capture multiple outer variables:

func multiplier(factor int) func(int) int {
    offset := 2
    return func(x int) int {
        return x*factor + offset
    }
}

m := multiplier(3)
fmt.Println(m(4)) // 14

In this example, the closure captures both factor and offset from its surrounding scope - factor is a parameter, while offset is a local variable.

Closures in Structs

Closures can also be stored in structs, just like any other function value. This is a useful pattern when you want objects with dynamic or stateful behavior.

type Counter struct {
    Next func() int
}

func NewCounter() Counter {
    n := 0
    return Counter{
        Next: func() int {
            n++
            return n
        },
    }
}

func main() {
    c := NewCounter()
    fmt.Println(c.Next()) // 1
    fmt.Println(c.Next()) // 2
}

Here, the Next field holds a closure that captures the variable n. Each instance of Counter has its own independent state, without needing a separate type or mutex.

This pattern shows how closures can act as lightweight objects: bundling behavior and state together.

Note on Method Receivers

Closures in Go don't implicitly capture the method receiver like some languages do. If you want a closure to use the receiver inside a method, you typically assign it to a local variable:

type Counter struct {
    n int
}

func (c *Counter) MakeIncrementer() func() int {
    r := c // capture receiver explicitly
    return func() int {
        r.n++
        return r.n
    }
}

This ensures the closure references the intended receiver rather than introducing unexpected behavior.

Unlike JavaScript or Python, Go closures capture lexical variables, not the implicit this or self.

Key Takeaways

Closures can be returned from functions, named, inlined, or even stored in structs.
They capture outer variables, not copies of their values.
Used this way, closures can replace small types or interfaces for lightweight encapsulation.

Closures and Concurrency

Closures are powerful in Go, but when you combine them with concurrency, their captured variables can act in unexpected ways if you're not careful.

Independent State Across Goroutines

Each closure keeps its own captured variables alive, even when used in concurrent goroutines:

func makeWorker(start int) func() int {
    counter := start
    return func() int {
        counter++
        return counter
    }
}

func main() {
    worker1 := makeWorker(0)
    worker2 := makeWorker(100)

    go func() { fmt.Println(worker1()) }() // prints 1
    go func() { fmt.Println(worker2()) }() // prints 101
}

Here, worker1 and worker2 have independent counter variables, so they don't interfere with each other. Each closure maintains independent state, even in separate goroutines.

Capturing Shared Variables Safely

When multiple closures share a variable, you must coordinate access. For example:

counter := 0
ch := make(chan int)

for i := 0; i < 3; i++ {
    go func() {
        // increments a shared variable
        ch <- 1
    }()
}

// aggregate safely
for i := 0; i < 3; i++ {
    counter += <-ch
}
fmt.Println(counter) // 3

The closure captures the outer variable ch (a channel), which is safe because channels serialize access. Using a buffered channel here wouldn't change the behavior of the closure: it still captures its own n and sends the values to the channel independently.

Closures themselves don't synchronize shared state, you still need channels or mutexes.

Practical Patterns with Closures

Closures in Go aren't just a language curiosity, they're a powerful tool for writing stateful, reusable, and flexible code. Here are a few practical patterns that go beyond the basics.

Memoization / Caching

Closures can capture an internal map or cache to store results of expensive computations:

func memoize(f func(int) int) func(int) int {
    cache := map[int]int{}
    return func(x int) int {
        if val, ok := cache[x]; ok {
            return val
        }
        result := f(x)
        cache[x] = result
        return result
    }
}

func main() {
    fib := memoize(func(n int) int {
        if n <= 1 {
            return n
        }
        return fib(n-1) + fib(n-2)
    })
    fmt.Println(fib(10)) // 55
}

Here, the memoize function returns a closure that caches results of the Fibonacci function, avoiding redundant calculations.

Event Handlers / Callbacks

Closures are perfect for defining event handlers or callbacks that need to maintain state:

type Button struct {
    onClick func()
}

func (b *Button) Click() {
    if b.onClick != nil {
        b.onClick()
    }
}

func main() {
    count := 0
    button := Button{
        onClick: func() {
            count++
            fmt.Println("Button clicked", count, "times")
        },
    }

    button.Click() // Button clicked 1 times
    button.Click() // Button clicked 2 times
}

In this example, the closure captures the count variable, allowing the button to keep track of how many times it has been clicked.

Encapsulated Pipelines / Producers

Closures can wrap stateful logic for channels and pipelines:

func producer(start int) func(chan int) {
    n := start
    return func(ch chan int) {
        for i := 0; i < 3; i++ {
            ch <- n
            n++
        }
    }
}

func main() {
    ch := make(chan int, 3)
    go producer(5)(ch)
    for i := 0; i < 3; i++ {
        fmt.Println(<-ch) // 5, 6, 7
    }
}

Here, the producer function returns a closure that sends a sequence of numbers to a channel, maintaining its own state with n.

Deferred Execution with Captured State

Using a closure with defer lets you capture variables at the moment the defer statement is executed, which is especially useful in loops or resource cleanup:

func main() {
    for i := 0; i < 3; i++ {
        defer func(x int) {
            fmt.Println(x)
        }(i) // capture current i
    }
}

Output:

2
1
0

Here, each deferred closure captures the value of i at the time of the defer statement, so they print in reverse order when the function exits.

How to Implement Interfaces Dynamically

Closures can also be used to implement interfaces without defining a full struct type. For example, a simple function can satisfy a single-method interface:

type Greeter interface {
    Greet() string
}

func MakeGreeter(name string) Greeter {
    return struct{ Greeter }{
        Greeter: func() string { return "Hello, " + name },
    }
}

func main() {
    g := MakeGreeter("Alice")
    fmt.Println(g.Greet()) // Hello, Alice
}

Here, the closure captures name, allowing the returned object to implement the Greet method dynamically.

Key Takeaways

Closures allow memoization and caching without extra structs.
Storing closures in structs provides customizable behavior for objects.
Closures can encapsulate stateful concurrent pipelines, keeping logic localized and safe.
Closures with defer capture variables at the time of deferment, useful for cleanup or logging.
They enable dynamic interface implementations without boilerplate types.

How Closures Affect Memory and Performance

Closures are powerful, but capturing variables from outer scopes has memory and performance implications.

Variables May Live Longer Than Expected

Because closures keep references to captured variables (and move them to the heap if necessary, as we saw earlier), these variables live as long as the closure itself, which can increase memory usage:

func main() {
    bigData := make([]byte, 10_000_000) // 10MB
    f := func() int { return len(bigData) }
    _ = f
}

In this example, bigData remains in memory as long as the closure f exists, even if bigData is no longer needed elsewhere.

Many Closures Can Add Overhead

Each closure carries a small environment for its captured variables. Creating thousands of closures is usually fine, but in high-performance or memory-sensitive code, this can add measurable overhead.

Captured variables may be heap-allocated.
Each closure has a small hidden struct for its environment.

Alternatives include structs or plain functions when you need maximum efficiency.

How to Test and Debug Closures

Closures can sometimes behave in unexpected ways when capturing variables or working with concurrency. Here are some tips to test and debug them effectively.

Isolate the Closure

Test the closure independently of its outer function to verify its behavior:

func TestCounter(t *testing.T) {
    counter := makeCounter()
    if counter() != 1 {
        t.Error("expected 1")
    }
    if counter() != 2 {
        t.Error("expected 2")
    }
}

This ensures the closure maintains state correctly.

Check Captured Variables

Remember: closures capture variables by reference, not value. Be mindful of loop variables or shared state:

for i := 0; i < 3; i++ {
    i := i // shadow loop variable
    t.Run(fmt.Sprintf("i=%d", i), func(t *testing.T) {
        if i != i { // simplified check
            t.Fail()
        }
    })
}

This helps avoid the loop trap in tests.

Use Logging or Debug Prints

Printing internal closure state is often the fastest way to debug subtle behavior:

adder := func(base int) func(int) int {
    return func(x int) int {
        fmt.Printf("base=%d, x=%d\n", base, x)
        return base + x
    }
}
result := adder(5)(10) // logs: base=5, x=10

Test Concurrency Carefully

When closures are used in goroutines, race conditions can creep in. Use the Go race detector:

go test -race ./...

This flags any shared variable access that isn’t properly synchronized.

Key Takeaways

Test closures independently to ensure captured state behaves as expected.
Be cautious with loop variables and shared state.
Use logging and the race detector to debug concurrency issues.

Best Practices and Takeaways for Using Closures in Go

Closures are a versatile feature in Go, but like any tool, they work best when used thoughtfully. Here are some practical guidelines:

Encapsulate state cleanly: Use closures to maintain private state without introducing extra structs or types. Counters, memoization caches, and small factories are common patterns.
Be careful in loops: Always capture loop variables correctly to avoid the classic loop trap. Shadowing the variable or passing it as a parameter to the closure are idiomatic solutions.
Handle concurrency explicitly: Closures can safely maintain independent state in goroutines, but they do not synchronize shared state automatically. When multiple closures share variables, coordinate access with channels or mutexes.
Mind memory usage: Captured variables may escape to the heap, so long-lived closures can retain more memory than expected. Avoid capturing large objects unless necessary.
Leverage closures in structs: Storing closures in struct fields allows objects to have dynamic or customizable behavior without extra boilerplate, making your code more flexible.

Conclusion

Closures in Go allow functions to carry state, encapsulate behavior, and interact safely with concurrency patterns, all while keeping your code clean and expressive. By understanding how closures capture variables, how they behave in loops and goroutines, and their memory implications, you can use them confidently to write more idiomatic and maintainable Go code.

How to Cache Golang API Responses for High Performance

Temitope Oyedele — Wed, 15 Oct 2025 10:27:00 +0000

Go makes it easy to build APIs that are fast out of the box. But as usage grows, speed at the language level is not enough. If every request keeps hitting the database, crunching the same data, or serializing the same JSON over and over, latency creeps up and throughput suffers. Caching is the tool that keeps performance high by storing work that has already been done so that future requests can reuse it instantly. Let’s look at four practical ways to cache APIs in Go, each explained with an analogy and backed by simple code you can adapt.

Response Caching with Local and Redis Storage
Database Query Result Caching
HTTP Caching with ETag and Cache-Control
Stale-While-Revalidate with Background Refresh
Wrapping Up

Response Caching with Local and Redis Storage

When the process of generating an API response becomes expensive, the fastest solution is to store the entire response. Think of a coffee shop during the morning rush. If every customer orders the same latte, the barista could grind beans and steam milk for each order, but the line would move slowly. A smarter move is to brew a pot once and pour from it repeatedly. To handle both speed and scale, the shop keeps a small pot at the counter for instant pours and a larger urn in the back for refills. In software terms, the counter pot is a local in-memory cache such as Ristretto or BigCache, and the urn is Redis, which allows multiple API servers to share the same cached responses.

In Go, this two-tier setup usually follows a cache-aside pattern: look in local memory first, fall back to Redis if needed, and only compute the result when both layers miss. Once computed, the value is saved in Redis for everyone and in memory for immediate reuse on the next call.

val, ok := local.Get(key)
if !ok {
    val, err = rdb.Get(ctx, key).Result()
    if err == redis.Nil {
        val = computeResponse() // expensive DB or logic
        _ = rdb.Set(ctx, key, val, 60*time.Second).Err()
    }
    local.Set(key, val, 1)
}
w.Header().Set("Content-Type", "application/json")
w.Write([]byte(val))

In the code above, the first attempt is to retrieve the response from the local cache, which returns instantly if the key or data exists. If not found, it queries Redis as the second layer. If Redis also returns nothing, the expensive computation runs and its result is stored in Redis with a sixty seconds expiration so other services can access it, then placed in the local cache for immediate reuse. After which, the response is written back to the client as JSON.

This gives you the best of both worlds: lightning-fast responses for repeat calls and a consistent cache across all your API servers.

Database Query Result Caching

Sometimes the API itself is simple but the real cost hides in the database. Imagine a newsroom waiting for election results. If every editor keeps calling the counting office for the same numbers, the phone lines may jam. Instead, one reporter calls once, writes the result on a board, and every editor copies from there. The board is the cache, and it saves both time and pressure on the office.

In Go, you can apply the same principle by caching query results. Rather than hitting the database for each identical request, you store the result in Redis with a key that represents the query intent. When the next request comes in, you pull from Redis, skip the database, and respond faster.

key := fmt.Sprintf("q:UserByID:%d", id)
if b, err := rdb.Get(ctx, key).Bytes(); err == nil {
    var u User
    _ = json.Unmarshal(b, &u)
    return u
}

u, _ := repo.GetUser(ctx, id) // real DB call
bb, _ := json.Marshal(u)
_ = rdb.Set(ctx, key, bb, 2*time.Minute).Err()
return u

Here, we construct a cache key that uniquely identifies the query using the user ID, then attempts to fetch the serialized result from Redis. If the key exists, it deserializes the bytes back into a User struct and returns immediately without touching the database. On a cache miss, it executes the actual database query through the repository, serializes the User object to JSON, stores it in Redis with a two-minute expiration, and returns the result.

This pattern dramatically reduces database load and response time for read-heavy APIs, but you must remember to clear or refresh entries when data changes, or set short time-to-live values to keep results reasonably fresh.

HTTP Caching with ETag and Cache-Control

Not all caching has to happen inside the server. The HTTP standard already provides tools that let clients or CDNs reuse responses. By setting headers like ETag and Cache-Control, you can tell the client whether the response has changed. If nothing is new, the client keeps its own copy and the server only sends a lightweight 304 response.

It is similar to a manager posting notices on an office board. Each sheet carries a small stamp. Employees compare the stamp against the one they already have. If it matches, they know their copy is still valid and skip taking a new one. Only when the stamp changes do they replace it.

In Go this is straightforward. Compute an ETag from the response body, compare it with what the client sends, and decide whether to return the full payload or just the 304.

etag := computeETag(responseBytes)
if match := r.Header.Get("If-None-Match"); match == etag {
    w.WriteHeader(http.StatusNotModified)
    return
}

w.Header().Set("ETag", etag)
w.Header().Set("Cache-Control", "public, max-age=60")
w.Write(responseBytes)

The code above generates an ETag, which is a fingerprint or hash of the response content, then checks if the client sent an If-None-Match header with a matching ETag from a previous request. If the ETags match, the content hasn't changed, so the server responds with a 304 Not Modified status and sends no body, saving bandwidth. When the ETags don't match or the client has no cached version, the server attaches the new ETag and a Cache-Control header that allows public caching for sixty seconds, then sends the full response.

This approach reduces bandwidth, lowers CPU usage, and pairs well with CDNs that can cache and serve responses directly.

Stale-While-Revalidate with Background Refresh

There are cases where serving slightly old data is acceptable if it keeps the API fast. Stock dashboards, analytics summaries, or feed endpoints often fit this model. Instead of making users wait for fresh data on every request, you can serve the cached value immediately and refresh it quietly in the background. This technique is called Stale-While-Revalidate.

Picture a stock ticker screen in a lobby. The numbers may be a few seconds behind, but they are still useful to anyone glancing at the board. Meanwhile, a background process fetches the latest figures and updates the ticker. The reader never stares at a blank screen and the system stays responsive even during spikes.

In Go, this can be built by storing not just the cached data but also timestamps that define when the data is fresh, when it can still be served as stale, and when it must be recomputed. The singleflight package helps ensure that only one goroutine does the refresh work, preventing a dogpile of updates.

entry := getEntry(key) // {data, freshUntil, staleUntil}
switch {
case time.Now().Before(entry.freshUntil):
    return entry.data
case time.Now().Before(entry.staleUntil):
    go refreshSingleflight(key) // background refresh
    return entry.data
default:
    return refreshSingleflight(key) // must refresh now
}

Here, the code retrieves a cache entry containing the data along with two timestamps marking the freshness and staleness boundaries. If the current time falls before the fresh threshold, the data is considered fully fresh and returned immediately. If time has passed the fresh threshold but remains within the stale window, the code returns the slightly outdated data instantly while launching a background goroutine to refresh it asynchronously, ensuring the next request gets updated information. Once time exceeds even the stale boundary, the data is too old to serve, so the code blocks and performs a synchronous refresh before returning.

This keeps latency low while still ensuring the cache updates regularly, a balance between freshness and performance.

Wrapping Up

Caching is not a single tactic but a set of strategies that fit different needs. Full response caching eliminates repeat work at the top level. Query result caching protects the database from repeated load. HTTP caching leverages the protocol to cut down data transfer. Stale-While-Revalidate strikes a compromise that favors speed without leaving data stale for too long.

In practice, these approaches are often layered. A Go API might use local memory and Redis for responses, apply query-level caching for hot tables, and set ETags so clients avoid unnecessary downloads. With the right mix, you can cut latency by orders of magnitude, handle far more traffic, and save both compute and database resources.

Learn How to Use Pointers in Go – With Example Code

Gabor Koos — Mon, 06 Oct 2025 15:04:41 +0000

Pointers are a fundamental but often dreaded concept in every programming language that supports them. Luckily for us, Go makes working with pointers straightforward and safe.

In this article, we will demystify pointers in Go. You'll learn:

What pointers are and how to use them
How to declare pointers and dereference them
Common pitfalls, like nil pointers and reference types
Pointer receivers in structs (a key reason pointers are so useful in Go)
A bonus look at weak pointers (Go 1.24+) for advanced memory management

By the end, you'll have a solid understanding of pointers and be confident using them in your own Go programs.

What We’ll Cover:

Prerequisites
What is a Pointer?
Declaring and Using Pointers
Why Use Pointers?
Common Pitfalls and Misunderstandings
- Nil Pointers
- Reference Types
Pointer Receivers
- Value Receivers vs Pointer Receivers
- Idiomatic Go
Exercises for the Reader
Bonus: Weak Pointers (Go 1.24+)
- What Are Weak Pointers?
- When to Use Weak Pointers?
Summary & Best Practices
Solutions to Exercises

Prerequisites

This article assumes you have a basic understanding of Go, including:

Variables and basic types (int, string, and so on)
Functions and function calls
Structs and methods

Familiarity with memory concepts (like copying vs referencing values) can be helpful, but it is not required. I’ll explain all examples in a beginner-friendly way.

What is a Pointer?

Understanding Memory

Memory in a computer is a large sequence of bytes, each with a unique address. Every variable in a program occupies one or more contiguous bytes in memory, depending on its type:

An int32 typically occupies 4 bytes.
An int64 typically occupies 8 bytes.
A bool usually occupies 1 byte.

Structs, arrays, and slices occupy the sum of their fields' sizes, plus potential padding for alignment (for quick access). Each variable has a unique memory address, which is where its data is stored.

For example, consider:

var a int32 = 100
var b bool = true

This will look something like this in memory:

a occupies 4 bytes and holds the value 100. b occupies 1 byte and holds true. A variable's address is simply the location in memory where its data starts (0×02022 for a, and 0x0207 for b in this example).

Stack vs Heap

In Go, variables can be allocated on the stack or the heap. The stack is a region of memory that stores local variables and function call information. It’s fast to allocate and deallocate, as it works in a last-in-first-out manner.

The heap is a larger pool of memory used for dynamic allocation. Variables allocated on the heap can outlive the function that created them, making them suitable for data that needs to be shared or modified across different parts of a program.

The Go runtime automatically manages memory allocation and garbage collection, so you don't need to worry about manually freeing memory like in some other languages.

The stack and heap are just implementation details. As a Go programmer, you typically don't need to worry about where a variable is allocated. The Go compiler and runtime handle this for you. You definitely don't have to worry about it in this article – just know they exist and that pointers can point to values in either location.

The Pointer

A pointer is simply a variable that stores the memory address of another variable. From the diagram above, you can see that an address is basically an integer value (which happens to represent a location in memory). On a 64-bit system, addresses are typically 8 bytes (64 bits) long, so a pointer variable will also occupy 8 bytes.

In Go, you declare a pointer using the * operator. Pointers also have a type, which is the type of the variable they point to. For example:

var p *int32 // a pointer to an int32

You can get the address of a variable using the & operator:

var a int32 = 100
var p *int32 = &a // p now holds the address of a

In memory:

a holds the value 100 at address 0x0202. p holds the address of a (0x0202), and p itself is stored at its own address (0x0207).

The reason pointers carry type information is that you can dereference them: follow the address to access the underlying value. This is also done using the * operator:

var a int32 = 100
var p *int32 = &a // p now holds the address of a

fmt.Println(*p) // prints the value at the address p points to, which is the value of a: 100

This dual use of * is a common source of confusion, so let's clarify:

In a type declaration (like var p *int32), * indicates that p is a pointer to an int32.
In an expression (like *p), * dereferences the pointer, giving you access to the value it points to.

Next, let's break down how to declare and use pointers in practice, so you can see how & and * work together.

Declaring and Using Pointers

Now that we know what pointers are conceptually, let's see how they look in real Go code.

Pointers to Basic Types

As we saw earlier, you can declare a pointer variable with the * operator in the type:

var p *int      // p is a pointer to an int, but currently nil
fmt.Println(p)  //

Like every variable in Go, if you don't initialize it, it defaults to the zero value for its type. For pointers, the zero value is nil, meaning it doesn't point to any valid memory address.

You can also use the built-in new function to allocate a value and get its pointer:

p := new(int)  // p is a pointer to an int with a zero value (0 for int)
fmt.Println(*p) // prints 0, the zero value for int

Getting an Address with `&`

The & operator retrieves the address of an existing variable:

x := 42
p := &x // p now holds the address of x

fmt.Println(*p) // 42
*p = 99         // change the value at the address p points to (which is x)
fmt.Println(x)  // 99
fmt.Println(p)  // prints the memory address of x, e.g., 0xc0000140b8
fmt.Println(&x) // prints the same address as p

Pointers to Structs

Pointers can point to any type. They are especially common with structs:

type User struct {
    Name string
    Age  int
}

func main() {
    u := User{"Alice", 30}
    p := &u                 // pointer to User
    fmt.Println((*p).Age)   // 30
    fmt.Println(p.Name)     // Alice - shorthand for (*p).Name
}

You can access fields with either (*p).Name or simply p.Name. Go automatically dereferences struct pointers for convenience.

We'll explore struct pointers more in the "Pointer Receivers" section.

Pointers to Other User Types

You can create pointers to any named type:

type Point struct {
    X, Y int
}
p := &Point{X: 1, Y: 2} // pointer to Point
fmt.Println(p.X, p.Y)   // 1 2 - shorthand for (*p).X and (*p).Y

The syntax works exactly the same for user-defined types as it does for built-ins.

Why Use Pointers?

At first glance, pointers may look like mental gymnastics. Why not just use values directly? Here are some key reasons to use pointers in Go:

Avoiding Copies

When you assign a value in Go, it's copied:

type User struct {
    Name string
    Age  int
}

u1 := User{"Alice", 30}
u2 := u1  // copy
u2.Age = 40

fmt.Println(u1.Age) // 30
fmt.Println(u2.Age) // 40

When the struct is small with just a few fields (like this example), copying is cheap. But if it's large (hundreds of fields or nested data), copying can be inefficient. Passing a pointer avoids making a full copy:

func Birthday(u *User) {
    u.Age++
}

u := User{"Bob", 29}
Birthday(&u)
fmt.Println(u.Age) // 30

Sometimes you want multiple parts of your program to work with the same object. With values, each assignment makes a copy:

type Counter struct {
    value int32
}

c1 := Counter{value: 0} // c1 is a Counter
c2 := c1  // c2 is a copy - another Counter

c2.value++
fmt.Println(c1.value) // 0
fmt.Println(c2.value) // 1

Using pointers ensures both variables refer to the same underlying data:

pc1 := &Counter{value: 0} // pc1 is a pointer to a Counter
pc2 := pc1   // copy of the pointer - both point to the same Counter

pc2.value++
fmt.Println(pc1.value) // 1
fmt.Println(pc2.value) // 1

This diagram illustrates the two memory layouts:

c1 and c2 are stored separately at 0x0202 and 0x0207, each with its own 4-byte value field. In the second example, pc1 and pc2 are stored at 0x0202 and 0x020a respectively, and both hold the same address (0x1002) pointing to a single Counter instance in the heap, having its own 4-byte value field.

Method Receivers

Go methods can have value receivers or pointer receivers. Pointer receivers are needed when:

The method should modify the struct
The struct is large and copying would be costly
You want consistency (it's common to make all receivers pointers if some need to be)

We'll cover this in detail in the "Pointer Receivers" section.

Interfacing with Low-Level APIs

Some libraries and system calls require you to pass memory addresses, not copies. Pointers make this possible, while still being type-safe in Go.

Common Pitfalls and Misunderstandings

Nil Pointers

If you declare a pointer without initializing it, it will be nil. Dereferencing a nil pointer will immediately cause a runtime panic:

var p *int
fmt.Println(*p) // panic: runtime error: invalid memory address or nil pointer dereference

This is Go's way of telling you that you tried to follow an address that doesn't exist. To safely use pointers, you always need to give them a valid target before dereferencing:

x := 42
p := &x          // p points to x
fmt.Println(*p)  // 42

q := new(int)    // allocates memory for an int, initializes it to 0
fmt.Println(*q)  // 0

Both & and new ensure that the pointer points to valid memory.

Reference Types

In Go, everything is passed by value. But this value depends on the inner representation of the type. For example, a slice is stored in memory as:

struct {
    ptr *ElementType // pointer to the underlying array
    len int          // length of the slice
    cap int          // capacity of the slice
}

When you pass a slice to a function, you are passing a copy of this struct. The ptr field still points to the same underlying array, so changes to the elements of the slice inside the function will affect the original slice.

Because of this behavior, slices are often referred to as reference types: you don’t need to use pointers with them to share or mutate data. Other reference types in Go include maps and channels. Strings are also considered reference types, but they are immutable, so you cannot change their contents.

Note that pointers themselves are not reference types: they are simply variables that hold memory addresses.

(To complicate things a bit further, if you pass a slice to a function and then re-slice it or append to it, you are modifying the copy of the slice struct. The original slice outside the function will not see these changes!)

Pointer Receivers

When defining methods on structs in Go, you can choose between value receivers and pointer receivers. Understanding the difference is key to writing correct and efficient Go code.

Value Receivers vs Pointer Receivers

Value receiver: the method gets a copy of the struct. Any modifications inside the method do not affect the original struct.

Pointer receiver: the method gets a copy of the pointer, which still points to the original struct. Modifications inside the method affect the original struct.

Example:

type Counter struct {
    value int
}

If you try to increment the counter using a value receiver, it won't work as expected:

func (c Counter) Inc() {
    c.value++ // INCORRECT: modifies the copy, not the original
}

c := Counter{value: 5}
c.Inc()
fmt.Println(c.value) // still 5

With a pointer receiver, it works correctly:

func (c *Counter) Inc() {
    // note the shorthand syntax to (*c).value
    c.value++ // CORRECT: modifies the original via the pointer
}

c := Counter{value: 5}
c.Inc()
fmt.Println(c.value) // now 6

Even though the method receives a copy of the pointer, both the copy and the original pointer point to the same struct in memory (in the heap), so changes inside the method affect the original.

Idiomatic Go

Small structs can use value receivers if mutation isn't needed.
Large structs or any struct that must be mutated should use pointer receivers.
If some methods need pointer receivers, it's common to make all methods use pointer receivers for consistency.

Pointer receivers are arguably the most common and practical use of pointers in Go. They allow methods to mutate state safely without unnecessary copies.

Exercises for the Reader

To solidify your understanding of pointers, try the following exercises:

Write a function that swaps two integers using pointers.
Create a struct representing a Rectangle with width and height. Write methods to calculate the area and to scale the rectangle by a factor, using pointer receivers.

Bonus: Weak Pointers (Go 1.24+)

Go 1.24 introduced weak references, which let you hold a reference to a value without preventing it from being garbage-collected. This is useful when you want a cache or auxiliary data structure without prolonging the lifetime of the objects.

What Are Weak Pointers?

A weak pointer is a pointer that does not count toward keeping the referenced object alive. If the only references to an object are weak, the garbage collector can still free it.

Weak pointers are provided by the runtime/weak package:

import "runtime/weak"

type Cache struct {
    data weak.Map[string, *User]
}

func main() {
    c := Cache{data: weak.MakeMap[string, *User]()}
    u := &User{"Alice", 30}

    c.data.Set("alice", u) // weak reference stored
}

// ...later
if user, ok := c.data.Get("alice"); ok {
    fmt.Println(user.Name) // Alice
} else {
    fmt.Println("User has been garbage collected")
}

If u is no longer referenced elsewhere, the garbage collector can reclaim it, even though it exists in the weak.Map.

Essentially, a weak pointer can turn into a nil pointer if the object it points to has been garbage collected. You should always check if it's nil before dereferencing it.

When to Use Weak Pointers

Caches: Keep objects around if they're still in use, but don't prevent GC if not.
Avoid memory leaks: Especially in long-running services where temporary objects could otherwise accumulate.
Auxiliary indexing: Like mapping IDs to objects without controlling their lifetimes.

Weak pointers are an advanced feature and should be used judiciously. Most Go programs will never need them, but in certain scenarios, they can be very useful.

Summary & Best Practices

Pointers store addresses of variables and allow you to share and mutate data efficiently.
Use & to get an address, * to dereference and declare.
Pointer receivers let methods mutate structs without unnecessary copies.
Be cautious with nil pointers to avoid panics.
Reference types (slices, maps, channels) already share underlying data.
Weak pointers (Go 1.24+) provide advanced memory management for caches or auxiliary structures.

While not as powerful as in languages like C/C++, Go pointers are safe and easy to use. Experiment with them in small programs, and you'll quickly see how they can help you write more efficient and easy-to-read Go code.

Solutions to Exercises

Swapping two integers using pointers:

func swap(a, b *int) {
    *a, *b = *b, *a
}

x, y := 1, 2
swap(&x, &y)
fmt.Println(x, y) // 2 1

Here, swap takes two pointers to integers and swaps their values by dereferencing them. Without pointers, you would only swap copies of the integers, leaving the originals unchanged.

Rectangle struct with methods:

type Rectangle struct {
    Width, Height float64
}

func (r *Rectangle) Area() float64 {
    return r.Width * r.Height
}

func (r *Rectangle) Scale(factor float64) {
    r.Width *= factor
    r.Height *= factor
}

rect := Rectangle{Width: 3, Height: 4}
fmt.Println(rect.Area()) // 12
rect.Scale(2)
fmt.Println(rect.Area()) // 48

Because Scale modifies the rectangle, it uses a pointer receiver. The Area method could use either a value or pointer receiver since it doesn't modify the struct, but using a pointer receiver is consistent and avoids copying the struct.

Build a Full Stack Movie Streaming App with Go, React, MongoDB, OpenAI

Beau Carnes — Wed, 01 Oct 2025 20:59:30 +0000

We’ve just posted a full-stack course on the freeCodeCamp.org YouTube channel that will teach you to build a complete, production-ready movie streaming application named MagicStream, complete with AI-powered movie recommendations. This course is designed to give you a deep, practical understanding of a modern, powerful tech stack. Gavin Lon teaches this course.

The foundation of this project is built on high-performance technologies. On the backend, the course uses Go (Golang) paired with the lightning-fast Gin-Gonic framework to create a robust and efficient Web API. This combination is fantastic for handling concurrent requests, making the application quick and reliable. For the frontend, you'll be building a responsive and engaging user interface using React. All your data, from user profiles to movie details, will be stored in MongoDB, offering flexible and scalable data management. This combination of Go, React, and MongoDB provides a well-rounded and impressive skillset for any full-stack developer.

You'll learn to connect your Go backend directly to OpenAI’s models using the power of the LangChainGo library. This connection allows you to build a genuine, intelligent service that can analyze data and provide personalized movie suggestions. Also, you'll implement secure user authentication, including registration, login, and access token validation via middleware. Security is a priority, so the course even covers the best practice of storing access tokens in http-only cookies to prevent Cross-Site Scripting (XSS) attacks.

Once the core features are built and working, you’ll will learn to deploy the applicaiton. You’ll get hands-on experience taking your database live by deploying MongoDB to Atlas, hosting your Go/Gin-Gonic API on Render, and deploying the React client to Vercel.

Watch the full course on the freeCodeCamp.org YouTube channel (15-hour watch).

Common Slice Mistakes in Go and How to Avoid Them

Temitope Oyedele — Tue, 30 Sep 2025 16:52:28 +0000

Slices are one of the most fundamental and powerful data structures in Go. They provide a dynamic array-like interface that's both flexible and efficient. However, they can be very tricky when implementing. And if not implemented correctly, they can cause subtle bugs that would be very challenging to track down.

You'll likely think it's a problem with your algorithm or logic, spending hours debugging complex workflows. In contrast, the real issue stems from a simple misunderstanding of how slices behave under the hood. The most frustrating part? Your code might work perfectly in development with small datasets, only to fail mysteriously in production with larger data or under concurrent access.

In this article, we'll explore seven common mistakes developers make when working with slices in Go and provide practical solutions to prevent them.

Passing Slices by Value and Expecting Structural Changes
Slice Header Sharing and Unintended Mutations
Memory Leaks with Large Slice References
Incorrect Loop Variable Usage with Slices of Pointers
Modifying Slice During Range Iteration
Nil Slice vs Empty Slice Confusion
Slice Bounds and Panic Errors
Wrapping Up

Passing Slices by Value and Expecting Structural Changes

A common misunderstanding is expecting that modifications to a slice's structure (length/capacity changes) in a function will affect the original slice outside the function.

While slice elements can be modified through function parameters (because slices contain a pointer to the underlying data), the slice header itself (containing length and capacity) is passed by value.

Below is an example of this misconception:

func appendToSlice(s []int) {
    s = append(s, 4) // This creates a new slice header
    fmt.Println("Inside function:", s) // [1, 2, 3, 4]
}

func main() {
    slice := []int{1, 2, 3}
    appendToSlice(slice)
    fmt.Println("After function call:", slice) // Still [1, 2, 3]
}

In this code, the append operation inside the function creates a new slice header, but this change doesn't affect the original slice in the calling function.

How to Prevent It

To modify a slice's structure from within a function, either return the modified slice or use a pointer to the slice:

// Method 1: Return the modified slice
func appendToSlice(s []int) []int {
    return append(s, 4)
}

// Method 2: Use a pointer to the slice
func appendToSlicePtr(s *[]int) {
    *s = append(*s, 4)
}

func main() {
    // Using method 1
    slice1 := []int{1, 2, 3}
    slice1 = appendToSlice(slice1)
    fmt.Println("Method 1:", slice1) // [1, 2, 3, 4]

    // Using method 2
    slice2 := []int{1, 2, 3}
    appendToSlicePtr(&slice2)
    fmt.Println("Method 2:", slice2) // [1, 2, 3, 4]
}

Both approaches ensure that changes to the slice structure are visible to the caller.

Another common mistake is not realizing that slices created from the same underlying array share data. And not knowing this can cause unexpected mutations when you modify one slice.

Slices in Go are reference types that contain a pointer to the underlying array, along with length and capacity information. When you create a slice from another slice, they both point to the same underlying data.

Below is an example of how this can lead to surprising behavior:

func main() {
    original := []int{1, 2, 3, 4, 5}
    subset := original[1:4] // Creates [2, 3, 4]

    fmt.Println("Original:", original) // [1, 2, 3, 4, 5]
    fmt.Println("Subset:", subset)     // [2, 3, 4]

    subset[0] = 99 // Modify the first element of subset

    fmt.Println("Original after modification:", original) // [1, 99, 3, 4, 5]
    fmt.Println("Subset after modification:", subset)     // [99, 3, 4]
}

In this code, modifying the subset slice also changes the original slice because they share the same underlying array.

How to Prevent It

To prevent unintended mutations, use the copy() function to create independent slices:

func main() {
    original := []int{1, 2, 3, 4, 5}

    // Create an independent copy
    subset := make([]int, 3)
    copy(subset, original[1:4])

    fmt.Println("Original:", original) // [1, 2, 3, 4, 5]
    fmt.Println("Subset:", subset)     // [2, 3, 4]

    subset[0] = 99

    fmt.Println("Original after modification:", original) // [1, 2, 3, 4, 5] - unchanged
    fmt.Println("Subset after modification:", subset)     // [99, 3, 4]
}

The copy() function ensures that the data is duplicated rather than shared, preventing unintended side effects.

Memory Leaks with Large Slice References

Keeping references to small slices that are derived from large slices is considered a minor yet serious mistake. This is because it prevents the garbage collector from freeing the large underlying array, leading to memory leaks.

When you create a slice from a larger slice, the new slice still references the entire original array, even if it only uses a small portion of it.

Below is an example of how this memory leak can occur:

func processLargeData() []byte {
    largeData := make([]byte, 1<<30) // Allocate 1GB
    // ... fill largeData with important information ...

    // Extract just the first 100 bytes
    return largeData[:100] // Memory leak: entire 1GB stays in memory
}

func main() {
    result := processLargeData()
    // Even though result is only 100 bytes, 1GB remains allocated
    fmt.Printf("Result length: %d\n", len(result))
}

In this code, even though we only need the first 100 bytes, the entire 1GB array remains in memory because our returned slice still references it.

How to Prevent It

To prevent memory leaks, copy the needed data to a new slice when working with large datasets:

func processLargeData() []byte {
    largeData := make([]byte, 1<<30) // Allocate 1GB
    // ... fill largeData with important information ...

    // Create independent copy of needed data
    result := make([]byte, 100)
    copy(result, largeData[:100])

    return result // largeData can now be garbage collected
}

func main() {
    result := processLargeData()
    // Only 100 bytes remain in memory
    fmt.Printf("Result length: %d\n", len(result))
}

By copying the data to a new slice, you allow the garbage collector to free the large array when it's no longer needed.

Incorrect Loop Variable Usage with Slices of Pointers

There are scenarios or instances where you create what looks like a perfectly reasonable loop to collect pointers, but somehow all your pointers end up pointing to the same value. This is because Go reuses the same loop variable throughout all iterations, so taking its address always results in the same memory location.

Below is an example of how this mistake manifests:

func main() {
    var ptrs []*int

    for i := 0; i < 3; i++ {
        ptrs = append(ptrs, &i) // Wrong: all pointers reference the same variable
    }

    // Print the values
    for j, ptr := range ptrs {
        fmt.Printf("ptrs[%d] = %d\n", j, *ptr)
    }
    // Output: All pointers show the same value (3)
}

In this code, all pointers in the slice point to the same loop variable i, which has the final value of 3 after the loop completes.

How to Prevent It

To fix this issue, create a new variable in each iteration or use slice indexing:

func main() {
    // Method 1: Create a new variable in each iteration
    var ptrs1 []*int
    for i := 0; i < 3; i++ {
        j := i // Create new variable
        ptrs1 = append(ptrs1, &j)
    }

    // Method 2: Use a slice and index into it
    values := []int{0, 1, 2}
    var ptrs2 []*int
    for i := range values {
        ptrs2 = append(ptrs2, &values[i])
    }

    // Method 3: Using make and direct assignment
    values2 := make([]int, 3)
    var ptrs3 []*int
    for i := 0; i < 3; i++ {
        values2[i] = i
        ptrs3 = append(ptrs3, &values2[i])
    }

    // All methods now work correctly
    fmt.Println("Method 1:", *ptrs1[0], *ptrs1[1], *ptrs1[2]) // 0 1 2
    fmt.Println("Method 2:", *ptrs2[0], *ptrs2[1], *ptrs2[2]) // 0 1 2
    fmt.Println("Method 3:", *ptrs3[0], *ptrs3[1], *ptrs3[2]) // 0 1 2
}

These approaches ensure that each pointer references a unique memory location with the correct value.

Modifying Slice During Range Iteration

Modifying a slice while iterating over it with a range loop, can lead to issues like skipped elements, infinite loops, or processing the wrong data, depending on the type of modification.

When you use range on a slice, Go evaluates the slice's length at the beginning of the loop. However, if you modify the slice during iteration, the actual slice length may change while the loop continues based on the original length.

Below is an example of how this can cause problems:

func removeEvenNumbers() {
    numbers := []int{1, 2, 3, 4, 5, 6, 7, 8}
    fmt.Println("Original:", numbers)

    // Dangerous: modifying slice during range iteration
    for i, num := range numbers {
        if num%2 == 0 {
            // Remove even number by slicing
            numbers = append(numbers[:i], numbers[i+1:]...)
        }
    }

    fmt.Println("After removal:", numbers) // Unexpected result!
}

func main() {
    removeEvenNumbers()
    // Output might be: [1 3 5 7 8] - notice 8 wasn't removed!
}

In this code, removing elements during iteration causes the indices to shift, leading to some elements being skipped. The number 8 remains because when 6 is removed, 8 shifts to a position that has already been processed by the loop.

How to Prevent It

To safely modify slices during iteration, iterate in reverse order, use a separate result slice, or collect indices first:

// Method 1: Iterate in reverse to avoid index shifting issues
func removeEvenNumbersReverse() {
    numbers := []int{1, 2, 3, 4, 5, 6, 7, 8}
    fmt.Println("Original:", numbers)

    for i := len(numbers) - 1; i >= 0; i-- {
        if numbers[i]%2 == 0 {
            numbers = append(numbers[:i], numbers[i+1:]...)
        }
    }

    fmt.Println("After removal:", numbers) // [1, 3, 5, 7]
}

// Method 2: Build a new slice with desired elements
func filterOddNumbers() []int {
    numbers := []int{1, 2, 3, 4, 5, 6, 7, 8}
    var result []int

    for _, num := range numbers {
        if num%2 != 0 { // Keep odd numbers
            result = append(result, num)
        }
    }

    return result // [1, 3, 5, 7]
}

// Method 3: Collect indices first, then modify
func removeEvenNumbersByIndex() {
    numbers := []int{1, 2, 3, 4, 5, 6, 7, 8}
    var toRemove []int

    // First pass: collect indices of even numbers
    for i, num := range numbers {
        if num%2 == 0 {
            toRemove = append(toRemove, i)
        }
    }

    // Second pass: remove in reverse order
    for i := len(toRemove) - 1; i >= 0; i-- {
        idx := toRemove[i]
        numbers = append(numbers[:idx], numbers[idx+1:]...)
    }

    fmt.Println("Result:", numbers) // [1, 3, 5, 7]
}

These approaches ensure that your modifications don't interfere with the iteration process, giving you predictable and correct results.

Nil Slice vs Empty Slice Confusion

Another source of confusion is not understanding the difference between nil slices and empty slices, which can lead to inconsistent behavior in your applications.

A nil slice has no underlying array, while an empty slice has an underlying array but contains no elements.

Below is an example that demonstrates the differences:

func main() {
    var nilSlice []int
    emptySlice := []int{}
    emptySlice2 := make([]int, 0)

    fmt.Printf("nilSlice == nil: %t\n", nilSlice == nil)       // true
    fmt.Printf("emptySlice == nil: %t\n", emptySlice == nil)   // false
    fmt.Printf("emptySlice2 == nil: %t\n", emptySlice2 == nil) // false

    // JSON marshaling behaves differently
    nilJSON, _ := json.Marshal(nilSlice)
    emptyJSON, _ := json.Marshal(emptySlice)

    fmt.Printf("Nil slice JSON: %s\n", nilJSON)   // null
    fmt.Printf("Empty slice JSON: %s\n", emptyJSON) // []
}

This difference can cause issues when working with JSON APIs or when functions expect specific slice states.

How to Prevent It

Be explicit about your intentions and handle both cases consistently. A good practice would be to check length instead of nil when it matters:

func processSlice(s []int) {
    if len(s) == 0 { // Works for both nil and empty slices
        fmt.Println("Slice is empty")
        return
    }
    fmt.Printf("Processing %d elements\n", len(s))
}

// Initialize nil slices when needed
func ensureSliceInitialized(s []int) []int {
    if s == nil {
        return make([]int, 0) // or []int{} if you prefer non-nil empty slice
    }
    return s
}

func main() {
    var nilSlice []int
    emptySlice := []int{}

    processSlice(nilSlice)  // Works consistently
    processSlice(emptySlice) // Works consistently

    nilSlice = ensureSliceInitialized(nilSlice)
    fmt.Printf("After initialization: %t\n", nilSlice == nil) // false
}

This approach ensures consistent behavior regardless of whether you're working with nil or empty slices.

Slice Bounds and Panic Errors

The final common mistake is not validating slice bounds before accessing elements. When you fail to validate slice bound, you’re making it prone to runtime panics that can crash your application.

Go doesn't provide automatic bounds checking for slice operations, so it's your responsibility to ensure that indices are within valid ranges.

Below is an example of unsafe slice operations:

func dangerousSliceOperations(s []int, index int, start int, end int) {
    // Dangerous: can panic if index is out of bounds
    value := s[index]
    fmt.Printf("Value at index %d: %d\n", index, value)

    // Also dangerous: can panic if bounds are invalid
    subset := s[start:end]
    fmt.Printf("Subset [%d:%d]: %v\n", start, end, subset)
}

func main() {
    slice := []int{1, 2, 3, 4, 5}

    // These will cause panics
    // dangerousSliceOperations(slice, 10, 2, 8) // index out of bounds
    // dangerousSliceOperations(slice, 0, -1, 3) // negative index
}

These operations will cause runtime panics when the bounds are invalid, potentially crashing your application.

How to Prevent It

To prevent this, you need to always validate bounds before accessing slice elements:

// Safe element access with error handling
func safeGetElement(s []int, index int) (int, error) {
    if index < 0 || index >= len(s) {
        return 0, fmt.Errorf("index %d out of bounds for slice of length %d", index, len(s))
    }
    return s[index], nil
}

// Safe slice operations with error handling
func safeGetSubslice(s []int, start, end int) ([]int, error) {
    if start < 0 || end > len(s) || start > end {
        return nil, fmt.Errorf("invalid slice bounds [%d:%d] for slice of length %d", start, end, len(s))
    }
    return s[start:end], nil
}

// Bounds-checking helper that clamps values
func clampedSlice(s []int, start, end int) []int {
    if start < 0 {
        start = 0
    }
    if end > len(s) {
        end = len(s)
    }
    if start > end {
        start = end
    }
    return s[start:end]
}

func main() {
    slice := []int{1, 2, 3, 4, 5}

    // Safe access with error handling
    if value, err := safeGetElement(slice, 2); err == nil {
        fmt.Printf("Element at index 2: %d\n", value)
    }

    if subset, err := safeGetSubslice(slice, 1, 4); err == nil {
        fmt.Printf("Subset [1:4]: %v\n", subset)
    }

    // Bounds-clamped access (never panics)
    clamped := clampedSlice(slice, -1, 10)
    fmt.Printf("Clamped slice: %v\n", clamped) // [1, 2, 3, 4, 5]
}

These approaches provide safe alternatives that either handle errors gracefully or ensure that operations never exceed valid bounds.

Wrapping Up

In this article, we looked at seven frequent problems that might happen when working with slices in Go. These issues often stem from the subtle behavior of Go's slice implementation, particularly around memory sharing, the distinction between slice headers and underlying arrays, and the reference semantics of slices.

By understanding these pitfalls and implementing the prevention strategies we've discussed, you can write more robust and efficient Go applications. Remember to always consider slice capacity vs length, be mindful of shared underlying data, validate bounds before accessing elements, and understand the implications of passing slices to functions.

Mastering these concepts will help you harness the full power of Go's slices while avoiding the common traps that can lead to bugs and performance issues in your applications.

Don't forget to share.

Go Language - freeCodeCamp.org

How to Create Dynamic Emails in Go with React Email

Table of Contents

What is React Email?

Go Templates

Go Template Delimiters

Create Dynamic Emails in Go with React Email

Set Up the Project

Set Up React Email

Create a React Email Template

Set Up Go Templates from HTML Files

Render the Dynamic Email in the Browser

Send and Test Email with go-mail and MailHog

Conclusion

How to Build a Bank Ledger in Golang with PostgreSQL using the Double-Entry Accounting Principle.

The Hidden Bugs in How Most Developers Store Money

Table of Contents

Project Resources:

Prerequisites and Project Overview

Backend Request Flow

The Double-Entry Foundation: How Every Penny is Accounted For

Why the Settlement Account Goes Negative

Enforcing the Rules in the Database

The Settlement Account: The System's Anchor

Type-Safe SQL with sqlc: No More Surprises

Why NUMERIC Becomes String (and Not float64)

Preventing Race Conditions: Locking with FOR UPDATE

Calculating the True Balance: Always Trust the Entries

The Store Layer: Transactions and Automatic Retries

Why Serializable Isolation?

Automatic Retries: Handling Real-World Concurrency

The Service Layer: Where Business Logic Meets Double-Entry

Deposit: Crediting the User, Debiting the Settlement

Withdraw: Debiting the User, Crediting the Settlement

Transfer: User-to-User, No Settlement Involved

ReconcileAccount: Trust, But Verify

The API Layer: Secure, Predictable, and Boring (By Design)

JWT Authentication: Secrets Matter

The Handler Pattern: Parse, Authorize, Validate, Call, Respond

Amount Normalization: Defensive by Default

Frontend Deployment Boundary

Running It Locally: Your First End-to-End Test

Testing: Prove the System Works

Service Layer: Core Financial Logic

Concurrency Test: Proving Serializable Isolation Works

Store Layer: Transaction Mechanics

API Layer: Authentication and Input Handling

Running the Tests

Deployment: Engineering Decisions That Matter in Production

Migrations on Container Start

Startup Retry Logic

DB URL Fallback Chain

HTTP Server Timeouts

Conclusion: Building for the Real World

How to Get Started Coding in Golang

What we'll cover:

Prerequisites

How to Install Go

How to Write Your First Go Program

How to Work with Variables and Numbers in Go

How to Declare Variables

How to Declare Integers

How to Format Strings in Go

Printing Strings to the Console

Format Specifiers

How to Work with Arrays and Slices in Go

Arrays in Go

Slices in Go

How to Work with Loops in Go

How to Iterate Loops

How to Use the range Keyword

How to Work with Functions in Go

Functions with return Values

How to Work with Maps in Go

How to Work with Structs in Go

Package Scope in Go

Importing Functions Within the Same Package

Importing Functions Across Different Packages

Conclusion

How to Find the Top-K Items: Heap and Streaming Approaches in Go

How to Use the `range` Keyword

Functions with `return` Values

`t.Errorf` vs `t.Fatalf`

Table-Driven `Add` Test

Writing Tests for `SafeDivide()`