These two operations look simple, but they hide a dangerous reliability problem. What if the database write succeeds but the message broker is temporarily unreachable? Or your service crashes between the two steps? You end up in an inconsistent state: your database has the new data, but the rest of the system never heard about it.

The Outbox Pattern is a well-established solution to this problem. In this tutorial, you'll learn what the pattern is, why it works, and how to implement it in Go with PostgreSQL and Google Cloud Pub/Sub.

Prerequisites

Before reading this tutorial, you should be familiar with:

• The basics of the Go programming language

• SQL and PostgreSQL

• The concept of database transactions

• Basic familiarity with event-driven or distributed systems (helpful but not required)

Table of Contents

1. The Problem: Two Operations, No Atomicity

2. How the Outbox Pattern Works

3. The Outbox Table Schema

4. The Message Relay

5. Go and PostgreSQL Implementation

   • The Orders Service

   • The Relay Service

6. Why Messages Can Be Delivered More Than Once

7. Alternative: PostgreSQL Logical Replication

8. Conclusion

The Problem: Two Operations, No Atomicity

To understand why the Outbox Pattern exists, you need to understand a core challenge in distributed systems: atomicity across different systems.

In a relational database, a transaction lets you group multiple operations so they either all succeed or all fail together. If you insert a row and update another row in the same transaction, you're guaranteed that both happen – or neither does.

The problem arises when you try to extend this guarantee across two different systems: for example, your database and your message broker (like Kafka, RabbitMQ, or Pub/Sub). These systems don't share a transaction boundary.

Here's a typical event-driven flow that breaks without the Outbox Pattern:

1. A user places an order.

2. Your service saves the order to the database ✅

3. Your service publishes an order.created event to the message broker ❌ (broker is down)

4. The order exists in the database, but downstream services never learned about it.

Or the reverse failure:

1. Your service publishes the event first ✅

2. Your service tries to save the order to the database ❌ (database times out)

3. Downstream services received a notification for an order that doesn't exist.

Either scenario leaves your system in an inconsistent state. This is the core problem the Outbox Pattern solves.

Here's what the process looks like when not using the Outbox Pattern:

How the Outbox Pattern Works

The Outbox Pattern solves the atomicity problem by keeping both operations inside the database:

1. Saves your business data (for example, a new order) to your database.

2. Writes the event message to a special table called the outbox table in the same database transaction.

3. A separate background process called the Message Relay polls the outbox table and publishes pending messages to the broker.

4. Once the broker confirms receipt, the relay marks the message as processed.

Because steps 1 and 2 happen in the same database transaction, they are atomic. Either both succeed or neither does. You can never end up with saved data but no corresponding event queued – or an event queued for data that was never saved.

The message is never published directly to the broker in your main application code. Instead, the database acts as a reliable staging area.

The Outbox Table Schema

The outbox table stores pending messages until the relay picks them up. Here's a typical PostgreSQL schema:

CREATE TABLE outbox (
    id          uuid PRIMARY KEY DEFAULT gen_random_uuid(),
    topic       varchar(255)  NOT NULL,
    message     jsonb         NOT NULL,
    state       varchar(50)   NOT NULL DEFAULT 'pending',
    created_at  timestamptz   NOT NULL DEFAULT now(),
    processed_at timestamptz
);

Let's walk through each column:

• id: A unique identifier for each message. Using UUIDs makes it easy to reference specific messages.

• topic: The destination topic or queue name in your message broker (for example, orders.created).

• message: The event payload, stored as JSON. This is the data your consumers will receive.

• state: Tracks whether the message has been sent. The two main values are pending (waiting to be published) and processed (successfully published).

• created_at: When the message was inserted. The relay uses this to process messages in order.

• processed_at: When the relay successfully published the message.

You may want to add additional columns depending on your needs: for example, a retry_count column to track how many times the relay has attempted to send a message, or an error column to log failure reasons.

The Message Relay

The Message Relay is a background process (often a goroutine, a sidecar, or a separate service) that bridges the outbox table and the message broker.

Its responsibilities are:

1. Periodically query the outbox table for messages with state = 'pending'.

2. Publish each message to the appropriate topic in the broker.

3. Once the broker confirms delivery, update the row's state to 'processed'.

4. Handle failures gracefully: if publishing fails, leave the message as 'pending' so it will be retried.

This design gives you at-least-once delivery: a message will always be sent, even if the relay crashes and restarts. The trade-off is that a message might occasionally be sent more than once (more on this below), so your consumers should handle duplicates.

Go and PostgreSQL Implementation

Let's build a concrete example. Imagine you have an orders service. When a new order is created, you want to:

1. Save the order to a PostgreSQL orders table.

2. Publish an order.created event to Google Cloud Pub/Sub.

You'll use pgx for the PostgreSQL driver.

The Orders Service

The key insight is that the order insert and the outbox insert happen inside the same transaction. If anything goes wrong, both are rolled back.

// orders/main.go

package main

import (
	"context"
	"encoding/json"
	"log"
	"os"

	"github.com/google/uuid"
	"github.com/jackc/pgx/v5"
	"github.com/jackc/pgx/v5/pgxpool"
)

// Order represents a customer order in our system.
type Order struct {
	ID       uuid.UUID `json:"id"`
	Product  string    `json:"product"`
	Quantity int       `json:"quantity"`
}

// OrderCreatedEvent is the payload published to the message broker.
// It contains only the fields that downstream services need to know about.
type OrderCreatedEvent struct {
	OrderID uuid.UUID `json:"order_id"`
	Product string    `json:"product"`
}

// createOrderInTx saves a new order and its outbox event atomically.
// Both operations share the same transaction (tx), so either both succeed
// or both are rolled back — ensuring consistency.
func createOrderInTx(ctx context.Context, tx pgx.Tx, order Order) error {
	// Step 1: Insert the business data (the actual order).
	_, err := tx.Exec(ctx,
		"INSERT INTO orders (id, product, quantity) VALUES (\(1, \)2, $3)",
		order.ID, order.Product, order.Quantity,
	)
	if err != nil {
		return err
	}
	log.Printf("Inserted order %s into database", order.ID)

	// Step 2: Serialize the event payload that consumers will receive.
	event := OrderCreatedEvent{
		OrderID: order.ID,
		Product: order.Product,
	}
	msg, err := json.Marshal(event)
	if err != nil {
		return err
	}

	// Step 3: Write the event to the outbox table.
	// This does NOT publish to Pub/Sub — it just queues it for the relay.
	_, err = tx.Exec(ctx,
		"INSERT INTO outbox (topic, message) VALUES (\(1, \)2)",
		"orders.created", msg,
	)
	if err != nil {
		return err
	}
	log.Printf("Inserted outbox event for order %s", order.ID)

	return nil
}

func main() {
	ctx := context.Background()

	pool, err := pgxpool.New(ctx, os.Getenv("DATABASE_URL"))
	if err != nil {
		log.Fatalf("Unable to connect to database: %v", err)
	}
	defer pool.Close()

	// Begin a transaction that will cover both the order insert
	// and the outbox insert.
	tx, err := pool.Begin(ctx)
	if err != nil {
		log.Fatalf("Unable to begin transaction: %v", err)
	}
	// If anything fails, the deferred Rollback is a no-op after a successful Commit.
	defer tx.Rollback(ctx)

	newOrder := Order{
		ID:       uuid.New(),
		Product:  "Super Widget",
		Quantity: 10,
	}

	if err := createOrderInTx(ctx, tx, newOrder); err != nil {
		log.Fatalf("Failed to create order: %v", err)
	}

	// Committing the transaction makes both writes permanent simultaneously.
	if err := tx.Commit(ctx); err != nil {
		log.Fatalf("Failed to commit transaction: %v", err)
	}

	log.Println("Successfully created order and queued outbox event.")
}

Notice that createOrderInTx receives a pgx.Tx (a transaction) rather than a pool connection. This is intentional: it enforces that the caller is responsible for managing the transaction boundary, making the atomicity guarantee explicit.

The Relay Service

The relay runs as a separate background process. It polls the outbox table, publishes messages, and marks them as processed.

A critical detail here is the use of FOR UPDATE SKIP LOCKED in the SQL query. This PostgreSQL feature lets you run multiple relay instances concurrently without them stepping on each other. When one instance locks a row to process it, other instances skip that row and move on to the next one.

// relay/main.go

package main

import (
	"context"
	"log"
	"time"

	"cloud.google.com/go/pubsub"
	"github.com/google/uuid"
	"github.com/jackc/pgx/v5/pgxpool"
)

// OutboxMessage mirrors the columns we need from the outbox table.
type OutboxMessage struct {
	ID      uuid.UUID
	Topic   string
	Message []byte
}

// processOutboxMessages picks up one pending message, publishes it to Pub/Sub,
// and marks it as processed — all within a single database transaction.
func processOutboxMessages(ctx context.Context, pool *pgxpool.Pool, pubsubClient *pubsub.Client) error {
	tx, err := pool.Begin(ctx)
	if err != nil {
		return err
	}
	defer tx.Rollback(ctx)

	// Query for the next pending message.
	// FOR UPDATE SKIP LOCKED ensures that if multiple relay instances are
	// running, they won't try to process the same message simultaneously.
	rows, err := tx.Query(ctx, `
		SELECT id, topic, message
		FROM outbox
		WHERE state = 'pending'
		ORDER BY created_at
		LIMIT 1
		FOR UPDATE SKIP LOCKED
	`)
	if err != nil {
		return err
	}
	defer rows.Close()

	var msg OutboxMessage
	if rows.Next() {
		if err := rows.Scan(&msg.ID, &msg.Topic, &msg.Message); err != nil {
			return err
		}
	} else {
		// No pending messages — nothing to do.
		return nil
	}

	log.Printf("Publishing message %s to topic %s", msg.ID, msg.Topic)

	// Publish the message to the Pub/Sub topic and wait for confirmation.
	result := pubsubClient.Topic(msg.Topic).Publish(ctx, &pubsub.Message{
		Data: msg.Message,
	})
	if _, err = result.Get(ctx); err != nil {
		// Publishing failed. We return the error here without committing,
		// so the transaction rolls back and the message stays 'pending'.
		// The relay will retry it on the next polling interval.
		return err
	}

	// Mark the message as processed now that the broker has confirmed receipt.
	_, err = tx.Exec(ctx,
		"UPDATE outbox SET state = 'processed', processed_at = now() WHERE id = $1",
		msg.ID,
	)
	if err != nil {
		return err
	}
	log.Printf("Marked message %s as processed", msg.ID)

	// Commit the transaction: the state update becomes permanent.
	return tx.Commit(ctx)
}

func main() {
	// In production, initialize real connections using environment variables
	// or a config file. These are left as placeholders for clarity.
	var (
		pool         *pgxpool.Pool
		pubsubClient *pubsub.Client
	)

	// Poll the outbox table every second.
	// Adjust the interval based on your latency requirements.
	ticker := time.NewTicker(1 * time.Second)
	defer ticker.Stop()

	for range ticker.C {
		if err := processOutboxMessages(context.Background(), pool, pubsubClient); err != nil {
			log.Printf("Error processing outbox: %v", err)
		}
	}
}

The polling interval (1 second in this example) controls the maximum latency between an event being written to the outbox and it being published to the broker. For most use cases, 1–5 seconds is perfectly acceptable. If you need lower latency, you can reduce the interval, or consider using PostgreSQL's LISTEN/NOTIFY feature to wake up the relay immediately when a new row is inserted.

Why Messages Can Be Delivered More Than Once

You might wonder: isn't the Outbox Pattern supposed to guarantee exactly once delivery?

It does not. It guarantees at-least-once delivery. Here's the edge case:

1. The relay publishes the message to Pub/Sub successfully.

2. Before it can update the outbox row to 'processed', the relay process crashes.

3. On restart, the relay sees the message is still 'pending' and publishes it again.

This is a rare but possible scenario. The standard way to handle it is to design your message consumers to be idempotent. This means that they can safely receive and process the same message multiple times without causing incorrect behavior.

Common strategies for idempotency include:

• Using the message's id as a deduplication key, and checking if you've already processed it before acting.

• Making your operations naturally idempotent. For example, using INSERT ... ON CONFLICT DO NOTHING instead of a plain INSERT.

Alternative: PostgreSQL Logical Replication

The polling approach described above is simple and works well, but it has two drawbacks: it introduces some latency (up to one polling interval), and it issues database queries even when there's nothing to process.

For high-throughput systems where these trade-offs matter, PostgreSQL offers a more advanced alternative: logical replication via the Write-Ahead Log (WAL).

Every change made to a PostgreSQL database is first written to the WAL – an append-only log used for crash recovery and replication. With logical replication, you can subscribe to changes in specific tables and receive them as a stream in near real-time.

Instead of your relay asking "Are there any new messages?" on a schedule, PostgreSQL will proactively notify your relay the moment a new row is inserted into the outbox table.

This approach is lower latency and more resource-efficient for high-volume workloads. The trade-off is added implementation complexity: you need to manage a replication slot in PostgreSQL and handle the WAL stream correctly.

In Go, you can use the pglogrepl library to interact with PostgreSQL's logical replication protocol.

For more details on how WAL and change data capture work in PostgreSQL, see the official Write-Ahead Logging documentation.

Conclusion

The Outbox Pattern solves a fundamental problem in distributed systems: how do you reliably perform a database write and publish a message to a broker in a consistent way?

The key idea is to use your database as the source of truth for both the business data and the pending messages. By writing to the outbox table in the same transaction as your business data, you get atomic guarantees from the database itself: no distributed transaction protocol required.

Here's a quick summary of the key concepts:

• The outbox table stores pending events as part of your regular database schema.

• The transaction wraps both the business write and the outbox write, making them atomic.

• The Message Relay is a background process that reads from the outbox and publishes to the broker.

• At-least-once delivery means your consumers must be idempotent.

• FOR UPDATE SKIP LOCKED allows multiple relay instances to run safely in parallel.

• Logical replication is an advanced alternative that avoids polling for high-throughput systems.

The pattern is simple in concept, but there are several ways to implement it depending on your scale and infrastructure. The polling approach shown in this tutorial is a solid starting point for most applications.

Resources

• Source code on GitHub

• PostgreSQL Write-Ahead Logging (WAL)

• pglogrepl – Go library for PostgreSQL logical replication

• pgx – PostgreSQL driver and toolkit for Go

• Explore more Go tutorials on packagemain.tech

This makes it easier to understand, implement, and consume those APIs. And standards matter, as they allow different teams, regardless of their technology stack, to effectively communicate about and work with the same API.

In this practical guide, I’ll want to walk you through all the important parts involved in architecting, implementing, and consuming an API using the OpenAPI standard.

Before we dive in, it's helpful to have a basic understanding of the following:

• The Go programming language

• RESTful APIs

• JSON/YAML

• Basic command-line usage

Table of Contents

1. What is the OpenAPI Specification (OAS)?

2. Architecting the API

   • openapi.yaml

   • Paths and Operations

   • Schemas

   • Extensions

3. How to Generate a Go Server

4. How to visualize API docs

5. Client Code

6. Conclusion

What is the OpenAPI Specification (OAS)?

The OpenAPI Specification (OAS) provides a consistent means to carry information through each stage of the API lifecycle. It is a specification language for HTTP APIs that defines structure and syntax in a way that is not wedded to the programming language the API is created in.

The OpenAPI Specification (OAS) was originally based on the Swagger 2.0 Specification from SmartBear Software. Later it was moved to the OpenAPI Initiative (OAI), a consortium of industry experts under the Linux Foundation.

The main idea of OpenAPI is to be able to describe APIs in agnostic terms, decoupling them from any specific programming language. Consumers of your API specification do not need to understand the guts of your application or try to learn Lisp or Haskell if that’s what you chose to write it in. They can understand exactly what they need from your API specification, written in a simple and expressive language.

This simple and expressive language is called DSL (domain specific language). It can be written in either JSON or YAML.

The latest version of OAS is v3.1.1 and the specification itself is huge. There are many features and corner cases, but we will try to go through the most important ones.

Architecting the API

It all starts with defining what the API should provide for its consumers and what it is for. While this stage isn't always purely technical, having a sketch of your API design in OAS when gathering requirements gives you a head start when starting the design.

Once the requirements are ready, it's time to open your OpenAPI editor and collaborate with your teammates.

And it's important to understand that it's not only about writing the JSON/YAML spec, but actually agreeing on the API design.

I recommend that you follow some API design guide – Google has one, for example. This will help you avoid mixed styles (like /resourceName/{id} and /resource_name/{id}, inconsistent use of HTTP methods, or unclear resource relationships.

openapi.yaml

The spec of your API starts in the entrypoint document openapi.yaml (recommended but not required name) or openapi.json. I've seen very big openapi.yaml files (50k lines), but it's possible to split your spec into multiple parts. Just keep in mind that this may not work well for some OpenAPI tools as they expect a single file. Google Maps OAS is a good example on how to split the schema, but also comes with a pre-processor to generate a single file.

There are some open source tools to bundle the OAS: swagger-cli (archived) and redocly-cli are great options.

swagger-cli bundle -o _bundle/openapi.yaml openapi.yaml

As I mentioned earlier, the spec is huge, but let's break it into smaller parts. For this tutorial I created a dummy "Smart Home" API. You can see the full spec and code here.

The root object is called OpenAPI Object and has the following structure:

# schema version
openapi: 3.1.1

# docs
info:
  title: Smart Home API
  description: API Specification for Smart Home API
  version: 0.0.1

# optional servers for public APIs
servers:
  - url: "https://..."

# tags are used to group the endpoints
tags:
  - name: device
    description: Manage devices
  - name: room
    description: Manage rooms

# endpoints go here
paths:
  # ...

# reusable objects such as schemas, error types, request bodies
components:
  # ...

# security mechanisms, should correspond to components.securitySchemes
security:
  - apiKeyAuth: []

We defined the skeleton of our schema, but the majority of OpenAPI schema lays in the paths and components props.

Paths and Operations

Let's now add a few endpoints to our schema. The operations are grouped by paths, so you can have multiple HTTP methods on a single path – for example GET /devices/{deviceId} and DELETE /devices/{deviceId}.

It's a good practice to define all types (request bodies, responses, errors) in the components section and reference them instead of manually defining them in the paths section. This allows for easier re-use of entities. For example, in our API we have a type Device which can be used in many endpoints.

paths:

  # the path has a parameter in it
  /devices/{deviceId}:
    get:
      tags:
        - device
      summary: Get Device
      operationId: getDevice

      parameters:
        - name: deviceId
          in: path
          required: true
          schema:
            $ref: "#/components/schemas/ULID"

      responses:

        "200":
          description: Success
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Device"

        "404":
          description: Not Found
          content:
            application/json:
              schema:
                # use common type for 404 errors
                $ref: "#/components/schemas/ErrorNotFound"

In the spec above, we defined two endpoints of our API and referenced the types which we still need to define: Device, ErrorNotFound and ULID. Notice that for the deviceId path param we also used a custom type instead of a standard string, which can be helpful in the future in case we want to change the format of our IDs (for example UUID, ULID. integer, and so on).

Notice that each operation has a unique operationId. While it's optional, it's very helpful to set one, so then it can be used on the server and client sides.

This is a basic configuration which you can extend further if you want to. For example, when serving this schema in Swagger, it's good to see the examples of our requests (and their variations). We can define it here in responses section, or directly in our components.schemas.

responses:
  "200":
    content:
      application/json:
        examples:
          new_device:
            value: # any value

Schemas

components is an integral part of OAS, and contains the following properties:

• schemas

• responses

• parameters

• requestBodies

• headers

• securitySchemes

You can see all here.

We could define our Device type like this:

components:
  schemas:
    Device:
      type: object
      properties:
        id:
          $ref: '#/components/schemas/ULID'
        name:
          type: string
      required:
        - id
        - name

But later you may have other types that have name or id fields, so it's recommended to define them separately and combine them in the final type using allOf:

components:
  schemas:
    WithId:
      type: object
      required:
        - id
      properties:
        id:
          $ref: "#/components/schemas/ULID"

    WithName:
      type: object
      required:
        - name
      properties:
        name:
          type: string

    Device:
      allOf:
        - $ref: "#/components/schemas/WithId"
        - $ref: "#/components/schemas/WithName"

allOf, oneOf, and anyOf are very powerful techniques for modeling your OAS.

Extensions

OpenAPI schemas can be extended with internal properties that do not affect the schema itself, but are useful for server or client generators. A good example is our ULID type for ids:

ULID:
  type: string
  minLength: 26
  maxLength: 26

  # example is useful for Swagger docs
  example: 01ARZ3NDEKTSV4RRFFQ69G5FAV

  x-go-type: ulid.ULID
  x-go-type-import:
    path: github.com/oklog/ulid/v2

The x- props will be used by the Go server generator to use existing Go types for this field instead of generating a new one.

How to Generate a Go Server

We didn't go through all possible schema properties here and just covered the main ones – so if you’re not familiar with OAS, you should now have a good understanding of this standard. You can read the whole specification here. But now as our schema is ready, we can generate a Go server from it.

You can find the full list of generators on opeanapi.tools – there are a lot of them. But the most popular one for Go servers is oapi-codegen.

oapi-codegen currently doesn’t support this OAS 3.1. issue. ogen does, though.

You can install it via go install:

go install github.com/oapi-codegen/oapi-codegen/v2/cmd/oapi-codegen@latest

The configuration for the oapi-codegen generator is straightforward. You can either provide command line arguments or specify the same arguments in a yaml configuration file. You can choose which HTTP router to use for the server, where to put the output file, and more. In our case let's use the echo router.

# oapi-codegen.yaml

package: api
output: pkg/api/api.gen.go

generate:
  strict-server: true
  models: true
  echo-server: true

We can now generate the server code using the following command:

oapi-codegen --config=oapi-codegen.yaml openapi.yaml

Let's now explore the generated api.gen.go file.

Since we enabled strict-server, which will generate code that parses request bodies and encodes responses automatically, the interface that we need to implement is called StrictServerInterface:

type StrictServerInterface interface {

  // List Devices
  // (GET /devices)
  ListDevices(ctx context.Context, request ListDevicesRequestObject) (ListDevicesResponseObject, error)

  // Get Device
  // (GET /devices/{deviceId})
  GetDevice(ctx context.Context, request GetDeviceRequestObject) (GetDeviceResponseObject, error)

}

All our types are also generated:

type ULID = ulid.ULID

type Device struct {
    Id   ULID   `json:"id"`
    Name string `json:"name"`
}

// ...

As well as code to parse the requests automatically and the Swagger definition.

Implementation

What's left for us to do is to create a server using echo, implement the generated interface, and glue everything together. We can write the following code in pkg/api/impl.go:

package api

import "context"

type Server struct{}

func NewServer() Server {
    return Server{}
}

func (Server) ListDevices(ctx context.Context, request ListDevicesRequestObject) (ListDevicesResponseObject, error) {
    // actual implementation
    return ListDevices200JSONResponse{}, nil
}

func (Server) GetDevice(ctx context.Context, request GetDeviceRequestObject) (GetDeviceResponseObject, error) {
    // actual implementation
    return GetDevice200JSONResponse{}, nil
}

I skipped the implementation part and just demonstrated how to return the responses. It's quite handy that oapi-codegen generated all possible responses for us.

That leaves us to start the echo server itself. Note that we don't need to write any endpoints manually now, and all request and response parsing is handled for us. Still, we need to validate the requests inside our implementation.

package main

import (
    "oapiexample/pkg/api"

    "github.com/labstack/echo/v4"
)

func main() {
    server := api.NewServer()

    e := echo.New()

    api.RegisterHandlers(e, api.NewStrictHandler(
        server,
        // add middlewares here if needed
        []api.StrictMiddlewareFunc{},
    ))

    e.Start("127.0.0.1:8080")
}

Now when we run our server using go run ., we can curl localhost:8080/devices to see the response!

Supported servers

oapi-codegen supports many web frameworks/servers, such as Chi, Fiber, Gin as well as standard net/http.

How to visualize API docs

Sometimes it's handy to have Swagger docs shipped together with your API – for testing, for example, or just as public documentation. oapi-codegen doesn't generate the Swagger UI out of the box, but we can have a simple HTML page that has a Swagger JS which loads our OAS.

You can find the HTML code for our pkg/api/index.html here.

And then we can use go:embed to embed the static files and add our Swagger endpoint:

//go:embed pkg/api/index.html
//go:embed openapi.yaml
var swaggerUI embed.FS

func main() {
    // ...

    // serve swagger docs
    e.GET("/swagger/*", echo.WrapHandler(http.StripPrefix("/swagger/", http.FileServer(http.FS(swaggerUI)))))
}

Now we can visit localhost:8080/swagger/ to see the Swagger UI with our OAS.

Tools like Postman are very popular for API documentation, and it's also possible to import your existing OpenAPI 3.0 and 3.1 definitions into Postman. Postman supports both YAML and JSON formats.

Generate OAS from code

There is also a practice to generate OpenAPI schemas from code, especially in typed languages. This approach has been popular, with the main selling point being that keeping your OpenAPI schema near the code will hopefully mean that developers keep it up to date as they work on the code.

This is not always the case, which is one of a few reasons this practice is dying out. And I am also not a big fan, as I haven’t seen a big value in this. Anyway, you can have a look at the following projects: go-swagger, swag, swaggest/rest.

Client Code

As mentioned earlier, OpenAPI is very powerful for collaboration between teams, and all you have to do now is to properly version your schema (see info.version part) and distribute it across the teams.

This part can be automated to some extent by packaging your OpenAPI schema and making it available. I've seen devs use Git submodules for that or GitHub actions to publish the version schemas.

Let's assume our client is a web application written in TypeScript, which is quite common for web APIs. Again, there are may generators available at opeanapi.tools online but the most popular one is openapi-typescript.

Here's how you can generate the TypeScript code for local or remote schemas:

# Local schema
npx openapi-typescript openapi.yaml -o ./client/schema.d.ts

# Remote schema
npx openapi-typescript https://.../openapi.yaml -o ./client/schema.d.ts

Conclusion

OpenAPI is a de-facto standard for designing, implementing, and consuming REST APIs, so it's crucial to understand how it works.

I hope this article has provided a useful introduction to the OpenAPI Specification, as well as practical tips and examples for how to use OAS to architect, implement, and consume APIs.

Resources

• Source code

• OpenAPI Initiative

• openapi.tools

• Swagger Editor

• oapi-codegen

• Explore more articles on packagemain.tech

In this article, we’ll go through a collection of CLI / TUI tools that have been widely adopted in the developer community, spanning various categories such as version control, system utilities, text editors, and more. I wanted to give you a diverse selection that caters to different needs and workflows.

For each tool, I’ll include an overview, highlighting its key features and use cases, along with clear and concise installation instructions for various operating systems, ensuring you can quickly get up and running with these valuable command-line companions.

Table of Contents

• Kubernetes Tools

• Container Tools

• File and Text Tools

• Git Tools

• Development Tools

• Networking Tools

• Workstation Tools

Kubernetes Tools

k9s — Kubernetes CLI To Manage Your Clusters In Style

K9s is a must-have tool for anyone working with Kubernetes. Its intuitive terminal-based UI, real-time monitoring capabilities, and powerful command options make it a standout in the world of Kubernetes management tools.

The K9s project is designed to continually watch Kubernetes cluster for changes and offer subsequent commands to interact with observed resources. This makes it easier to manage applications, especially in a complex, multi-cluster environment. The project’s aim is to make Kubernetes management more accessible and less daunting, especially for those who are not Kubernetes experts.

Just launch k9s in your terminal and start exploring the Kubernetes resources with ease.

To install K9s:

# via Homebrew for macOS
brew install derailed/k9s/k9s

# via snap for Linux
snap install k9s --devmode

# via Chocolatey for Windows
choco install k9s

# via go install
go install github.com/derailed/k9s@latest

kubectx — switch between contexts (clusters) on kubectl faster.

Kubectx is the most popular tool for switching Kubernetes contexts, but it has the fewest features! It displays all the contexts in your Kubernetes config as a selectable list and lets you pick one. That’s it!

This project comes with 2 tools:

• kubectx is a tool that helps you switch between contexts (clusters) on kubectl faster.

• kubens is a tool to switch between Kubernetes namespaces (and configure them for kubectl) easily.

These tools make it very easy to switch between Kubernetes clusters and namespaces if you work with many of them daily. Here you can see it in action:

To install kubectx:

# via Homebrew for macOS
brew install kubectx

# via apt for Debian
sudo apt install kubectx

# via pacman for Arch Linux
sudo pacman -S kubectx

# via Chocolatey for Windows
choco install kubens kubectx

kubescape — Kubernetes security platform for your IDE, CI/CD pipelines, and clusters.

I hope you take the security of your Kubernetes clusters seriously. If so, kubescape is really great for testing if your Kubernetes cluster is deployed securely according to multiple frameworks.

Kubescape can scan clusters, YAML files, and Helm charts and detects the misconfigurations according to multiple sources.

I usually use it in my CI/CD to scan for vulnerabilities automatically when changing Kubernetes manifests or Helm templates.

To install kubescape:

# via Homebrew for macOS
brew install kubescape

# via apt for Debian
sudo add-apt-repository ppa:kubescape/kubescape
sudo apt update
sudo apt install kubescape

# via Chocolatey for Windows
choco install kubescape

Container Tools

ctop — A top-like interface for container metrics.

ctop is basically a better version of docker stats. It provides a concise and condensed overview of real-time metrics for multiple containers. It comes with built-in support for Docker and runC, and connectors for other container and cluster systems are planned for future releases.

Using ctop is simple. Once you have the tool open, you’ll see all of your currently active containers listed.

To install ctop:

# via Homebrew for macOS
brew install ctop

# via pacman for Arch Linux
sudo pacman -S ctop

# via scoop for Windows
scoop install ctop

lazydocker — A simple terminal UI for both docker and docker-compose.

While Docker's command-line interface is powerful, sometimes you might want a more visual approach without the overhead of a full GUI. This is especially true when managing Docker containers on a headless Linux server where installing a web-based GUI might be undesirable.

Lazydocker was created by Jesse Duffield to help make managing docker containers a bit easier. Simply put, Lazydocker is a terminal UI (written in Golang) for the docker and docker-compose commands.

To install lazydocker:

# via Homebrew for macOS
brew install lazydocker

# via Chocolatey for Windows
choco install lazydocker

# via go install
go install github.com/jesseduffield/lazydocker@latest

dive — A tool for exploring each layer in a Docker image.

A Docker image is made up of layers, and with every layer you add on, more space will be taken up by the image. Therefore, the more layers in the image, the more space the image will require.

That’s where dive shines, it helps you explore your Docker image and layer contents. It can also help you find ways to shrink the size of your Docker/OCI image.

To install dive:

# via Homebrew for macOS
brew install dive

# via pacman for Arch Linux
pacman -S dive

# via go install
go get github.com/wagoodman/dive

File and Text Tools

jq — Command-line JSON processor.

You may be aware of this one already as it’s well known in the developer community.

Unfortunately, shells such as Bash can’t interpret and work with JSON directly. That’s where you can use jq as a command-line JSON processor that’s similar to sed, awk, grep, and so on for JSON data. It’s written in portable C and doesn’t have any runtime dependencies. This lets you slice, filter, map, and transform structured data with ease.

To install jq, you can download the latest releases from the GitHub release page.

bat — A cat(1) clone with wings.

This is the most used CLI on my machine currently. A few years ago it was cat, which is great but doesn’t provide syntax highlighting, or Git integration

Bat’s syntax highlighting supports many programming and markup languages, helping you make your code more readable directly in the terminal. Git integration lets you see modifications in relation to the index, highlighting the lines you’ve added or changed.

Simply run bat filename and enjoy its output.

To install bat:

# via Homebrew for macOS
brew install bat

# via apt for Debian
sudo apt install bat

# via pacman for Arch Linux
pacman -S bat

# via Chocolatey for Windows
choco install bat

ripgrep — Recursively search directories for a regex pattern while respecting your gitignore.

ripgrep is definitely becoming a popular alternative (if not the most popular) to the grep command. Even some editors like Visual Studio Code are using ripgrep to power their search offerings.

The major selling point is its default behavior for recursive search and speed.

I now rarely use grep on my personal machine, as ripgrep is much faster.

To install ripgrep:

# via Homebrew for macOS
brew install ripgrep

# via apt for Debian
sudo apt-get install ripgrep

# via pacman for Arch Linux
pacman -S ripgrep

# via Chocolatey for Windows
choco install ripgrep

Git Tools

lazygit — Simple terminal UI for git commands.

lazygit is another great terminal UI for Git commands developed by Jesse Duffield using Go.

I don’t mind using the Git CLI directly for simple things, but it is famously verbose for more advanced use cases. I am just too lazy to memorize longer commands.

And lazigit has made me a more productive Git user than ever.

To install lazygit:

# via Homebrew for macOS
brew install jesseduffield/lazygit/lazygit

# via pacman for Arch Linux
pacman -S lazygit

# via scoop for Windows
scoop install lazygit

Development Tools

ATAC — A simple API client (Postman-like) in your terminal.

ATAC stands for Arguably a Terminal API Client. It’s based on popular clients like Postman, Insomnia, and Bruno, but it runs inside your terminal without needing any particular graphical environment.

It works best for developers who need an offline, cross-platform API client right at their fingertips (terminal).

To install ATAC:

# via Homebrew for macOS
brew tap julien-cpsn/atac
brew install atac

# via pacman for Arch Linux
pacman -S atac

k6 — A modern load testing tool, using Go and JavaScript.

I’ve used many load-testing tools in my career, such as vegeta or even ab in the past. But now I mostly use k6s as it has everything I need and has a great GUI and TUI.

Why it works well for me:

• k6 has really good documentation

• Many integrations available: Swagger, JMeter scripts, and so on.

• Results reporting is quite good

To install k6:

# via Homebrew for macOS
brew install k6

# via apt for Debian
sudo apt-get install k6

# via Chocolatey for Windows
choco install k6

httpie — modern, user-friendly command-line HTTP client for the API era.

Don’t get me wrong, curl is great, but not very human-friendly.

HTTPie has a simple and expressive syntax, supports JSON and form data, handles authentication and headers, and displays colorized and formatted output.

To install httpie:

# via Homebrew for macOS
brew install httpie

# via apt for Debian
sudo apt install httpie

# via pacman for Arch Linux
pacman -Syu httpie

# via Chocolatey for Windows
choco install httpie

asciinema — Terminal session recorder.

I call it a terminal YouTube :)

asciinema is a great tool when you want to share your terminal sessions with someone else, instead of recording heavy videos.

I use it often when I develop some CLI tools and want to share the demo of how they work (on GitHub, for example).

To install asciinema:

# via Homebrew for macOS
brew install asciinema

# via apt for Debian
sudo apt install asciinema

# via pacman for Arch Linux
sudo pacman -S asciinema

Networking

doggo — A command-line DNS client.

It's totally inspired by dog which is written in Rust.

In the past I would use dig to inspect the DNS, but its output is often verbose and difficult to parse visually.

doggo addresses these shortcomings by offering two key improvements:

• doggo provides the JSON output support for easy scripting and parsing.

• doggo offers a human-readable output format that uses color-coding and a tabular layout to present DNS information clearly and concisely.

To install doggo:

# via Homebrew for macOS
brew install doggo

# via scoop for Windows
scoop install doggo

# via go install
go install github.com/mr-karan/doggo/cmd/doggo@latest

gping — Ping, but with a graph.

The well-known ping command is not the most interesting to look at, and interpreting its output in a useful way can be difficult.

gping gives a plot of the ping latency to a host, and the most useful feature is the ability to run concurrent pings to multiple hosts and plot all of them on the same graph.

To install gping:

# via Homebrew for macOS
brew install gping

# via Chocolatey for Windows
choco install gping

# via apt for Debian
apt install gping

Workstation

tmux — A terminal multiplexer.

Why is tmux such a big deal?

You may have run into situations where you need to view multiple terminal consoles at the same time. For example, you may have a few servers running (for example, web, database, debugger) and you might want to monitor all the output coming from these servers in real-time to validate behavior or run commands.

Before tmux, you might have just opened a few different tabs in the terminal and switched between them to see the output.

Thankfully, there’s an easier way — tmux.

In a nutshell, here are some of its most popular features:

• Window/Pane management

• Session management with persistence

• Sharable sessions with other users

• Scriptable configurations

To install tmux:

# via Homebrew for macOS
brew install tmux

# via apt for Debian
apt install tmux

# via pacman for Arch Linux
pacman -S tmux

zellij — A terminal workspace with batteries included.

Since I listed tmux here, it also makes sense to include a new competitor, Zellij, which has been gaining traction in the developer community. Both have their own unique features and purposes.

Compared to traditional terminal multiplexers, zellij offers a more user-friendly interface, modern design elements, built-in layout systems, and a plugin system, making it easier for newcomers to get started.

I still like tmux. It has a special place in my heart because it has served a great purpose for years. But zellij is another good option.

To install zellij:

# via Homebrew for macOS
brew install zellij

# via apt for Debian
apt install zellij

# via pacman for Arch Linux
pacman -S zellij

btop — A monitor of resources.

I can’t live without btop, and it’s installed on all my machines via my personal dotfiles. I rarely use now built-in OS GUIs to check the resource utilization on my host machine, because btop can do it much better.

I use to to quickly explore what uses the most memory, monitor and kill some processes, and more.

To install btop:

# via Homebrew for macOS
brew install btop

# via snap for Debian
sudo snap install btop

Conclusion

These CLIs/TUIs should work well in any modern terminal. I personally use Ghostty currently and it works great, but other popular options like iTerm2, Kitty, and the default terminal applications on macOS and Linux should also provide a seamless experience. The key is to ensure your terminal supports features like 256-color palettes and UTF-8 encoding for optimal display of these tools.

There’s a huge amount of CLIs/TUIs out there, and I couldn’t list them all (though I tried to list some of the best). This selection represents a starting point for exploring the rich ecosystem of command-line tools available to developers. I encourage you to explore further, discover new tools that fit your specific needs, and contribute back to the community by sharing your findings.

Explore more articles on packagemain.tech

On the other hand, there were (and still are) more general-purpose editors, but they lacked in functionality when it came to more advanced language-specific features like code completion, “go to definition”, and so on. As an example, code highlighting was done using regular expressions.

With the growing number of code editors and programming languages available, this became the classic M*N complexity problem.

But then Microsoft introduced the Language Server Protocol (LSP) as a solution to the problem above, which elegantly transforms this M*N complexity into a more manageable M+N situation.

Table of Contents

• Why Microsoft Created the LSP

• What is the Language Server Protocol (LSP)?

• Language Server Features

• How Does LSP Work?

• Language Server for Go

• LSP Adoption

• Conclusion

• Resources

Why Microsoft Created the LSP

The LSP was initially driven by the needs of VS Code. VS Code had to work with hundreds of different Language Servers as part of its extensions. But there were multiple problems:

• Language mismatch: Language Servers are often written in different languages than the editor (like VS Code's Node.js).

• Performance impact: Language features can be resource-intensive, potentially slowing down the editor.

• Integration complexity: Multiple editors and multiple languages introduces the M*N complexity mentioned above.

Exactly because of that, Microsoft introduced LSP to standardize the communication between language tools and editors, allowing language servers to be written in any language, run separately for better performance, and easily integrate with any LSP-compliant editor. This simplifies language support for both tool providers and editor vendors.

You can find more info in this Language Server Extension Guide.

What is the Language Server Protocol (LSP)?

The LSP separates language servers from code editors (language clients). By making language servers independent processes dedicated to language understanding, the LSP enables any editor to utilize a standard language server. This means that a single standard language server can be used by all editors.

This interoperability is achieved through a defined set of standard messages and procedures that govern communication between language servers and editors. The LSP defines the format of the messages sent using JSON-RPC between the development tool and the language server.

Language Server Features

The list of features may vary for each individual language server, but usually they provide the following functionalities:

• Auto-completion

• Go to definition/declaration

• Find references

• Code formatting

• Diagnostics

• Documentation

And more.

For example, here you can see the list of editor features that gopls (the Go Language Server) provides.

And here you can see the full LSP specification for available features.

There are hundreds of Language Servers out there. Typically, every mature programming language (or markup language) has at least one Language Server. You can see the full list of language servers that implement LSP here.

How Does LSP Work?

The Language Server Protocol is built upon JSON-RPC. It specifically uses JSON RPC 2.0. You can think of JSON-RPC as a remote procedure call protocol that uses JSON for data encoding.

In a nutshell, it works like this. First, the editor establishes a connection with the language server. Then as the developer types code, the editor sends incremental changes to the language server. It then sends back insights like code completion suggestions and diagnostics.

Let’s see one real example for auto-completion. The request from the Language Client (editor) for this case will be:

{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "textDocument/completion",
  "params": {
    "textDocument": {
      "uri": "file:///home/alex/code/test/main.go"
    },
    "position": {
      "line": 35,
      "character": 21
    }
  }
}

As you can see, it sends the information about current cursor position and the buffer file. Let’s break it down:

• ID: The client sets this field to identify the request uniquely. Once the request is processed, it will return a response with the same request ID so that the client can match which response is for what request.

• Method: A string containing the name of the method to be invoked.

• Params: The parameters to be passed to the method. This can be structured as an array or an object.

Language server can access this file, analyze it, and respond with suggestions:

{
  "jsonrpc": "2.0",
  "id": 1,
  "result": {
    "isIncomplete": false,
    "items": [
      {
        "label": "Println",
        "kind": 3,
        "insertText": "Println(${1:format}, ${2:a ...interface{}})$0",
        "insertTextFormat": 2,
        "detail": "func Println(a ...interface{}) (n int, err error)",
        "documentation": "Println formats ..."
      },
      // ... other items
    ]
  }
}

Language Server for Go

The most popular and commonly used language server for Go is gopls. It is used by many editors, for example by the Visual Studio Code Go extension. Previously, there was another popular Language Server for Go by the Sourcegraph team called go-langserver, but this is no longer under active maintenance.

Many editors install gopls Language Server automatically if it’s not present on the host machine. But you can install it manually as well:

# check if its' installed and which version
gopls version
# golang.org/x/tools/gopls v0.16.2

# install or upgrade
go install golang.org/x/tools/gopls@latest

gopls also provides an experimental CLI interface:

gopls references ./main.go:35:8

LSP Adoption

The trend is definitely towards LSP adoption. Many editors that didn't initially support it are adding LSP capabilities due to its benefits.

But it's important to note that not all editors use LSP. Classic editors like Vi, Vim (in its basic configuration), and emacs (without specific plugins) traditionally rely on simpler methods for language support, such as syntax highlighting based on regular expressions.

Also, when your editor uses a Language Server, it can have a noticeable impact on CPU and memory, especially for large projects or complex languages. The good news is that you can choose a more efficient language server or disable them in your editor.

Here is how I can inspect what language servers my Zed editor is currently running:

ps aux | grep language-server

node yaml-language-server --stdio
node tailwindcss-language-server --stdio
...

Conclusion

Thanks to the Language Server Protocol, advanced coding capabilities are becoming universally accessible across various programming languages and coding environments.

It’s good to know how your code editors work, so it’s beneficial to have an understanding of this widely used technology called LSP. In short, LSP knowledge helps you understand and better utilize modern coding tools.

LSP is a win for both language providers and tooling vendors.

Resources

• Language Server Protocol

• gopls

• Discover more articles on packagemain.tech

In this article, I want to show another alternative in case you use GitHub Actions as your CI platform (the most popular CI/CD solution at the moment). This alternative is called Service Containers, and I’ve realized that not many developers seem to know about it.

In this hands-on tutorial, I’ll demonstrate how to create a GitHub Actions workflow for integration tests with external dependencies (MongoDB and Redis) using the demo Go application we created in that previous tutorial. We’ll also review the pros and cons of GitHub Service Containers.

Prerequisites

• A basic understanding of GitHub Actions workflows.

• Familiarity with Docker containers.

• Basic knowledge of Go toolchain.

Table of Contents

• What are Service Containers?

• Why not Docker Compose?

• Job Runtime

• Readiness Healthcheck

• Private Container Registries

• Sharing Data Between Services

• Golang Integration Tests

• Personal Experience & Limitations

• Conclusion

• Resources

What are Service Containers?

Service Containers are Docker containers that offer a simple and portable way to host dependencies like databases (MongoDB in our example), web services, or caching systems (Redis in our example) that your application needs within a workflow.

This article focuses on integration tests, but there are many other possible applications for service containers. For example, you can also use them to run supporting tools required by your workflow, such as code analysis tools, linters, or security scanners.

Why Not Docker Compose?

Sounds similar to services in Docker Compose, right? Well, that’s because it is.

But while you could technically use Docker Compose within a GitHub Actions workflow by installing Docker Compose and running docker-compose up, service containers provide a more integrated and streamlined approach that’s specifically designed for the GitHub Actions environment.

Also, while they are similar, they solve different problems and have different general purposes:

• Docker Compose is good when you need to manage a multi-container application on your local machine or a single server. It’s best suited for long-living environments.

• Service Containers are ephemeral and exist only for the duration of a workflow run, and they’re defined directly within your GitHub Actions workflow file.

Just keep in mind that the feature set of service containers (at least as of now) is more limited compared to Docker Compose, so be ready to discover some potential bottlenecks. We will cover some of them at the end of this article.

Job Runtime

You can run GitHub jobs directly on a runner machine or in a Docker container (by specifying the container property). The second option simplifies the access to your services by using labels you define in the services section.

To run directly on a runner machine:

.github/workflows/test.yaml

jobs:
  integration-tests:
    runs-on: ubuntu-24.04

    services:
      mongo:
        image: mongodb/mongodb-community-server:7.0-ubi8
        ports:
          - 27017:27017

    steps:
      - run: |
          echo "addr 127.0.0.1:27017"

Or you can run it in a container (Chainguard Go Image in our case):

jobs:
  integration-tests:
    runs-on: ubuntu-24.04
    container: cgr.dev/chainguard/go:latest

    services:
      mongo:
        image: mongodb/mongodb-community-server:7.0-ubi8
        ports:
          - 27017:27017
    steps:
      - run: |
          echo "addr mongo:27017"

You can also omit the host port, so the container port will be randomly assigned to a free port on the host. You can then access the port using the variable.

Benefits of omitting the host port:

• Avoids port conflicts – for example when you run many services on the same host.

• Enhances Portability – your configurations become less dependent on the specific host environment.

jobs:
  integration-tests:
    runs-on: ubuntu-24.04
    container: cgr.dev/chainguard/go:1.23

    services:
      mongo:
        image: mongodb/mongodb-community-server:7.0-ubi8
        ports:
          - 27017/tcp
    steps:
      - run: |
          echo "addr mongo:${{ job.services.mongo.ports['27017'] }}"

Of course, there are pros and cons to each approach.

Running in a container:

• Pros: Simplified network access (use labels as hostnames), and automatic port exposure within the container network. You also get better isolation/security as the job runs in an isolated environment.

• Cons: Implied overhead of containerization.

Running on the runner machine:

• Pros: Potentially less overhead than running the job inside a container.

• Cons: Requires manual port mapping for service container access (using localhost:). There’s also less isolation/security, as the job runs directly on the runner machine. This potentially affects other jobs or the runner itself if something goes wrong.

Readiness Healthcheck

Prior to running the integration tests that connect to your provisioned containers, you’ll often need to make sure that the services are ready. You can do this by specifying docker create options such as health-cmd.

This is very important – otherwise the services may not be ready when you start accessing them.

In the case of MongoDB and Redis, these will be the following:

    services:
      mongo:
        image: mongodb/mongodb-community-server:7.0-ubi8
        ports:
          - 27017/27017
        options: >-
          --health-cmd "echo 'db.runCommand("ping").ok' | mongosh mongodb://localhost:27017/test --quiet"
          --health-interval 5s
          --health-timeout 10s
          --health-retries 10

      redis:
        image: redis:7
        ports:
          - 6379:6379
        options: >-
          --health-cmd "redis-cli ping"
          --health-interval 5s
          --health-timeout 10s
          --health-retries 10

In the Action logs, you can see the readiness status:

Private Container Registries

In our example, we’re using public images from Dockerhub, but it’s possible to use private images from you private registries as well, such as Amazon Elastic Container Registry (ECR), Google Artifact Registry, and so on.

Make sure to store the credentials in Secrets and then reference them in the credentials section.

services:
  private_service:
    image: ghcr.io/org/service_repo
    credentials:
      username: ${{ secrets.registry_username }}
      password: ${{ secrets.registry_token }}

Sharing Data Between Services

You can use volumes to share data between services or other steps in a job. You can specify named Docker volumes, anonymous Docker volumes, or bind mounts on the host. But it’s not directly possible to mount the source code as a container volume. You can refer to this open discussion for more context.

To specify a volume, you specify the source and destination path: <source>:<destinationPath>

The <source> is a volume name or an absolute path on the host machine, and <destinationPath> is an absolute path in the container.

volumes:
  - /src/dir:/dst/dir

Volumes in Docker (and GitHub Actions using Docker) provide persistent data storage and sharing between containers or job steps, decoupling data from container images.

Project Setup

Before diving into the full source code, let's set up our project for running integration tests with GitHub Service Containers.

1. Create a new GitHub repository.

2. Initialize a Go module using go mod init

3. Create a simple Go application.

4. Add integration tests in integration_test.go

5. Create a .github/workflows directory.

6. Create a file named integration-tests.yaml inside the .github/workflows directory.

Golang Integration Tests

Now as we can provision our external dependencies, let’s have a look at how to run our integration tests in Go. We will do it in the steps section of our workflow file.

We will run our tests in a container which uses Chainguard Go image. This means we don’t have to install/setup Go. If you want to run your tests directly on a runner machine, you need to use the setup-go Action.

You can find the full source code with tests and this workflow here.

.github/workflows/integration-tests.yaml

name: "integration-tests"

on:
  workflow_dispatch:
  push:
    branches:
      - main

jobs:
  integration-tests:
    runs-on: ubuntu-24.04
    container: cgr.dev/chainguard/go:latest

    env:
      MONGO_URI: mongodb://mongo:27017
      REDIS_URI: redis://redis:6379

    services:
      mongo:
        image: mongodb/mongodb-community-server:7.0-ubi8
        ports:
          - 27017:27017
        options: >-
          --health-cmd "echo 'db.runCommand("ping").ok' | mongosh mongodb://localhost:27017/test --quiet"
          --health-interval 5s
          --health-timeout 10s
          --health-retries 10

      redis:
        image: redis:7
        ports:
          - 6379:6379
        options: >-
          --health-cmd "redis-cli ping"
          --health-interval 5s
          --health-timeout 10s
          --health-retries 10

    steps:
      - name: Check out repository code
        uses: actions/checkout@v4

      - name: Download dependencies
        run: go mod download

      - name: Run Integration Tests
        run: go test -tags=integration -timeout=120s -v ./...

To summarize what’s going on here:

1. We run our job in a container with Go (container)

2. We spin up two services: MongoDB and Redis (services)

3. We configure healthchecks to make sure our services are “Healthy” when we run the tests (options)

4. We perform a standard code checkout

5. Then we run the Go tests

Once the Action is completed (it took ~1 min for this example), all the services will be stopped and orphaned so we don’t need to worry about that.

Personal Experience & Limitations

We’ve been using service containers for running backend integration tests at BINARLY for some time, and they work great. But the initial workflow creation took some time and we encountered the following bottlenecks:

• It’s not possible to override or run custom commands in an action service container (as you would do in Docker Compose using the command property). Open pull request

  • Workaround: we had to find a solution that doesn’t require that. In our case, we were lucky and could do the same with environment variables.

• It’s not directly possible to mount the source code as a container volume. Open discussion

  • While this is indeed a big limitation, you can copy the code from your repository into your mounted directory after the service container has started.

Conclusion

GitHub service containers are a great option to scaffold an ephemeral testing environment by configuring it directly in your GitHub workflow. With configuration being somewhat similar to Docker Compose, it’s easy to run any containerised application and communication with it in your pipeline. This ensures that GitHub runners take care of shutting everything down upon completion.

If you use Github Actions, this approach works extremely well as it is specifically designed for the GitHub Actions environment.

Resources

• Source Code

• GitHub Documentation

• Discover more articles on packagemain.tech

Even though you can define the major edge cases, you still can’t predict how your program will behave in the case of some weird unexpected input. In other words, you can typically only find bugs you expect to find.

That’s where fuzz testing or fuzzing comes to the rescue. And in this tutorial, you’ll learn how to perform fuzz testing in Go.

Table of Contents

• What is Fuzz Testing?

• Fuzz Testing in Go

• Fuzzing HTTP Services

• Conclusion

• Resources

What is Fuzz Testing?

Fuzzing is an automated software testing technique that involves inputting a large amount of valid, nearly-valid, or invalid random data into a computer program and observing its behavior and output. So the goal of fuzzing is to reveal bugs, crashes, and security vulnerabilities in source code you might not find through traditional testing methods.

The Fuzz Testing in Go video which I made a few years ago shows a very simple example of the Go code that may work well unless you provide a certain input:

func Equal(a []byte, b []byte) bool {
  for i := range a {
    // can panic with runtime error: index out of range.
    if a[i] != b[i] {
      return false
    }
  }

  return true
}

This sample function works perfectly as long as the length of two slices are equal. But it will panic when the first slice is longer than the second (index out of range error). Furthermore, it doesn’t return a correct result when the second slice is the subset of the first one.

The fuzzing technique would easily spot this bug by bombarding this function with various inputs.

It’s a good practice to integrate fuzzing into your team’s software development lifecycle (SDLC) as well. For example, Microsoft uses fuzzing as one of the stages in its SDLC, to find potential bugs and vulnerabilities.

Fuzz Testing in Go

There are many fuzzing tools that have been available for a while – such as oss-fuzz, for example – but since Go 1.18, fuzzing was added to Go’s standard library. So it’s now part of the regular testing package since it’s a kind of test. You can also use it together with the other testing primitives which is nice.

The steps to create a fuzz test in Go are the following:

1. In a _test.go file, create a function that starts with Fuzz which accepts *testing.F

2. Add corpus seeds using f.Add() to allow the fuzzer to generate the data based on it.

3. Call the fuzz target using f.Fuzz() by passing fuzzing arguments which our target function accepts.

4. Start the fuzzer using the regular go test command, but with the –fuzz=Fuzz flag

Note that the fuzzing arguments can only be the following types:

• string, byte, []byte

• int, int8, int16, int32/rune, int64

• uint, uint8, uint16, uint32, uint64

• float32, float64

• bool

A simple fuzz test for the Equal function above may look like this:

// Fuzz test
func FuzzEqual(f *testing.F) {
  // Seed corpus addition
  f.Add([]byte{'f', 'u', 'z', 'z'}, []byte{'t', 'e', 's', 't'})

  // Fuzz target with fuzzing arguments
  f.Fuzz(func(t *testing.T, a []byte, b []byte) {
    // Call our target function and pass fuzzing arguments
    Equal(a, b)
  })
}

By default, fuzz tests run forever, so you either need to specify the time limit or wait for fuzz tests to fail. You can specify which tests to run using the --fuzz argument.

go test --fuzz=Fuzz -fuzztime=10s

If there are any errors during the execution, the output should look similar to this:

go test --fuzz=Fuzz -fuzztime=30s
--- FAIL: FuzzEqual (0.02s)
    --- FAIL: FuzzEqual (0.00s)
        testing.go:1591: panic: runtime error: index out of range
    Failing input written to testdata/fuzz/FuzzEqual/84ed65595ad05a58
    To re-run:
    go test -run=FuzzEqual/84ed65595ad05a58

Notice that the input for which the fuzz test has failed are written into a file in the testdata folder and can be re-played by using that input identifier.

go test -run=FuzzEqual/84ed65595ad05a58

The testdata folder can be checked into the repository and be used for regular tests, because fuzz tests can also act as regular tests when executed without the --fuzz flag.

Fuzzing HTTP Services

It’s also possible to fuzz test the HTTP services by writing a test for your HandlerFunc and using the httptest package. This can be very useful if you need to test the whole HTTP service, not only the underlying functions.

Let’s now introduce a more real example such as an HTTP Handler that accepts some user input in the request body and then write a fuzz test for it.

Our handler accepts a JSON request with limit and offset fields to paginate some static mocked data. Let's define the types first.

type Request struct {
  Limit  int `json:"limit"`
  Offset int `json:"offset"`
}

type Response struct {
  Results    []int `json:"items"`
  PagesCount int   `json:"pagesCount"`
}

Our handler function then parses the JSON, paginates the static slice, and returns a new JSON in response.

func ProcessRequest(w http.ResponseWriter, r *http.Request) {
 var req Request

  // Decode JSON request
  if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
    http.Error(w, err.Error(), http.StatusBadRequest)
    return
  }

 // Apply offset and limit to some static data
 all := make([]int, 1000)
 start := req.Offset
 end := req.Offset + req.Limit
 res := Response{
   Results:    all[start:end],
   PagesCount: len(all) / req.Limit,
 }

 // Send JSON response
 if err := json.NewEncoder(w).Encode(res); err != nil {
   http.Error(w, err.Error(), http.StatusInternalServerError)
   return
 }

 w.WriteHeader(http.StatusOK)
}

As you may have already noticed, this function doesn’t handle slice operations very well and can easily panic. Also, it can panic if it tries to divide by 0. It's great if we can spot this during the development or using only unit tests, but sometimes not everything is visible to our eye, and our handler may pass the input to other functions and so forth.

Following our FuzzEqual example above, let’s implement a fuzz test for the ProcessRequest handler. The first thing we need to do is to provide the sample inputs for the fuzzer. This is the data that the fuzzer will use and modify into new inputs that are tried. We can craft some sample JSON request and use f.Add() with the []byte type.

func FuzzProcessRequest(f *testing.F) {
  // Create sample inputs for the fuzzer
  testRequests := []Request{
    {Limit: -10, Offset: -10},
    {Limit: 0, Offset: 0},
    {Limit: 100, Offset: 100},
    {Limit: 200, Offset: 200},
  }

  // Add to the seed corpus
  for _, r := range testRequests {
    if data, err := json.Marshal(r); err == nil {
      f.Add(data)
    }
  }

  // ...
}

After that we can use the httptest package to create a test HTTP server and make requests to it.

Note: Since our fuzzer can generate invalid non-JSON requests, it’s better just to skip them and ignore with t.Skip(). We can also skip BadRequest errors.

func FuzzProcessRequest(f *testing.F) {
  // ...
  // Create a test server
  srv := httptest.NewServer(http.HandlerFunc(ProcessRequest))
  defer srv.Close()

  // Fuzz target with a single []byte argument
  f.Fuzz(func(t *testing.T, data []byte) {
    var req Request
    if err := json.Unmarshal(data, &req); err != nil {
      // Skip invalid JSON requests that may be generated during fuzz
      t.Skip("invalid json")
    }

    // Pass data to the server
    resp, err := http.DefaultClient.Post(srv.URL, "application/json", bytes.NewBuffer(data))
    if err != nil {
      t.Fatalf("unable to call server: %v, data: %s", err, string(data))
    }
    defer resp.Body.Close()

    // Skip BadRequest errors
    if resp.StatusCode == http.StatusBadRequest {
      t.Skip("invalid json")
    }

    // Check status code
    if resp.StatusCode != http.StatusOK {
      t.Fatalf("non-200 status code %d", resp.StatusCode)
    }
  })
}

Our fuzz target has a single argument with a type []byte that contains the full JSON request, but you can change it to have multiple arguments.

Everything is ready now to run our fuzz tests. When fuzzing HTTP servers, you may need to adjust the amount of parallel workers, otherwise the load may overwhelm the test server. You can do that by setting -parallel=1 flag.

go test --fuzz=Fuzz -fuzztime=10s -parallel=1
go test --fuzz=Fuzz -fuzztime=30s
--- FAIL: FuzzProcessRequest (0.02s)
    --- FAIL: FuzzProcessRequest (0.00s)
        runtime error: integer divide by zero
        runtime error: slice bounds out of range

And as expected, we will see the above errors uncovered.

We can also see the fuzz inputs in the testdata folder to see which JSON contributed to this failure. Here is a sample content of the file:

go test fuzz v1
[]byte("{"limit":0,"offset":0}")

To fix that issue, we can introduce input validation and default settings:

if req.Limit <= 0 {
  req.Limit = 1
}
if req.Offset < 0 {
  req.Offset = 0
}
if req.Offset > len(all) {
  start = len(all) - 1
}
if end > len(all) {
  end = len(all)
}

With this change, the fuzz tests will run for 10 seconds and exit without an error.

Conclusion

Writing fuzz tests for your HTTP services or any other methods is a great way to detect hard-to-find bugs. Fuzzers can detect hard-to-spot bugs that happen for only some weird unexpected input.

It’s amazing to see that fuzzing is a part of Go’s built-in testing library, making it easy to combine with regular tests. Note: prior to Go 1.18, developers used go-fuzz, which is a great tool for fuzzing as well.

Resources

• Source Code

• Fuzz Testing in Go

• Go Fuzzing

There are many public and private registries available to developers such as Docker Hub, Amazon ECR, and Google Cloud Artifact Registry. But sometimes, instead of relying on an external vendor, you might want to host your images yourself. This gives you more control over how the registry is configured and where the container images are hosted.

This article is a hands-on tutorial that’ll teach you how to self-host a Container Registry.

Table of Contents

• What is a Container Image?

• What is a Container Registry?

• Why you might want to self-host a Container Registry

• How to self-host a Container Registry

• Step 1: Install Docker and Docker Compose on the server

• Step 2: Configure and run the registry container

• Step 3: Run NGINX for handling TLS

• Ready to go!

• Other options

• Conclusion

You will get the most out of this article if you’re already familiar with the tools like Docker and NGINX, and have a general understanding of what a container is.

What is a Container Image?

Before we talk about container registries, let's first understand what a container image is. In a nutshell, a container image is a package that includes all of the files, libraries, and configurations to run a container. They are composed of layers where each layer represents a set of file system changes that add, remove, or modify files.

The most common way to create a container image is to use a Dockerfile.

# build an image
docker build -t pliutau/hello-world:v0 .

# check the images locally
docker images
# REPOSITORY    TAG       IMAGE ID       CREATED          SIZE
# hello-world   latest    9facd12bbcdd   22 seconds ago   11MB

This creates a container image that is stored on your local machine. But what if you want to share this image with others or use it on a different machine? This is where container registries come in.

What is a Container Registry?

A container registry is a storage catalog where you can push and pull container images from. The images are grouped into repositories, which are collections of related images with the same name. For example, on Docker Hub registry, nginx is the name of the repository that contains different versions of the NGINX images.

Some registries are public, meaning that the images hosted on them are accessible to anyone on the Internet. Public registries such as Docker Hub are a good option to host open-source projects.

On the other hand, private registries provide a way to incorporate security and privacy into enterprise container image storage, either hosted in cloud or on-premises. These private registries often come with advanced security features and technical support.

There is a growing list of private registries available such as Amazon ECR, GCP Artifact Registry, GitHub Container Registry, and Docker Hub also offers a private repository feature.

As a developer, you interact with a container registry when using the docker push and docker pull commands.

docker push docker.io/pliutau/hello-world:v0

# In case of Docker Hub we could also skip the registry part
docker push pliutau/hello-world:v0

Let's look at the anatomy of a container image URL:

docker pull docker.io/pliutau/hello-world:v0@sha256:dc11b2...
                |            |            |          |
                ↓            ↓            ↓          ↓
             registry    repository      tag       digest

Why You Might Want to Self-host a Container Registry

Sometimes, instead of relying on a provider like AWS or GCP, you might want to host your images yourself. This keeps your infrastructure internal and makes you less reliant on external vendors. In some heavily regulated industries, this is even a requirement.

A self-hosted registry runs on your own servers, giving you more control over how the registry is configured and where the container images are hosted. At the same time it comes with a cost of maintaining and securing the registry.

How to Self-host a Container Registry

There are several open-source container registry solutions available. The most popular one is officially supported by Docker, called registry, with its implementation for storing and distributing of container images and artifacts. This means that you can run your own registry inside a container.

Here are the main steps to run a registry on a server:

• Install Docker and Docker Compose on the server.

• Configure and run the registry container.

• Run NGINX for handling TLS and forwarding requests to the registry container.

• Setup SSL certificates and configure a domain.

Step 1: Install Docker and Docker Compose on the server

You can use any server that supports Docker. For example, you can use a DigitalOcean Droplet with Ubuntu. For this demo I used Google Cloud Compute to create a VM with Ubuntu.

neofetch

# OS: Ubuntu 20.04.6 LTS x86_64
# CPU: Intel Xeon (2) @ 2.200GHz
# Memory: 3908MiB

Once we're inside our VM, we should install Docker and Docker Compose. Docker Compose is optional, but it makes it easier to manage multi-container applications.

# install docker engine and docker-compose
sudo snap install docker

# verify the installation
docker --version
docker-compose --version

Step 2: Configure and run the registry container

Next we need to configure our registry container. The following compose.yaml file will create a registry container with a volume for storing the images and a volume for storing the password file.

services:
  registry:
    image: registry:latest
    environment:
      REGISTRY_AUTH: htpasswd
      REGISTRY_AUTH_HTPASSWD_REALM: Registry Realm
      REGISTRY_AUTH_HTPASSWD_PATH: /auth/registry.password
      REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY: /data
    volumes:
      # Mount the password file
      - ./registry/registry.password:/auth/registry.password
      # Mount the data directory
      - ./registry/data:/data
    ports:
      - 5000

The password file defined in REGISTRY_AUTH_HTPASSWD_PATH is used to authenticate users when they push or pull images from the registry. We should create a password file using the htpasswd command. We should also create a folder for storing the images.

mkdir -p ./registry/data

# install htpasswd
sudo apt install apache2-utils

# create a password file. username: busy, password: bee
htpasswd -Bbn busy bee > ./registry/registry.password

Now we can start the registry container. If you see this message, than everything is working as it should:

docker-compose up

# successfull run should output something like this:
# registry | level=info msg="listening on [::]:5000"

Step 3: Run NGINX for handling TLS

As mentioned earlier, we can use NGINX to handle TLS and forward requests to the registry container.

The Docker Registry requires a valid trusted SSL certificate to work. You can use something like Let's Encrypt or obtain it manually. Make sure you have a domain name pointing to your server (registry.pliutau.com in my case). For this demo I already obtained the certificates using certbot and put it in the ./nginx/certs directory.

Since we're running our Docker Registry in a container, we can run NGINX in a container as well by adding the following service to the compose.yaml file:

services:
  registry:
    # ...
  nginx:
    image: nginx:latest
    depends_on:
      - registry
    volumes:
      # mount the nginx configuration
      - ./nginx/nginx.conf:/etc/nginx/nginx.conf
      # mount the certificates obtained from Let's Encrypt
      - ./nginx/certs:/etc/nginx/certs
    ports:
      - "443:443"

Our nginx.conf file could look like this:

worker_processes auto;

events {
    worker_connections 1024;
}

http {
    upstream registry {
        server registry:5000;
    }

    server {
        server_name registry.pliutau.com;
        listen 443 ssl;

        ssl_certificate /etc/nginx/certs/fullchain.pem;
        ssl_certificate_key /etc/nginx/certs/privkey.pem;

        location / {
            # important setting for large images
            client_max_body_size                1000m;

            proxy_pass                          http://registry;
            proxy_set_header  Host              $http_host;
            proxy_set_header  X-Real-IP         $remote_addr;
            proxy_set_header  X-Forwarded-For   $proxy_add_x_forwarded_for;
            proxy_set_header  X-Forwarded-Proto $scheme;
            proxy_read_timeout                  900;
        }
    }
}

Ready to go!

After these steps we can run our registry and Nginx containers.

docker-compose up

Now, on the client side, you can push and pull the images from your registry. But first we need to login to the registry.

docker login registry.pliutau.com

# Username: busy
# Password: bee
# Login Succeeded

Time to build and push our image to our self-hosted registry:

docker build -t registry.pliutau.com/pliutau/hello-world:v0 .

docker push registry.pliutau.com/pliutau/hello-world:v0
# v0: digest: sha256:a56ea4... size: 738

On your server you can check the uploaded images in the data folder:

ls -la ./registry/data/docker/registry/v2/repositories/

Other options

Following the example above, you can also run the registry on Kubernetes. Or you could use a managed registry service like Harbor, which is an open-source registry that provides advanced security features and is compatible with Docker and Kubernetes.

Also, if you want to have a UI for your self-hosted registry, you could use a project like joxit/docker-registry-ui and run it in a separate container.

Conclusion

Self-hosted Container Registries allow you to have complete control over your registry and the way it's deployed. At the same time it comes with a cost of maintaining and securing the registry.

Whatever your reasons for running a self-hosted registry, you now know how it's done. From here you can compare the different options and choose the one that best fits your needs.

You can find the full source code for this demo on GitHub. Also, you can watch it as a video on our YouTube channel.

This article explores various approaches to handling database migrations in a Kubernetes environment, with a focus on Go tooling. You'll get the most out of this article if you already have some experience with Go, Kubernetes, and relational databases.

Table of Contents

• The challenge of migrations in Kubernetes

• Popular migration tools for Golang

• Run migrations inside the application

• Run migrations in initContainers

• Run migrations as a Kubernetes Job

• Helm hooks

• Best practices for Kubernetes migrations

• Conclusion

• Resources

The Challenge of Migrations in Kubernetes

Kubernetes introduces new challenges for database migrations:

• Multiple replicas starting simultaneously. These can span the same migration twice which may introduce some database locks.

• Separation of concerns between application and migration logic. This means it’s good to be able to run or rollback migrations without redeploying your application.

Popular Migration Tools for Golang

As I mentioned in another post, there are a few different tools you can use to manage your migrations. They are quite similar, so I personally don’t have a strong preference between once or another. I just wanted to provide a few options so you know what the popular tools are.

1. golang-migrate

• Widely used and supports numerous databases.

• Simple CLI and API.

• Supports various migration sources (local files, S3, Google Storage).

2. goose

• Supports main SQL databases.

• Allows migrations written in Go for complex scenarios.

• Flexible versioning schemas.

3. atlas

• Powerful database schema management tool.

• Supports declarative and versioned migrations.

• Offers integrity checks and migration linting.

• Provides GitHub Actions and Terraform provider.

Run Migrations Inside the Application

A naïve implementation would be to run the code of the migration directly inside your main function before you start your server.

Example using golang-migrate:

package main

import (
    "database/sql"
    "fmt"
    "log"
    "net/http"

    "github.com/golang-migrate/migrate/v4"
    "github.com/golang-migrate/migrate/v4/database/postgres"
    _ "github.com/golang-migrate/migrate/v4/source/file"
    _ "github.com/lib/pq"
)

func main() {
    // Database connection parameters
    url := "postgres://user:pass@localhost:5432/dbname"

    // Connect to the database
    db, err := sql.Open("postgres", url)
    if err != nil {
        log.Fatalf("could not connect to database: %v", err)
    }
    defer db.Close()

    // Run migrations
    if err := runMigrations(db); err != nil {
        log.Fatalf("could not run migrations: %v", err)
    }

    // Run the application, for example start the server
    if err := http.ListenAndServe(":8080", nil); err != nil {
        log.Fatalf("server failed to start: %v", err)
    }
}

func runMigrations(db *sql.DB) error {
    driver, err := postgres.WithInstance(db, &postgres.Config{})
    if err != nil {
        return fmt.Errorf("could not create database driver: %w", err)
    }

    m, err := migrate.NewWithDatabaseInstance(
        "file://migrations", // Path to your migration files
        "postgres",          // Database type
        driver,
    )
    if err != nil {
        return fmt.Errorf("could not create migrate instance: %w", err)
    }

    if err := m.Up(); err != nil && err != migrate.ErrNoChange {
        return fmt.Errorf("could not run migrations: %w", err)
    }

    log.Println("migrations completed successfully")
    return nil
}

However, these could cause different issues like your migrations being slow and Kubernetes considering that the pod didn’t start successfully and therefore killing it. You could run those migrations in a Go routine, but how do you handle failures then?

In cases when multiple pods are created at the same time, you would have a potential concurrency problem.

It also means your migrations need to be inside your Docker image.

Even with its downsides, this approach might work well for quick and stable database changes and small projects.

Run Migrations in initContainers

By using initContainers in your Kubernetes Deployment, it will run the migration before the main application container starts. This is a good first solution for when scaling is not a problem yet.

If the initContainer fails, the blue/green deployment from Kubernetes won’t go further and your previous pods stay where they are. This prevents having a newer version of the code without the planned migration.

Example:

initContainers:
  - name: migrations
    image: migrate/migrate:latest
    command: ['/migrate']
    args: ['-source', 'file:///migrations', '-database','postgres://user:pass@db:5432/dbname', 'up']

This approach might work well for quick and stable database changes for deployments with a single Pod. And it already separates the application and migration layers.

Run Migrations as a Kubernetes Job

You could create a Kubernetes Job that runs your migrations, and trigger that job during the deployment process before rolling out the application.

Example:

apiVersion: batch/v1
kind: Job
metadata:
  name: db-migrate
spec:
  template:
    spec:
      containers:
      - name: migrate
        image: your-migration-image:latest
        command: ['/app/migrate']

You can also combine it with initContainers, making sure that the pod starts only when the job is successful.

initContainers:
  - name: migrations-wait
    image: ghcr.io/groundnuty/k8s-wait-for:v2.0
    args:
      - "job"
      - "my-migration-job"

This approach can solve the problems related to multiple replicas mentioned above.

Helm Hooks

If you use Helm, it has hooks that you can use for running migrations during chart installation/upgrade. You just define a pre-install or pre-upgrade hook in your Helm chart.

pre-install-hook.yaml:

apiVersion: batch/v1
kind: Job
metadata:
  name: {{ include "mychart.fullname" . }}-migrations
  annotations:
    "helm.sh/hook": pre-install,pre-upgrade
    "helm.sh/hook-weight": "-5"
    "helm.sh/hook-delete-policy": hook-succeeded
spec:
  template:
    spec:
      containers:
        - name: migrations
          image: your-migrations-image:tag
          command: ["./run-migrations.sh"]

In this example, the pre-install hook executes after templates are rendered, but before any resources are created in Kubernetes.

This of course works only when you use Helm, meaning you need to find something else if you decide not to use Helm.

Best Practices for Kubernetes Migrations

Decouple migrations from application code:

1. Create a separate Docker image for migrations. This ensures that migration logic is encapsulated and doesn't interfere with the application codebase.

2. Use tools like Atlas to manage migrations independently. Tools like Atlas provide features for automating migration processes, scheduling, and rollback.

Use version control for migrations:

1. Store migration files in your Git repository. This ensures a complete history of migration changes, making it easier to track and revert changes.

2. Use sequential or timestamp-based versioning. Sequential versioning guarantees the correct order of migrations which is very important for relational databases.

Ensure idempotent migrations:

1. Ensure migrations can be run multiple times without side effects. Idempotent migrations prevent accidental data corruption or inconsistencies if a migration is run multiple times.

Have a rollback strategy

1. Implement and test rollback procedures for each migration. Having a rollback strategy ensures that you can revert changes if a migration fails or causes unexpected issues.

Perform monitoring and logging

1. Use tools like Atlas Cloud for visibility into migration history. Atlas Cloud provides detailed logs and history of migrations, making it easy to track changes and troubleshoot issues.

Conclusion

Managing database migrations in a Kubernetes environment requires careful planning and execution.

By leveraging tools like golang-migrate, goose, or atlas, and following best practices, you can create robust, scalable, and maintainable migration strategies.

Remember to decouple migrations from application code, use version control, and implement proper monitoring to ensure smooth database evolution in your Kubernetes-based architecture.

Resources

• Discover more articles from packagemain.tech

• Helm Hooks

Go is a language with quite a diverse standard library, which includes the well-known database/sql package. And it has its own libraries and solutions for working with SQL, that suit different needs, preferences, and teams.

In this article, we'll explore and compare the most popularly used Go packages that let you work with SQL. We’ll look at some hands-on examples, as well as the pros and cons. We’ll also briefly touch on the topic of database migrations and how to manage them in Go.

You'll get the most out of this article if you already have some experience with Go, SQL, and relational databases (doesn't matter which one).

Table of Contents

• Demo Schema

• Raw SQL and database/sql

• Raw SQL and sqlx

• ORMs

• Generated Go code from SQL using sqlc

• Database Migrations

• Conclusion

• Resources

Demo Schema

For this article, we'll use a simple schema with three tables: users, posts, and blogs. For simplicity, we'll be using SQLite as our database engine. But if you want to choose another database engine, that shouldn’t be a problem, as all the libraries we'll be exploring support multiple SQL dialects.

Here is our database schema in SQL:

CREATE TABLE users (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    name TEXT NOT NULL,
    email TEXT NOT NULL UNIQUE
);

CREATE TABLE blogs (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    name TEXT NOT NULL,
    url TEXT NOT NULL UNIQUE
);

CREATE TABLE posts (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    title TEXT NOT NULL,
    content TEXT NOT NULL,
    user_id INTEGER NOT NULL,
    blog_id INTEGER NOT NULL,
    FOREIGN KEY (user_id) REFERENCES users (id) ON DELETE CASCADE,
    FOREIGN KEY (blog_id) REFERENCES blogs (id) ON DELETE CASCADE
);

And here is its Entity-Relationship Diagram (ERD):

Raw SQL and database/sql

Let’s imagine your application needs to perform the following action:

Find the users who have posted at least two articles, along with the total number of posts they've made.

In pure SQL, you could translate that into the following query:

SELECT u.name, COUNT(p.id) AS post_count
FROM users AS u
JOIN posts AS p ON u.id = p.user_id
GROUP BY u.id
HAVING post_count >= 2;

A brief explanation of this query: we JOIN the users and posts tables, then GROUP by user_id. The HAVING clause filters the results to only include users who have posted at least 2 posts, and COUNT aggregates the amount of posts.

As mentioned above, Go provides a built-in package called database/sql with the necessary tools for working with SQL databases. It was developed with simplicity in mind, but supports all the necessary functionality such as transactions, parameterized queries, connection pool management, and so on.

As long as you’re comfortable writing your own queries and handling errors and results, it’s a great option. And some would say that it’s the best option, as there is no hidden logic and you can always copy the query and analyze it with EXPLAIN.

Here is how you can get the results of the query above in Go code using database/sql (some parts like connection are omitted):

type userStats struct {
  UserName  sql.NullString
  PostCount sql.NullInt64
}

func getUsersStats(conn *sql.DB, minPosts int) ([]userStats, error) {
  query := `SELECT u.name, COUNT(p.id) AS post_count
FROM users AS u
JOIN posts AS p ON u.id = p.user_id
GROUP BY u.id
HAVING post_count >= ?;`

  rows, err := conn.Query(query, minPosts)
  if err != nil {
    return nil, err
  }
  defer rows.Close()

  users := []userStats{}
  for rows.Next() {
    var user userStats

    if err := rows.Scan(&user.UserName, &user.PostCount); err != nil {
      return nil, err
    }

    users = append(users, user)
  }

  if err := rows.Err(); err != nil {
    return nil, err
  }

  return users, nil
}

In this code we:

• Use the raw SQL query with an unnamed parameter, and pass the value of this parameter in conn.Query()

• Iterate over returned rows and manually scan each row into a struct userStats defined above. Note that the struct uses sql.Null* types to handle nullable values properly.

• We need to manually check for possible errors and close the rows to release the resources.

Pros:

• No additional abstraction/complexity added. Easy to debug raw SQL queries.

• Performance. The database/sql package is quite performant.

• Provides a good enough abstraction from different database backends.

Cons:

• The code becomes a bit verbose as there is a need to scan each row, define proper types, and handle errors.

• No compile-time type safety.

You can find the full source for this article in this Github Repository.

Raw SQL and sqlx

Now let’s have a look at some external packages which are popular in the Go community.

If you’re already familiar with database/sql and like its simplicity, you may enjoy working with sqlx. It’s built on top of the standard library and just extends its features.

It’s very easy to integrate existing codebases using database/sql with sqlx, because it leaves the underlying interfaces such as sql.DB, sql.Tx, and so on untouched.

The core features of sqlx are:

• Named parameters.

• Easier row scanning into structs with embedded struct support.

• Better separation between single and multiple rows by using the Get() and Select() methods.

• Ability to bind a slice of values as a single parameter to an IN query.

Here is how you can get the results of the query above using sqlx:

type userStats struct {
  UserName  string `db:"name"`
  PostCount string `db:"post_count"`
}

func getUsersStats(conn *sqlx.DB, minPosts int) ([]userStats, error) {
  users := []userStats{}

  query := `SELECT u.name, COUNT(p.id) AS post_count
FROM users AS u
JOIN posts AS p ON u.id = p.user_id
GROUP BY u.id
HAVING post_count >= ?;`

  if err := conn.Select(&users, query, minPosts); err != nil {
    return nil, err
  }

  return users, nil
}

In this code, we use the Select() method which handles the scanning of the rows. It also closes the rows automatically so we don’t have to deal with that.

The code is much shorter than the database/sql version, but it can hide some implementation details from us. For example, be aware that Select loads the whole set into memory at once.

Pros:

• Not that different from database/sql. Still easy to debug raw SQL queries.

• A bunch of great features to reduce code verbosity.

Cons:

• Same as database/sql

ORMs

Object-relational mapping (ORM) is a technique (some call it a design pattern) of accessing a relational database by working with objects without having to craft complex SQL statements. It’s very popular in object-oriented languages – Ruby on Rails has its Active Record, Python has SQLAlchemy, Typescript has Drizzle, and so on.

And Go has GORM. In a nutshell, it lets you write queries as Go code by calling various methods on objects, which are then translated into SQL queries. But not only that, it has other features like database migrations, database resolvers, and more.

You may need to spend a bit of time initially setting up your GORM models, but later it can reduce a lot of boilerplate code.

Our simple schema and query example may not be the best to visualize the strengths and weaknesses of GORM, but should be enough to demonstrate how we can run a similar query and scan the results:

type User struct {
  gorm.Model
  ID    int
  Name  string
  Posts []Post
}

type Post struct {
  gorm.Model
  ID     int
  UserID int
}

type userStats struct {
  Name  string
  Count int `gorm:"column:post_count"`
}

func getUsersStats(conn *gorm.DB, minPosts int) ([]userStats, error) {
  var users []userStats

  err := conn.Model(&User{}).
    Select("name", "COUNT(p.id) AS post_count").
    Joins("JOIN posts AS p ON users.id = p.user_id").
    Group("users.id").
    Having("post_count >= ?", minPosts).
    Find(&users).Error

  return users, err
}

The SQL query generated by gorm will be roughly the same as the one we wrote manually in the database/sql variant.

To summarize the code above:

• We declared our User and Post models and extended it with the default gorm.Model struct. Later we can use these two models to build any queries we want by using gorm methods.

• We also defined our small result type userStats

• We used methods such as Select(), Joins(), Group(), and Having() to produce the query we want.

With such an easy example, it’s hard to see the potential issues – everything looks just right. But when your project becomes more complex, you will most definitely encounter some issues with that. Just look at the StackOverflow questions marked with go-gorm.

It's good to be careful about using ORMs in performance-critical systems or where you need direct control over database interactions. This is because gorm uses a lot of reflection, and can add overhead and sometimes obscure what's happening at the database level. Any project where the functionality is wrapped in another huge layer runs the risk of increasing the overall complexity.

Pros:

• Abstraction from different database backends.

• Big feature set: migrations, hooks, database resolvers, and more.

• Saves quite a bit of tedious coding.

Cons:

• Another layer of complexity and overhead. Hard to debug raw SQL queries.

• Performance drawbacks. May not be as efficient for some critical applications.

• Initial setup can require some time to configure all the models.

Generated Go Code from SQL using sqlc

This nicely brings us to another unique approach of generating Go code from SQL queries using sqlc. With sqlc, you write your schema and SQL queries, then use a CLI tool to generate Go code from it and then use the generated code to interact with databases.

This ensures that your queries are syntactically correct and type-safe. It’s ideal for those who prefer writing SQL but are looking for an efficient way to integrate it into a Go application.

sqlc needs to know your database schema and queries in order to generate code, so it requires some initial setup. We can add our schema and query above to the files schema.sql and query.sql. Then using the following config, we can generate the Go code:

version: "2"
sql:
  - engine: "sqlite"
    queries: "query.sql"
    schema: "schema.sql"
    gen:
      go:
        package: "main"
        out: "."

We also need to name our query in query.sql and mark the parameters:

-- name: GetUsersStats :many
SELECT u.name, COUNT(p.id) AS post_count
FROM users AS u
JOIN posts AS p ON u.id = p.user_id
GROUP BY u.id
HAVING post_count >= ?;

After we run sqlc generate, we can use the following generated types and functions which make our code type-safe and quite short.

func getUsersStats(conn *sql.DB, minPosts int) ([]GetUsersStatsRow, error) {
  queries := New(conn)

  ctx := context.Background()
  return queries.GetUsersStats(ctx, minPosts)
}

What makes sqlc special is that it understands your database schema, and uses that to validate the SQL you write. So your SQL queries are being validated against the actual database table, and sqlc will give you a compile-time error if something is wrong.

Pros:

• Type safety with generated Go code.

• Still easy to debug SQL code.

• Saves quite a bit of tedious coding.

• Performance.

Cons:

• Initial configuration setup for database schema and queries.

• Not perfect static analysis. Sometimes you need to explicitly set the parameter type, and so on.

If you’re good with SQL statements and prefer not to use much code to express your database interactions, this is your package.

Database Migrations

Since we’re on the topic of SQL databases here, let’s briefly review how database migrations work in Go. The schema of the database almost always evolves over time and no one wants to do these changes manually. So there are tools developed to help with that.

The main goal of database migration tools is to ensure that all environments have the same schema and developers can easily apply the changes or roll them back.

I mentioned above that GORM can do the migrations as well if your project uses it as its ORM. If you use database/sql, sqlx or sqlc you’ll have to use separate projects to manage them.

The most popular projects are:

• golang-migrate: one of the most famous tools for handling database migrations. It supports many database drivers and migration sources, and takes a simple and direct approach for handling database migrations.

• goose: another solid option when choosing a migration tool. It also has support for the main database drivers. Two of its main features are support for migrations written in Go and more control of the migration application process.

You can then integrate these tools directly into your application or in CI/CD. But running them properly in CI/CD requires some setup (for example in case of deploying to Kubernetes), and I’ll dive deeper into that in my upcoming articles.

Conclusion

There are many well-written, tested, and supported database packages for Go that can help you with faster development and writing cleaner code. There is also the very powerful database/sql included in the standard library that can do most of your daily work.

But whether you should use it or not depends on your needs as a developer, your preferences, and your project. In this article, I tried to highlight the strengths and weaknesses of each option.

You can find the full source for this article on this Github Repository.

I’ll end this article with this famous meme:

Resources

• database/sql

• sqlx

• GORM

• sqlc

• Discover more articles from packagemain.tech

And though it seems easy, there are some nuances to it and multiple ways of compiling the same program which results in different executables.

In this article, we’ll explore statically and dynamically linked executables, internal and external linkers, and examine binaries using tools like file, ld, and ldd.

Here's what we'll cover:

• Overview

• What is Static and Dynamic linking?

• Statically Linked Program

• What is a binary anyway?

• Dynamically Linked Program

• Can we make it statically linked?

• Internal vs External linker

• Cross-Compilation

• Bonus Point: Reduce binary size

• Beware: LD_PRELOAD trick

• Conclusion

• Further Reads

What is Static and Dynamic linking?

Static linking is the practice of copying all the libraries your program needs directly into the final executable file image.

And Go loves and wants that whenever it’s possible. This is because it's more portable, as it doesn’t require the presence of the library on the host system where it runs. So your binary can run on any system no matter which distro/version, and it won't depend on any system libraries.

Dynamic linking, on the other hand, is when external or shared libraries are copied into the executable file by name during run time.

And it has its own advantages, too. For example the program can re-use popular libc libraries that are available on the host system and not re-implement them. You can also benefit from host updates without re-linking your program. It can also reduce the executable file size in many cases.

Statically Linked Program

Let’s review a program that will always get statically linked. This program doesn’t call C code using cgo, so everything can be packaged in a static binary. Our program only prints a simple message to stdout, which Go can do internally without needing to use something from libc.

package main

import "fmt"

func main() {
    fmt.Println("hi, user")
}

What is a Binary Anyway?

We can use a file program to examine the file type first.

$ go build main1.go

$ file main1 | tr , '\n'
main1: ELF 64-bit LSB executable
 ARM aarch64
 version 1 (SYSV)
 statically linked
 Go BuildID=...
 with debug_info
 not stripped

It tells us that it’s an ELF (Executable and Linkable Format) executable file. It also tells us that it’s “statically linked“.

We won’t dive into what ELF is, but there are other executable file formats. ELF is the default one on Linux, Mach-O is the default one for macOS, PE/PE32+ for Windows, and so on.

Note: in this article we’ll be working with Linux (Ubuntu) and its tooling, but the same is possible on other platforms.

And there is another Linux program called ldd that can tell us if the binary is statically or dynamically linked.

$ ldd main1
not a dynamic executable

Dynamically Linked Program

As mentioned above, Go has a mechanism called cgo to call C code from Go. Even Go’s stdlib uses it in multiple places – for example in the net package, where it uses the standard C library to work with DNS.

Importing such packages or using cgo in your code by default produces a dynamically-linked binary, linked to those libc libraries.

package main

import (
    "fmt"
    "log"
    "net"
)

func main() {
    ipv4Addr, ipv4Net, err := net.ParseCIDR("192.0.2.1/24")
    if err != nil {
        log.Fatal(err)
    }
    fmt.Println(ipv4Addr)
    fmt.Println(ipv4Net)
}

We can use our file and ldd programs again to examine the second binary.

$ go build main2.go

$ file main2 | tr , '\n'
main2: ELF 64-bit LSB executable
 ARM aarch64
 version 1 (SYSV)
 dynamically linked
 interpreter /lib/ld-linux-aarch64.so.1
 Go BuildID=...
 with debug_info
 not stripped

$ ldd main2
    linux-vdso.so.1 (0x0000ffff87c81000)
    libc.so.6 => /lib/aarch64-linux-gnu/libc.so.6 (0x0000ffff87a80000)
    /lib/ld-linux-aarch64.so.1 (0x0000ffff87c44000)

The file program now shows us that it is a dynamically liked binary and ldd shows us the dynamic dependencies of our binary. In this case it relies on libc.so.6 and ld-linux which is a dynamic linker for Linux systems.

Can We Make it Statically Linked?

There are multiple reasons why you might want your binaries to be static, but the main one is to make deployment and distribution easier. But! It’s not always necessary, and by linking libc you benefit from host updates. Also, in case of our net package, you use those complex DNS lookup functions included in libc.

What’s interesting is that Go’s net package also has a pure-Go version, which makes it possible to disable cgo during compile time. You can do it by specifying build tags or by fully disabling cgo using CGO_ENABLED=0.

$ go build -tags netgo main2.go
$ ldd main2
not a dynamic executable

$ CGO_ENABLED=0 go build main2.go
$ ldd main2
not a dynamic executable

The above proves that we end up with a static binary in both cases.

Internal vs External Linker

Linker is a program that reads the Go archive or object for a package main, along with its dependencies, and combines them into an executable binary.

By default, Go’s toolchain uses its internal linker (go tool link), but you can specify which linker to use during the compilation time. This can give you a combination of benefits of a static binary as well as full-fledged libc capabilities.

On Linux, the default linker is gcc’s ld. And we can tell it to produce a static binary.

$ go build -ldflags "-linkmode 'external' -extldflags '-static'" main2.go
# command-line-arguments
/usr/bin/ld: /tmp/go-link-629224677/000004.o: in function `_cgo_97ab22c4dc7b_C2func_getaddrinfo':
/tmp/go-build/cgo_unix_cgo.cgo2.c:60:(.text+0x30):
warning: Using 'getaddrinfo' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
$ ldd main2
not a dynamic executable

It works, but we have a warning here. In our case glibc uses libnss to support a number of different providers for address resolution services and you cannot statically link libnss.

Other cgo packages may produce similar warnings and you’ll have to check the documentation to see if they’re critical or not.

Cross-Compilation

As mentioned in the introduction, cross-compilation is a very nice feature of Go. It lets you compile your program for almost any platform/architecture. But it can be very tricky if your program uses cgo, because it’s generally tricky to cross-compile C code.

$ CGO_ENABLED=0 GOOS=darwin GOARCH=arm64 go build main2.go
$ CGO_ENABLED=1 GOOS=darwin GOARCH=arm64 go build main2.go
# runtime/cgo
cgo: C compiler "clang" not found: exec: "clang":
executable file not found in $PATH

You can overcome that by installing the toolchain for the target OS and/or architecture.

If you can, it’s always better to just not use cgo for cross-compilation. You’ll get stable binaries which are statically linked.

Bonus Point: Reduce Binary Size

As you may notice, the output of the file command above had the following: “with debug_info not stripped“. This means that our binary has debugging information in it. But we usually don’t need it, and removing it may reduce the binary size.

$ go build main1.go
$ du -sh main1
1.9M    main1

$ go build -ldflags="-w -s" main1.go
$ du -sh main1
1.3M    main1

$ file main1 | tr , '\n'
main1: ELF 64-bit LSB executable
 ARM aarch64
 version 1 (SYSV)
 statically linked
 Go BuildID=...
 stripped

Beware: LD_PRELOAD Trick

The Linux system program ld-linux.so (dynamic linker/loader) uses LD_PRELOAD to load specified shared libraries. In particular, before any other library, the dynamic loader will first load shared libraries that are in LD_PRELOAD.

The LD_PRELOAD trick is a powerful technique used in dynamically linked binaries to override or intercept function calls to shared libraries.

By setting the LD_PRELOAD environment variable to point to a custom shared object file, users can inject their own code into a program's execution, effectively replacing or augmenting existing library functions.

This method allows for various applications, such as debugging, testing, and even modifying program behaviour without altering the original source code.

LD_PRELOAD=/path/to/my/malloc.so /bin/ls

It also shows that statically linked binaries are more secure, as they don’t have this issue since they don’t seek any external libraries. Also, there is a “secure-execution mode” – a security feature implemented by the dynamic linker on Linux systems to restrict certain behaviours when running programs that require elevated privileges.

Conclusion

Computers are not magic, you just have to understand them.

And understanding Go compilation and execution processes is crucial for developing robust cross-platform applications.

Hopefully, after reading this article, you now have a better understanding of how Go compilation works.

Further Reads

• Explore more articles from packagemain.tech

• Source Code

• src/cmd/cgo/doc.go

• cmd/link

• Debugging a weird 'file not found' error

• How the heck do we get to main()

• A General Overview of What Happens Before main()

• Rust Before Main

In this article, we'll explore how to implement SSE in Go, discussing its benefits, use cases, and providing practical examples. By the end, you should know the basics of building real-time applications with efficient, unidirectional communication.

What are Server-Sent Events?

SSE is a web technology that allows servers to push data to clients over a single HTTP connection.

Unlike WebSockets, SSE is unidirectional, making it simpler to implement and ideal for scenarios where real-time updates from the server are required, but client-to-server communication is not necessary.

Developing a web application that uses SSE is straightforward. You'll need a bit of code on the server to stream events to the front-end, but the client side code works almost identically to websockets when it comes to handling incoming events. This is a one-way connection, so you can't send events from a client to a server.

Benefits of SSE

1. Simplicity: SSE is easier to implement compared to WebSockets.

2. Native browser support: Most modern browsers support SSE out of the box.

3. Automatic reconnection: Clients automatically attempt to reconnect if the connection is lost.

4. Efficient: Uses a single HTTP connection, reducing overhead.

How to Implement SSE in Go

For our example here, we'll create a simple SSE server in Go which just sends the data to the client every second with a current timestamp. The client can then connect to our server on port 8080 and receive these messages.

A real example could be something more sophisticated like sending notifications, displaying progress bar updates, and so on.

package main

import (
    "fmt"
    "net/http"
    "time"
)

func sseHandler(w http.ResponseWriter, r *http.Request) {
    // Set http headers required for SSE
    w.Header().Set("Content-Type", "text/event-stream")
    w.Header().Set("Cache-Control", "no-cache")
    w.Header().Set("Connection", "keep-alive")

    // You may need this locally for CORS requests
    w.Header().Set("Access-Control-Allow-Origin", "*")

    // Create a channel for client disconnection
    clientGone := r.Context().Done()

    rc := http.NewResponseController(w)
    t := time.NewTicker(time.Second)
    defer t.Stop()
    for {
        select {
        case <-clientGone:
            fmt.Println("Client disconnected")
            return
        case <-t.C:
            // Send an event to the client
            // Here we send only the "data" field, but there are few others
            _, err := fmt.Fprintf(w, "data: The time is %s\n\n", time.Now().Format(time.UnixDate))
            if err != nil {
                return
            }
            err = rc.Flush()
            if err != nil {
                return
            }
        }
    }
}

func main() {
    http.HandleFunc("/events", sseHandler)
    fmt.Println("server is running on :8080")
    if err := http.ListenAndServe(":8080", nil); err != nil {
        fmt.Println(err.Error())
    }
}

Key Components of the SSE Implementation

The event stream is a simple stream of text data which must be encoded using UTF-8. Messages in the event stream are separated by a pair of newline characters – \n\n. A colon as the first character of a line is in essence a comment, and is ignored.

In our server it is done here:

rc := http.NewResponseController(w)
fmt.Fprintf(w, "data: The time is %s\n\n", time.Now().Format(time.UnixDate))
// To make sure that the data is sent immediately
rc.Flush()

The server that sends events needs to respond using the MIME type text/event-stream. We do it by setting the response header here:

w.Header().Set("Content-Type", "text/event-stream")

You may have noticed that we set few other headers as well. One is to keep the HTTP connection open, and another to bypass CORS:

w.Header().Set("Cache-Control", "no-cache")
w.Header().Set("Connection", "keep-alive")
w.Header().Set("Access-Control-Allow-Origin", "*")

And the last important piece is to detect the disconnect. In Go, we'll receive it as a message in a specified channel:

clientGone := r.Context().Done()

for {
    select {
    case <-clientGone:
        fmt.Println("Client disconnected")
        return
    }
}

Each message received has some combination of the following fields, one per line. In our server we send only the data field which is enough, as other fields are optional. More details here.

• event – a string identifying the type of event described.

• data – the data field for the message.

• id – the event ID to set the EventSource object's last event ID value.

• retry – the reconnection time.

How to Receive the Events on the Client Side

On the front end or client side, you will have to use the EventSource interface. It's a browser API encapsulating Server-Sent Events. In the following example, our browser application receives the events from the server and prints them in a list.

<!doctype html>
<html>
    <body>
        <ul id="list"></ul>
    </body>

    <script type="text/javascript">
        const eventSrc = new EventSource("http://127.0.0.1:8080/events");

        const list = document.getElementById("list");

        eventSrc.onmessage = (event) => {
            const li = document.createElement("li");
            li.textContent = `message: ${event.data}`;

            list.appendChild(li);
        };
    </script>
</html>

Here is how it may look in your browser:

Best Practices for SSE in Golang

Event Formatting

In a real world project, a simple string of data may not be enough. In these cases, using a structured format like JSON can be a good option to send multiple data fields once. Here's an example:

{
  "status": "in_progress",
  "completion": 51.22
}

Reconnection Strategy and Error Handling

Something could always go wrong on both sides: the server might reject the connection for some reason or a client might abruptly disconnect.

In each case, you'll need to implement a backoff strategy for graceful reconnections. It's better to miss one message than completely break the event loop.

In JavaScript, you can check for errors in EventSource and then act accordingly:

eventSrc.onerror = (err) => {
  console.error("EventSource failed:", err);
};

Load Balancing

For high-traffic applications, you may consider using a Load Balancer, for example NGINX. If you plan to have many clients connecting to your server, it's good to test it beforehand by simulating the load from the clients.

Use Cases for SSE

1. Real-time dashboards

2. Live sports scores

3. Social media feeds

4. Stock market tickers

5. Progress indicators for long-running tasks

Conclusion

Server-Sent Events provide an efficient and straightforward way to implement real-time, server-to-client communication in Golang applications. By leveraging SSE, developers can create responsive and dynamic web applications with minimal overhead and complexity.

As you build your SSE-powered applications, remember to consider scalability, error handling, and client-side implementation to ensure a robust and efficient real-time communication system.

Explore more articles from packagemain.tech

The challenge is that architecture diagrams – if they even exist – are often outdated relics from a bygone era.

This is why creating and maintaining effective and clear diagrams should be effortless. Up-to-date visuals ensure everyone stays on the same page, eliminating confusion and wasted time.

Table of Contents

• What is the C4 Model?

• Level 1: Context

• Level 2: Containers

• Level 3: Components

• Level 4: Code

• Supplementary Diagrams

• Diagrams as Code

• Automate Rendering in Your CI

• Resources

What is the C4 Model?

The C4 model was created as a way to help software development teams describe and communicate software architecture.

C4 stands for “Context, Containers, Components, and Code”. Those are the four levels that should be enough to describe a complex system.

The best way to explain the concept is to think about how we use Google Maps. When we are exploring an area in Google Maps, we will often start zoomed out to help us get context. Once we find the rough area we are interested in we can zoom in to get a little more detail.

Level 1: Context

This level is the most zoomed out. It's a bird’s eye view of the system in the greater context of the world. The diagram concentrates on actors and systems.

For the examples below, we will use a simple Task Management Software System to demonstrate all these 4 levels.

This diagram portrays the Task Management Software System's interactions with external systems and the different user groups that utilize it. We can see that the Task Management software relies on two external systems: Email and Calendar, and two types of actors (users) use it: Customer and Admin User.

Level 2: Containers

The containers level is a more detailed view of your system (don’t confuse C4 containers with Docker containers).

It reveals how various functional units like applications and databases work together and distribute responsibilities.

This diagram also highlights the key technologies employed and showcases the communication flow between these containers. It presents a simplified, technology-centric view of the system's core components and their interactions.

If you have Microservice architecture, then each Microservice would be a container.

Examples of containers are:

• Single page application

• Web server

• Serverless function

• Database

• API

• Message buses

And so on.

This level delves into the internal composition of the Task Management Software System. It showcases that our Task Management software system consists of containers such as User Web UI, Admin Web UI, API and a Database. API is also the container that is connected to external systems, for example to send emails or create events in calendar.

Level 3: Components

The next level of zoom is components. This shows the major structural building blocks of your application, and is often a conceptual view of the application. The term component is loose here. It could represent a controller or a service containing business logic.

This diagram focuses on the internal structure of the API container within the Task Management Software System. It reveals that the API container houses crucial functionalities like CRUD operations (Create, Read, Update, Delete) for data manipulation and user authentication mechanisms. The CRUD components is the one that talks to the database.

Level 4: Code

The deepest level of zoom is the code diagram. Although this diagram exists, it is often not used as the code paints a very similar picture. However, in highly regulated environments and complex legacy projects this level can help to paint a better picture of inner intricacies of the software.

Supplementary Diagrams

Besides the 4 diagrams above, there are a couple more worth mentioning:

• Deployment diagram

• Dynamic diagram: to describe the process or a flow

On this diagram we show a Login Flow, which is not a container or component, but rather a software process that happens in our software system. It shows that Web/Admin UIs use JWT-based authentication for communication with the API and the JWT token is stored in local storage on a client side.

Diagrams as Code

The power of C4 comes with a diagram-as-code approach. This means treating your diagrams just like your codebase:

• Version control: Store them in a source control system (like Git) for easy tracking and collaboration.

• Collaboration: Work together on diagrams using pull requests, similar to code reviews.

• Automation: Integrate them into your build pipelines for automatic rendering with your preferred tools.

Helpful Tool: Structurizr

There are few tools to help with modeling and diagramming, but the most popular nowadays is Structurizr with their custom DSL (Domain Specific Language).

All you need is to get familiar with the DSL syntax, which is pretty simple. As long as you get used to it you will be able to create or update diagrams in no time.

Below you can see the DSL for our Task Management Software System.

workspace {
    model {
        # Actors
        customer = person "Customer" "" "person"
        admin = person "Admin User" "" "person"

        # External systems
        emailSystem = softwareSystem "Email System" "Mailgun" "external"
        calendarSystem = softwareSystem "Calendar System" "Calendly" "external"

        # Task Management System
        taskManagementSystem  = softwareSystem "Task Management System"{
            webContainer = container "User Web UI" "" "" "frontend"
            adminContainer = container "Admin Web UI" "" "" "frontend"
            dbContainer = container "Database" "PostgreSQL" "" "database"
            apiContainer = container "API" "Go" {
                authComp = component "Authentication"
                crudComp = component "CRUD"
            }
        }

        # Relationships (Actors & Systems)
        customer -> webContainer "Manages tasks"
        admin -> adminContainer "Manages users"
        apiContainer -> emailSystem "Sends emails"
        apiContainer -> calendarSystem "Creates tasks in Calendar"

        # Relationships (Containers)
        webContainer -> apiContainer "Uses"
        adminContainer -> apiContainer "Uses"
        apiContainer -> dbContainer "Persists data"

        # Relationships (Components & Containers)
        crudComp -> dbContainer "Reads from and writes to"
        webContainer -> authComp "Authenticates using"
        adminContainer -> authComp "Authenticates using"
    }
}

Let's dive into the most important parts:

workspace [name] [description] {
    model {
    }
}

Here we define our workspace which should have at least one model. A workspace can optionally be given a name and description.

customer = person "Customer" "" "person"
admin = person "Admin User" "" "person"

In this section we define our persons (for example, a user, actor, role, or persona) in the following format: person <name> [description] [tags].

You can use a similar format (name, description, tags) to identify the external systems:

        emailSystem = softwareSystem "Email System" "Mailgun" "external"
        calendarSystem = softwareSystem "Calendar System" "Calendly" "external"

To describe the internal software system we need to write a block that also shows its containers and components:

taskManagementSystem  = softwareSystem "Task Management System"{
    webContainer = container "User Web UI" "" "" "frontend"
    adminContainer = container "Admin Web UI" "" "" "frontend"
    dbContainer = container "Database" "PostgreSQL" "" "database"
    apiContainer = container "API" "Go" {
        authComp = component "Authentication"
        crudComp = component "CRUD"
    }
}

• Container format: container <name> [description] [technology] [tags]

• Component format: component <name> [description] [technology] [tags]

The rest of the model is the most interesting part where we define the relationships between all parts (systems, containers, components):

apiContainer -> emailSystem "Sends emails"

The following format is used: <identifier> -> <identifier> [description] [technology] [tags].

There are other features available in Structurizr DSL, such as styling, themes, visibility, etc. You can find find them here.

Automate Rendering in Your CI

Since you can host your models on GitHub, it is very easy to automate the pipeline for rendering the diagrams in the tools of your choice.

In our case, Structurizr has a GitHub Action that allows you to run structurizr-cli, a command line utility for Structurizr that lets you create software architecture models based upon the C4 model using a textual domain specific language (DSL).

This sample repository contains a workflow that simply generates a static page and publishes it to GitHub Pages.

name: Deploy static content to Github Pages

on:
  push:
    branches: ["main"]

permissions:
  contents: read
  pages: write
  id-token: write

concurrency:
  group: "pages"
  cancel-in-progress: false

jobs:
  build:
    runs-on: ubuntu-latest
    container:
      image: ghcr.io/avisi-cloud/structurizr-site-generatr
      options: --user root
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Create site
        run: |
          /opt/structurizr-site-generatr/bin/structurizr-site-generatr generate-site -w diagram.dsl
      - uses: actions/upload-artifact@v3
        with:
          name: website
          path: build/site
  deploy:
    needs: build
    environment:
      name: github-pages
      url: ${{ steps.deployment.outputs.page_url }}
    runs-on: ubuntu-latest
    steps:
      - uses: actions/download-artifact@v3
        with:
          name: website
          path: build/site
      - name: Setup Pages
        uses: actions/configure-pages@v3
      - name: Upload artifact
        uses: actions/upload-pages-artifact@v1
        with:
          path: "build/site"
      - name: Deploy to GitHub Pages
        id: deployment
        uses: actions/deploy-pages@v2

This Github Action uses Structurizr CLI action to compile our DSL file as HTML and publish it to Github Pages.

Conclusion

I believe that creating and maintaining effective and clear diagrams should be effortless. Up-to-date visuals ensure everyone stays on the same page, eliminating confusion and wasted time.

The C4 model and a bit of automation with Structurizr DSL can help make this process faster and keep diagrams close to the codebase. The whole process can now be automated as well into your SDLC.

Resources

• Github Repository

• C4 Model

• DSL Language Reference

• C4 DSL Visual Studio Code Extension

• structurizr-cli-action

• Discover more articles from packagemain.tech

The purpose of integration tests is to validate that different software components, subsystems, or applications work well together combined as a group.

It’s an important step in the testing pyramid that can help you identify any issues that arise when components are combined – for example compatibility issues, data inconsistence, or communication issues.

This article is a hands-on guide to integration Tests in Go using Testcontainers. We'll define integrations tests as tests of communication between a backend application and external components such as the database and cache.

Table of Contents

• Different Ways to Run Integration Tests

• Our Guinea Pig Service: a Simple URL Shortener

• Unit Tests with Mocked Dependencies

• Integration Tests with Real Dependencies

• Integration Tests with Testcontainers

• How Testcontainers Work

• Conclusion

• Resources

Different Ways to Run Integration Tests

This diagram shows only 3 types of tests – but there are other kinds as well: components tests, system tests, load testing, and so on.

While unit tests are easy to run (you just execute tests as you would execute your code), integration tests usually require some scaffolding (spin up temporary testing environment with databases and other dependencies). In the companies where I've worked, I’ve seen the following approaches to address the integration testing environment problem.

Option 1: Using throwaway databases and other dependencies, which must be provisioned before the integration tests start and destroyed afterwards.

Depending on your application complexity, the effort involved in this option can be quite high, as you must ensure that the infrastructure is up and running and data is pre-configured in a specific desired state.

Option 2: Using the existing shared databases and other dependencies. You may create a separate environment for integration tests or even use the existing one (staging for example) that integration tests can use.

But there are many disadvantages here, and I would not recommend it. Because it is a shared environment, multiple tests can run in parallel and modify the data simultaneously. So you may end up with inconsistent data state for multiple reasons.

Option 3: Using in-memory or embedded variations of the required services for integration testing. While this is a good approach, not all dependencies have in-memory variations, and even if they do, these implementations may not have the same features as your production database.

Option 4: Using Testcontainers to bootstrap and manage your testing dependencies right inside your testing code. This ensures a full isolation between test runs, reproducibility and better CI experience. We will dive into that in a second.

Our Guinea Pig Service: a Simple URL Shortener

To demonstrate the tests, we'll use a super simple URL shortener API written in Go. It uses MongoDB for data storage and Redis as a read-through cache. It has two endpoints which we’ll be testing in our tests:

• /create?url= generates the hash for a given URL and stores it in the database.

• /get?key= returns the original URL for a given key.

We won’t delve into the details of the endpoints much, but you can find the full code in this Github repository. Still, let’s see how we define our “server“ struct:

type server struct {
  DB    DB
  Cache Cache
}

func NewServer(db DB, cache Cache) (*server, error) {
  if err := db.Init(); err != nil {
    return nil, err
  }
  if err := cache.Init(); err != nil {
    return nil, err
  }
  return &server{DB: db, Cache: cache}, nil
}

The NewServer function allows us to initialize a server with the database and cache instances that implement DB and Cache interfaces.

type DB interface {
  Init() error
  StoreURL(url string, key string) error
  GetURL(key string) (string, error)
}

type Cache interface {
  Init() error
  Set(key string, val string) error
  Get(key string) (string, bool)
}

Unit Tests with Mocked Dependencies

Because we had all dependencies defined as interfaces, we can easily generate mocks for them using mockery and use them in our unit tests.

mockery --all --with-expecter
go test -v ./...

With the help of unit tests, we can cover quite well the low level components of our application: endpoints, hash key logic, and so on. All we need is to mock the function calls of database and cache dependencies.

unit_test.go:

func TestServerWithMocks(t *testing.T) {
  mockDB := mocks.NewDB(t)
  mockCache := mocks.NewCache(t)

  mockDB.EXPECT().Init().Return(nil)
  mockDB.EXPECT().StoreURL(mock.Anything, mock.Anything).Return(nil)
  mockDB.EXPECT().GetURL(mock.Anything).Return("url", nil)

  mockCache.EXPECT().Init().Return(nil)
  mockCache.EXPECT().Get(mock.Anything).Return("url", true)
  mockCache.EXPECT().Set(mock.Anything, mock.Anything).Return(nil)

  s, err := NewServer(mockDB, mockCache)
  assert.NoError(t, err)

  srv := httptest.NewServer(s)
  defer srv.Close()

  // actual tests happen here, see the code in the repository
  testServer(srv, t)
}

mocks.NewDB(t) and mocks.NewCache(t) have been auto-generated by mockery and we use EXPECT() to mock the functions. Notice that we created a separate function testServer(srv, t) that we will use later in other tests as well, but providing a different server struct.

As you may already understand, these unit tests are not testing the communications between our application and our database/cache, and we may easily miss some very critical bugs.

To be more confident with our application, we should write integration tests along with unit tests to ensure that our application is fully functional.

Integration Tests with Real Dependencies

As Option 1 and 2 mention above, we can provision our dependencies beforehand and run our tests against these instances. One option would be to have a Docker Compose configuration with MongoDB and Redis, which we start before the tests and shutdown after. The seed data could be a part of this configuration, or done separately.

compose.yaml:

services:
  mongodb:
    image: mongodb/mongodb-community-server:7.0-ubi8
    restart: always
    ports:
      - "27017:27017"

  redis:
    image: redis:7.4-alpine
    restart: always
    ports:
      - "6379:6379"

realdeps_test.go:

//go:build realdeps
// +build realdeps

package main

func TestServerWithRealDependencies(t *testing.T) {
  os.Setenv("MONGO_URI", "mongodb://localhost:27017")
  os.Setenv("REDIS_URI", "redis://localhost:6379")

  s, err := NewServer(&MongoDB{}, &Redis{})
  assert.NoError(t, err)

  srv := httptest.NewServer(s)
  defer srv.Close()

  testServer(srv, t)
}

Now these tests don’t use mocks, but simply connect to the already provisioned database and cache. Note: we added a “realdeps“ build tag so these tests should be executed by specifying this tag explicitly.

docker-compose up -d
go test -tags=realdeps -v ./...
docker-compose down

Integration Tests with Testcontainers

However, creating reliable service dependencies using Docker Compose requires good knowledge of Docker internals and how to best run specific technologies in a container. For example, creating a dynamic integration testing environment may result in port conflicts, containers not being fully running and available, and so on.

With Testcontainers, we can now do the same – but inside our test suite, using our language API. This means we can control our throwaway dependencies better and make sure they’re isolated per each test run. You can run pretty much anything in Testcontainers, as long as it has a Docker-API compatible container runtime.

integration_test.go:

//go:build integration
// +build integration

package main

import (
  "context"
  "net/http/httptest"
  "os"
  "testing"
  "github.com/stretchr/testify/assert"
  "github.com/testcontainers/testcontainers-go/modules/mongodb"
  "github.com/testcontainers/testcontainers-go/modules/redis"
)

func TestServerWithTestcontainers(t *testing.T) {
  ctx := context.Background()

  mongodbContainer, err := mongodb.Run(ctx, "docker.io/mongodb/mongodb-community-server:7.0-ubi8")
  assert.NoError(t, err)
  defer mongodbContainer.Terminate(ctx)

  redisContainer, err := redis.Run(ctx, "docker.io/redis:7.4-alpine")
  assert.NoError(t, err)
  defer redisContainer.Terminate(ctx)

  mongodbEndpoint, _ := mongodbContainer.Endpoint(ctx, "")
  redisEndpoint, _ := redisContainer.Endpoint(ctx, "")

  os.Setenv("MONGO_URI", "mongodb://"+mongodbEndpoint)
  os.Setenv("REDIS_URI", "redis://"+redisEndpoint)

  s, err := NewServer(&MongoDB{}, &Redis{})
  assert.NoError(t, err)

  srv := httptest.NewServer(s)
  defer srv.Close()

  testServer(srv, t)
}

This is very similar to the previous test: we just initialized two containers at the top of our test.

The first run may take a while to download the images. But the subsequent runs are almost instant.

How Testcontainers Work

To run tests with Testcontainers, you need a Docker-API compatible container runtime or to install Docker locally. Try stopping your Docker engine and it won’t work.

But this should not be an issue for most developers, because having a Docker runtime in your CI/CD or locally is a very common practice nowadays. You can easily have this environment in Github Actions, for example.

When it comes to supported languages, Testcontainers support a big list of popular languages and platforms including Java, .NET, Go, NodeJS, Python, Rust, Haskell, and others.

There is also a growing list of preconfigured implementations (called modules) which you can find here. But as I mentioned earlier, you can run any Docker image.

In Go, you could use the following code to provision Redis instead of using a preconfigured module:

// Using available module
redisContainer, err := redis.Run(ctx, "redis:latest")

// Or using GenericContainer
req := testcontainers.ContainerRequest{
  Image:        "redis:latest",
  ExposedPorts: []string{"6379/tcp"},
  WaitingFor:   wait.ForLog("Ready to accept connections"),
}

redisC, err := testcontainers.GenericContainer(ctx, testcontainers.GenericContainerRequest{
  ContainerRequest: req,
  Started:          true,
})

Conclusion

While the development and maintenance of integration tests require significant effort, they are crucial part of the SDLC ensuring that components, subsystems, or applications work well together combined as a group.

Using Testcontainers, we can simplify the provisioning and de-provisioning of throwaway dependencies for testing, making the test runs fully isolated and more predicatble.

Resources

• Github repository

• Testcontainers

• Discover more articles from packagemain.tech

In the world of software, a similar concept exists: the hard shutdown. This abrupt termination can cause problems just like its physical counterpart. Thankfully, there's a better way: the graceful shutdown.

For applications deployed in orchestrated environments (like Kubernetes), graceful handling of termination signals is crucial.

By integrating graceful shutdown, you provide advance notification to the service. This enables it to complete ongoing requests, potentially save state information to disk, and ultimately avoid data corruption during shutdown.

In this guide, we'll dive into the world of graceful shutdowns, specifically focusing on their implementation in Go applications running on Kubernetes.

Signals in Unix Systems

One of the key tools for achieving graceful shutdowns in Unix-based systems is the concept of signals. These are, in basic terms, a simple way to communicate one specific thing to a process, from another process.

By understanding how signals work, you can leverage them to implement controlled termination procedures within your applications, ensuring a smooth and data-safe shutdown process.

There are many signals, and you can find them here. But our concern in this article is only shutdown signals:

• SIGTERM – sent to a process to request its termination. Most commonly used, and we’ll be focusing on it later.

• SIGKILL – “quit immediately”, can not be interfered with.

• SIGINT – interrupt signal (such as Ctrl+C)

• SIGQUIT – quit signal (such as Ctrl+D)

These signals can be sent from the user (Ctrl+C / Ctrl+D), from another program/process, or from the system itself (kernel / OS). For example, a SIGSEGV aka segmentation fault is sent by the OS.

Our Guinea Pig Service

To explore the world of graceful shutdowns in a practical setting, let's create a simple service we can experiment with. This "guinea pig" service will have a single endpoint that simulates some real-world work (we’ll add a slight delay) by calling Redis's INCR command. We'll also provide a basic Kubernetes configuration to test how the platform handles termination signals.

The ultimate goal: ensure our service gracefully handles shutdowns without losing any requests/data. By comparing the number of requests sent in parallel with the final counter value in Redis, we'll be able to verify if our graceful shutdown implementation is successful.

We won’t go into details of setting up the Kubernetes cluster and Redis, but you can find the full setup in this Github repository.

The verification process is the following:

1. Deploy Redis and Go application to Kubernetes.

2. Use vegeta to send 1000 requests (25/s over 40 seconds).

3. While vegeta is running, initialize a Kubernetes Rolling Update by updating the image tag.

4. Connect to Redis to verify the “counter“, it should be 1000.

Let’s start with our base Go HTTP Server.

hard-shutdown/main.go:

package main

import (
  "net/http"
  "os"
  "time"

  "github.com/go-redis/redis"
)

func main() {
  redisdb := redis.NewClient(&redis.Options{
    Addr: os.Getenv("REDIS_ADDR"),
  })

  server := http.Server{
    Addr: ":8080",
  }

  http.HandleFunc("/incr", func(w http.ResponseWriter, r *http.Request) {
    go processRequest(redisdb)
    w.WriteHeader(http.StatusOK)
  })

  server.ListenAndServe()
}

func processRequest(redisdb *redis.Client) {
  // simulate some business logic here
  time.Sleep(time.Second * 5)
  redisdb.Incr("counter")
}

When we run our verification procedure using this code, we’ll see that some requests fail and the counter is less than 1000 (the number may vary each run).

Which clearly means that we lost some data during the rolling update. 😢

How to Handle Signals in Go

Go provides a signal package that allows you to handle Unix Signals. It’s important to note that by default, the SIGINT and SIGTERM signals cause the Go program to exit. And in order for our Go application not to exit so abruptly, we need to handle incoming signals.

There are two options to do so.

The first is using channel:

c := make(chan os.Signal, 1)
signal.Notify(c, syscall.SIGTERM)

The second is using context (the preferred approach nowadays):

ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGTERM)
defer stop()

NotifyContext returns a copy of the parent context that is marked done (its Done channel is closed) when one of the listed signals arrives, when the returned stop() function is called, or when the parent context's Done channel is closed – whichever happens first.

There are few problems with our current implementation of HTTP Server:

1. We have a slow processRequest goroutine, and since we don’t handle the termination signal, the program exits automatically. This means that all running goroutines are terminated as well.

2. The program doesn’t close any connections.

Let’s rewrite it.

graceful-shutdown/main.go:

package main

// imports

var wg sync.WaitGroup

func main() {
  ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGTERM)
  defer stop()

  // redisdb, server

  http.HandleFunc("/incr", func(w http.ResponseWriter, r *http.Request) {
    wg.Add(1)
    go processRequest(redisdb)
    w.WriteHeader(http.StatusOK)
  })

  // make it a goroutine
  go server.ListenAndServe()

  // listen for the interrupt signal
  <-ctx.Done()

  // stop the server
  if err := server.Shutdown(context.Background()); err != nil {
    log.Fatalf("could not shutdown: %v\n", err)
  }

  // wait for all goroutines to finish
  wg.Wait()

  // close redis connection
  redisdb.Close()

  os.Exit(0)
}

func processRequest(redisdb *redis.Client) {
  defer wg.Done()

  // simulate some business logic here
  time.Sleep(time.Second * 5)
  redisdb.Incr("counter")
}

Here’s the summary of updates:

• Added signal.NotifyContext to listen for the SIGTERM termination signal.

• Introduced a sync.WaitGroup to track in-flight requests (processRequest goroutines).

• Wrapped the server in a goroutine and used server.Shutdown with context to gracefully stop accepting new connections.

• Used wg.Wait() to ensure all in-flight requests (processRequest goroutines) finish before proceeding.

• Resource Cleanup: Added redisdb.Close() to properly close the Redis connection before exiting.

• Clean Exit: Used os.Exit(0) to indicate a successful termination.

Now, if we repeat our verification process, we will see that all 1000 requests are processed correctly. 🎉

Web Frameworks / HTTP Library

Frameworks like Echo, Gin, Fiber and others will spawn a goroutine for each incoming request. This gives it a context and then calls your function / handler depending on the routing you decided. In our case, it would be the anonymous function given to HandleFunc for the “/incr” path.

When you intercept a SIGTERM signal and ask your framework to gracefully shutdown, two important things happen (to oversimplify):

• Your framework stops accepting incoming requests

• It waits for any existing incoming requests to finish (implicitly waiting for the goroutines to end).

Note: Kubernetes also stops directing incoming traffic from the loadbalancer to your pod once it has labelled it as Terminating.

Optional: Shutdown Timeout

Terminating a process can be complex, especially if there are many steps involved like closing connections. To ensure things run smoothly, you can set a timeout. This timeout acts as a safety net, gracefully exiting the process if it takes longer than expected.

shutdownCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()

go func() {
  if err := server.Shutdown(shutdownCtx); err != nil {
    log.Fatalf("could not shutdown: %v\n", err)
  }
}()

select {
case <-shutdownCtx.Done():
  if shutdownCtx.Err() == context.DeadlineExceeded {
    log.Fatalln("timeout exceeded, forcing shutdown")
  }

  os.Exit(0)
}

Kubernetes Termination Lifecycle

Since we used Kubernetes to deploy our service, let’s dive deeper into how it terminates the pods. Once Kubernetes decides to terminate the pod, the following events will take place:

1. Pod is set to the “Terminating” State and removed from the endpoints list of all Services.

2. preStop Hook is executed if defined.

3. SIGTERM signal is sent to the pod. But hey, now our application knows what to do!

4. Kubernetes waits for a grace period (terminationGracePeriodSeconds), which is 30s by default.

5. SIGKILL signal is sent to pod, and the pod is removed.

As you can see, if you have a long-running termination process, it may be necessary to increase the terminationGracePeriodSeconds setting. This allows your application enough time to shut down gracefully.

Conclusion

Graceful shutdowns safeguard data integrity, maintain a seamless user experience, and optimize resource management. With its rich standard library and emphasis on concurrency, Go empowers developers to effortlessly integrate graceful shutdown practices – a necessity for applications deployed in containerized or orchestrated environments like Kubernetes.

You can find the Go code and Kubernetes manifests in this Github repository.

Resources

• os/signal package

• Kubernetes Pod Lifecycle

• Explore more articles from packagemain.tech