Google Cloud Axion is Google's own custom ARM-based chip, built to handle the demands of modern cloud workloads. The performance and cost numbers are striking: Google claims Axion delivers up to 60% better energy efficiency and up to 65% better price-performance compared to comparable x86 machines.

AWS has Graviton. Azure has Cobalt. ARM is no longer niche. It's the direction the entire cloud industry is moving.

But there's a problem that catches almost every team off guard when they start this transition: container architecture mismatch.

If you build a Docker image on your M-series Mac and push it to an x86 server, it crashes on startup with a cryptic exec format error.

The server isn't broken. It just can't read the compiled instructions inside your image. An ARM binary and an x86 binary are written in fundamentally different languages at the machine level. The CPU literally can't execute instructions it wasn't designed for.

We're going to solve this problem completely in this tutorial. You'll build a single Docker image tag that automatically serves the correct binary on both ARM and x86 machines — no separate pipelines, no separate tags. Then you'll provision Google Cloud ARM nodes in GKE and configure your Kubernetes deployment to route workloads precisely to those cost-efficient nodes.

Here's what you'll build, step by step:

• A Go HTTP server that reports the CPU architecture it's running on at runtime

• A multi-stage Dockerfile that cross-compiles for both linux/amd64 and linux/arm64 without slow QEMU emulation

• A multi-arch image in Google Artifact Registry that acts as a single entry point for any architecture

• A GKE cluster with two node pools: a standard x86 pool and an ARM Axion pool

• A Kubernetes Deployment that pins your workload exclusively to the ARM nodes

By the end, you'll hit a live endpoint and see the word arm64 staring back at you from a Google Cloud ARM node. Let's get into it.

Table of Contents

• Prerequisites

• Step 1: Set Up Your Google Cloud Project

• Step 2: Create the GKE Cluster

• Step 3: Write the Application

• Step 4: Enable Multi-Arch Builds with Docker Buildx

• Step 5: Write the Dockerfile

• Step 6: Build and Push the Multi-Arch Image

• Step 7: Add the Axion ARM Node Pool

• Step 8: Deploy the App to the ARM Node Pool

• Step 9: Verify the Deployment

• Step 10: Cost Savings and Tradeoffs

• Cleanup

• Conclusion

• Project File Structure

Prerequisites

Before you start, make sure you have the following ready:

• A Google Cloud project with billing enabled. If you don't have one, create it at console.cloud.google.com. The total cost to follow this tutorial is around $5–10.

• gcloud CLI installed and authenticated. Run gcloud auth login to sign in and gcloud config set project YOUR_PROJECT_ID to point it at your project.

• Docker Desktop version 19.03 or later. Docker Buildx (the tool we'll use for multi-arch builds) ships bundled with it.

• kubectl installed. This is the CLI for interacting with Kubernetes clusters.

• Basic familiarity with Docker (images, layers, Dockerfile) and Kubernetes (pods, deployments, services). You don't need to be an expert, but you should know what these things are.

Step 1: Set Up Your Google Cloud Project

Before writing a single line of application code, let's get the cloud infrastructure side ready. This is the foundation everything else will build on.

Enable the Required APIs

Google Cloud services are off by default in any new project. Run this command to turn on the three APIs we'll need:

gcloud services enable \
  artifactregistry.googleapis.com \
  container.googleapis.com \
  containeranalysis.googleapis.com

Here's what each one does:

• artifactregistry.googleapis.com — enables Artifact Registry, where we'll store our Docker images

• container.googleapis.com — enables Google Kubernetes Engine (GKE), where our cluster will run

• containeranalysis.googleapis.com — enables vulnerability scanning for images stored in Artifact Registry

Create a Docker Repository in Artifact Registry

Artifact Registry is Google Cloud's managed container image store — the place where our built images will live before being deployed to the cluster. Create a dedicated repository for this tutorial:

gcloud artifacts repositories create multi-arch-repo \
  --repository-format=docker \
  --location=us-central1 \
  --description="Multi-arch tutorial images"

Breaking down the flags:

• --repository-format=docker — tells Artifact Registry this repository stores Docker images (as opposed to npm packages, Maven artifacts, and so on)

• --location=us-central1 — the Google Cloud region where your images will be stored. Use a region that's close to where your cluster will run to minimize image pull latency. Run gcloud artifacts locations list to see all options.

• --description — a human-readable label for the repository, shown in the console.

Authenticate Docker to Push to Artifact Registry

Docker needs credentials before it can push images to Google Cloud. Run this command to wire up authentication automatically:

gcloud auth configure-docker us-central1-docker.pkg.dev

This adds a credential helper entry to your ~/.docker/config.json file. What that means in practice: any time Docker tries to push or pull from a URL under us-central1-docker.pkg.dev, it will automatically call gcloud to get a valid auth token. You won't need to run docker login manually.

Step 2: Create the GKE Cluster

With Artifact Registry ready to receive images, let's create the Kubernetes cluster. We'll start with a standard cluster using x86 nodes and add an ARM node pool later once we have an image to deploy.

gcloud container clusters create axion-tutorial-cluster \
  --zone=us-central1-a \
  --num-nodes=2 \
  --machine-type=e2-standard-2 \
  --workload-pool=PROJECT_ID.svc.id.goog

Replace PROJECT_ID with your actual Google Cloud project ID.

What each flag does:

• --zone=us-central1-a — creates a zonal cluster in a single availability zone. A regional cluster (using --region) would spread nodes across three zones for higher resilience, but for this tutorial a single zone keeps things simple and avoids capacity issues that can affect specific zones. If us-central1-a is unavailable, try us-central1-b.

• --num-nodes=2 — two x86 nodes in this zone. We need at least 2 to have enough capacity alongside our ARM node pool later.

• --machine-type=e2-standard-2 — the machine type for this default node pool. e2-standard-2 is a cost-effective x86 machine with 2 vCPUs and 8 GB of memory, good for general workloads.

• --workload-pool=PROJECT_ID.svc.id.goog — enables Workload Identity, which is Google's recommended way for pods to authenticate with Google Cloud APIs. It avoids the need to download and store service account key files inside your cluster.

This command takes a few minutes. While it runs, you can move on to writing the application. We'll come back to the cluster in Step 6.

Step 3: Write the Application

We need an application to containerize. We'll use Go for three specific reasons:

1. Go compiles into a single, statically-linked binary. There's no runtime to install, no interpreter — just the binary. This makes for extremely lean container images.

2. Go has first-class, built-in cross-compilation support. We can compile an ARM64 binary from an x86 Mac, or vice versa, by setting two environment variables. This will matter a lot when we get to the Dockerfile.

3. Go exposes the architecture the binary was compiled for via runtime.GOARCH. Our server will report this at runtime, giving us hard proof that the correct binary is running on the correct hardware.

Start by creating the project directories:

mkdir -p hello-axion/app hello-axion/k8s
cd hello-axion/app

Initialize the Go module from inside app/. This creates go.mod in the current directory:

go mod init hello-axion

go mod init is Go's built-in command for starting a new module. It writes a go.mod file that declares the module name (hello-axion) and the minimum Go version required. Every modern Go project needs this file — without it, the compiler doesn't know how to resolve packages.

Now create the application at app/main.go:

package main

import (
    "fmt"
    "net/http"
    "os"
    "runtime"
)

func handler(w http.ResponseWriter, r *http.Request) {
    hostname, _ := os.Hostname()
    fmt.Fprintf(w, "Hello from freeCodeCamp!\n")
    fmt.Fprintf(w, "Architecture : %s\n", runtime.GOARCH)
    fmt.Fprintf(w, "OS           : %s\n", runtime.GOOS)
    fmt.Fprintf(w, "Pod hostname : %s\n", hostname)
}

func healthz(w http.ResponseWriter, r *http.Request) {
    w.WriteHeader(http.StatusOK)
    fmt.Fprintln(w, "ok")
}

func main() {
    http.HandleFunc("/", handler)
    http.HandleFunc("/healthz", healthz)
    fmt.Println("Server starting on port 8080...")
    if err := http.ListenAndServe(":8080", nil); err != nil {
        fmt.Fprintf(os.Stderr, "server error: %v\n", err)
        os.Exit(1)
    }
}

Verify both files were created:

ls -la

You should see go.mod and main.go listed.

Let's walk through what this code does:

• import "runtime" — imports Go's built-in runtime package, which exposes information about the Go runtime environment, including the CPU architecture.

• runtime.GOARCH — returns a string like "arm64" or "amd64" representing the architecture this binary was compiled for. When we deploy to an ARM node, this value will be arm64. This is the core of our proof.

• os.Hostname() — returns the pod's hostname, which Kubernetes sets to the pod name. This lets us see which specific pod responded when we test the app later.

• handler — the main HTTP handler, registered on the root path /. It writes the architecture, OS, and hostname to the response.

• healthz — a separate handler registered on /healthz. It returns HTTP 200 with the text ok. Kubernetes will use this endpoint to check whether the container is alive and ready to serve traffic — we'll wire this up in the deployment manifest later.

• http.ListenAndServe(":8080", nil) — starts the server on port 8080. If it fails to start (for example, if the port is already in use), it prints the error and exits with a non-zero code so Kubernetes knows something went wrong.

Step 4: Enable Multi-Arch Builds with Docker Buildx

Before we write the Dockerfile, we need to understand a fundamental constraint, because it directly shapes how the Dockerfile must be written.

Why Your Docker Images Are Architecture-Specific By Default

A CPU only understands instructions written for its specific Instruction Set Architecture (ISA). ARM64 and x86_64 are different ISAs — different vocabularies of machine-level operations. When you compile a Go program, the compiler translates your source code into binary instructions for exactly one ISA. That binary can't run on a different ISA.

When you build a Docker image the normal way (docker build), the binary inside that image is compiled for your local machine's ISA. If you're on an Apple Silicon Mac, you get an ARM64 binary. Push that image to an x86 server, and when Docker tries to execute the binary, the kernel rejects it:

standard_init_linux.go:228: exec user process caused: exec format error

That's the operating system saying: "This binary was written for a different processor. I have no idea what to do with it."

The Solution: A Single Image Tag That Serves Any Architecture

Docker solves this with a structure called a Manifest List (also called a multi-arch image index). Instead of one image, a Manifest List is a pointer table. It holds multiple image references — one per architecture — all under the same tag.

When a server pulls hello-axion:v1, here's what actually happens:

1. Docker contacts the registry and requests the manifest for hello-axion:v1

2. The registry returns the Manifest List, which looks like this internally:

{
  "manifests": [
    { "digest": "sha256:a1b2...", "platform": { "architecture": "amd64", "os": "linux" } },
    { "digest": "sha256:c3d4...", "platform": { "architecture": "arm64", "os": "linux" } }
  ]
}

1. Docker checks the current machine's architecture, finds the matching entry, and pulls only that specific image layer. The x86 image never downloads onto your ARM server, and vice versa.

One tag, two actual images. Completely transparent to your deployment manifests.

Set Up Docker Buildx

Docker Buildx is the CLI tool that builds these Manifest Lists. It's powered by the BuildKit engine and ships bundled with Docker Desktop. Run the following to create and activate a new builder instance:

docker buildx create --name multiarch-builder --use

• --name multiarch-builder — gives this builder a memorable name. You can have multiple builders. This command creates a new one named multiarch-builder.

• --use — immediately sets this new builder as the active one, so all future docker buildx build commands use it.

Now boot the builder and confirm it supports the platforms we need:

docker buildx inspect --bootstrap

• --bootstrap — starts the builder container if it isn't already running, and prints its full configuration.

You should see output like this:

Name:          multiarch-builder
Driver:        docker-container
Platforms:     linux/amd64, linux/arm64, linux/arm/v7, linux/386, ...

The Platforms line lists every architecture this builder can produce images for. As long as you see linux/amd64 and linux/arm64 in that list, you're ready to build for both x86 and ARM.

Step 5: Write the Dockerfile

Now we can write the Dockerfile. We'll use two techniques together: a multi-stage build to keep the final image tiny, and a cross-compilation trick to avoid slow CPU emulation.

Create app/Dockerfile with the following content:

# -----------------------------------------------------------
# Stage 1: Build
# -----------------------------------------------------------
# $BUILDPLATFORM = the machine running this build (your laptop)
# \(TARGETOS / \)TARGETARCH = the platform we are building FOR
# -----------------------------------------------------------
FROM --platform=$BUILDPLATFORM golang:1.23-alpine AS builder

ARG TARGETOS
ARG TARGETARCH

WORKDIR /app

COPY go.mod .
RUN go mod download

COPY main.go .

RUN GOOS=\(TARGETOS GOARCH=\)TARGETARCH go build -ldflags="-w -s" -o server main.go

# -----------------------------------------------------------
# Stage 2: Runtime
# -----------------------------------------------------------

FROM alpine:latest

RUN addgroup -S appgroup && adduser -S appuser -G appgroup
USER appuser

WORKDIR /app
COPY --from=builder /app/server .

EXPOSE 8080
CMD ["./server"]

There's a lot happening here. Let's go through it carefully.

Stage 1: The Builder

FROM --platform=$BUILDPLATFORM golang:1.23-alpine AS builder

This is the most important line in the file. \(BUILDPLATFORM is a special build argument that Docker Buildx automatically injects — it equals the platform of the machine running the build (your laptop). By pinning the builder stage to \)BUILDPLATFORM, the Go compiler always runs natively on your machine, not inside a CPU emulator. This is what makes multi-arch builds fast.

Without --platform=$BUILDPLATFORM, Buildx would have to use QEMU — a full CPU emulator — to run an ARM64 build environment on your x86 machine (or vice versa). QEMU works, but it's typically 5–10 times slower than native execution. For a project with many dependencies, that's the difference between a 2-minute build and a 20-minute build.

ARG TARGETOS and ARG TARGETARCH

These two lines declare that our Dockerfile expects build arguments named TARGETOS and TARGETARCH. Buildx injects these automatically based on the --platform flag you pass at build time. For a linux/arm64 target, TARGETOS will be linux and TARGETARCH will be arm64.

COPY go.mod . and RUN go mod download

We copy go.mod first, before copying the rest of the source code. Docker builds images layer by layer and caches each layer. By copying only the module file first, we create a cached layer for go mod download.

On future builds, as long as go.mod hasn't changed, Docker skips the download step entirely — even if the source code changed. This speeds up iterative development significantly.

RUN GOOS=\(TARGETOS GOARCH=\)TARGETARCH go build -ldflags="-w -s" -o server main.go

This is the cross-compilation step. GOOS and GOARCH are Go's built-in cross-compilation environment variables. Setting them tells the Go compiler to produce a binary for a different target than the machine it's running on. We set them from the \(TARGETOS and \)TARGETARCH build args injected by Buildx.

The -ldflags="-w -s" flag strips the debug symbol table and the DWARF debugging information from the binary. This has no effect on runtime behavior but reduces the binary size by roughly 30%.

Stage 2: The Runtime Image

FROM alpine:latest

This starts a brand-new image from Alpine Linux — a minimal Linux distribution that weighs about 5 MB. Critically, alpine:latest is itself a multi-arch image, so Docker automatically selects the arm64 or amd64 Alpine variant depending on which platform this stage is built for.

Everything from Stage 1 — the Go toolchain, the source files, the intermediate object files — is discarded. The final image contains only Alpine Linux plus our binary. Compared to a naive single-stage Go image (~300 MB), this approach produces an image under 15 MB.

RUN addgroup -S appgroup && adduser -S appuser -G appgroup and USER appuser

These two lines create a non-root user and set it as the active user for the container. Running containers as root is a security risk — if an attacker exploits a vulnerability in your application, they gain root access inside the container. Running as a non-root user limits the blast radius.

COPY --from=builder /app/server .

This is how multi-stage builds work: the --from=builder flag tells Docker to copy files from the builder stage (Stage 1), not from your local disk. Only the compiled binary (server) makes it into the final image.

Step 6: Build and Push the Multi-Arch Image

With the application and Dockerfile in place, we can now build images for both architectures and push them to Artifact Registry — all in a single command.

From inside the app/ directory, run:

docker buildx build \
  --platform linux/amd64,linux/arm64 \
  -t us-central1-docker.pkg.dev/PROJECT_ID/multi-arch-repo/hello-axion:v1 \
  --push \
  .

Replace PROJECT_ID with your actual GCP project ID.

Here's what each part of this command does:

• docker buildx build — uses the Buildx CLI instead of the standard docker build. Buildx is required for multi-platform builds.

• --platform linux/amd64,linux/arm64 — instructs Buildx to build the image twice: once targeting x86 Intel/AMD machines, and once targeting ARM64. Both builds run in parallel. Because our Dockerfile uses the $BUILDPLATFORM cross-compilation trick, both builds run natively on your machine without QEMU emulation.

• -t us-central1-docker.pkg.dev/PROJECT_ID/multi-arch-repo/hello-axion:v1 — the full image path in Artifact Registry. The format is always REGION-docker.pkg.dev/PROJECT_ID/REPO_NAME/IMAGE_NAME:TAG.

• --push — multi-arch images can't be loaded into your local Docker daemon (which only understands single-architecture images). This flag tells Buildx to skip local storage and push the completed Manifest List — with both architecture variants — directly to the registry.

• . — the build context, the directory Docker scans for the Dockerfile and any files the build needs.

Watch the output as the build runs. You'll see BuildKit working on both platforms simultaneously:

 => [linux/amd64 builder 1/5] FROM golang:1.23-alpine
 => [linux/arm64 builder 1/5] FROM golang:1.23-alpine
 ...
 => pushing manifest for us-central1-docker.pkg.dev/.../hello-axion:v1

Verify the Multi-Arch Image in Artifact Registry

Once the push completes, navigate to GCP Console → Artifact Registry → Repositories → multi-arch-repo and click on hello-axion.

You won't see a single image — you'll see something labelled "Image Index". That's the Manifest List we created. Click into it, and you'll find two child images with separate digests, one for linux/amd64 and one for linux/arm64.

You can also inspect this from the command line:

docker buildx imagetools inspect \
  us-central1-docker.pkg.dev/PROJECT_ID/multi-arch-repo/hello-axion:v1

The output lists every manifest inside the image index. You'll see entries for linux/amd64 and linux/arm64 — those are our two real images. You'll also see two entries with Platform: unknown/unknown labelled as attestation-manifest. These are build provenance records that Docker Buildx automatically attaches to prove how and where the image was built (a supply chain security feature called SLSA attestation).

The two entries you care about are linux/amd64 and linux/arm64. Note the digest for the arm64 entry — we'll use it in the verification step to confirm the cluster pulled the right variant.

Step 7: Add the Axion ARM Node Pool

We have a universal image. Now we need somewhere to run it.

Recall the cluster we created in Step 2 — it's running e2-standard-2 x86 machines. We're going to add a second node pool running ARM machines. This is the key architectural move: a mixed-architecture cluster where different workloads can be routed to different hardware.

Choosing Your ARM Machine Type

Google Cloud currently offers two ARM-based machine series in GKE:

This tutorial uses t2a-standard-2 because it's widely available. The commands are identical for c4a-standard-2 — just swap the --machine-type value. If t2a-standard-2 isn't available in your zone, GKE will tell you immediately when you run the node pool creation command below, and you can try a neighbouring zone.

Create the ARM Node Pool

Add the ARM node pool to your existing cluster:

gcloud container node-pools create axion-pool \
  --cluster=axion-tutorial-cluster \
  --zone=us-central1-a \
  --machine-type=t2a-standard-2 \
  --num-nodes=2 \
  --node-labels=workload-type=arm-optimized

What each flag does:

• --cluster=axion-tutorial-cluster — the name of the cluster we created in Step 2. Node pools are always added to an existing cluster.

• --zone=us-central1-a — must match the zone you used when creating the cluster.

• --machine-type=t2a-standard-2 — GKE detects this is an ARM machine type and automatically provisions the nodes with an ARM-compatible version of Container-Optimized OS (COS). You don't need to configure anything special for ARM at the OS level.

• --num-nodes=2 — two ARM nodes in the zone, enough to schedule our 3-replica deployment alongside other cluster overhead.

• --node-labels=workload-type=arm-optimized — attaches a custom label to every node in this pool. We'll use this label in our deployment manifest to target these specific nodes. Using a descriptive custom label (rather than just relying on the automatic kubernetes.io/arch=arm64 label) is good practice in real clusters — it communicates the intent of the pool, not just its hardware.

This command takes a few minutes. Once it completes, let's confirm our cluster now has both node pools:

gcloud container clusters get-credentials axion-tutorial-cluster --zone=us-central1-a

kubectl get nodes --label-columns=kubernetes.io/arch

The get-credentials command configures kubectl to authenticate with your new cluster. The get nodes command then lists all nodes and adds a column showing the kubernetes.io/arch label.

You should see something like:

NAME                                    STATUS   ARCH    AGE
gke-...default-pool-abc...              Ready    amd64   15m
gke-...default-pool-def...              Ready    amd64   15m
gke-...axion-pool-jkl...                Ready    arm64   3m
gke-...axion-pool-mno...                Ready    arm64   3m

amd64 for the default x86 pool, arm64 for our new Axion pool. This kubernetes.io/arch label is applied automatically by GKE — you don't set it, it's derived from the hardware.

Step 8: Deploy the App to the ARM Node Pool

We have a multi-arch image and a mixed-architecture cluster. Here's something important to understand before writing the deployment manifest: Kubernetes doesn't know or care about image architecture by default.

If you applied a standard Deployment right now, the scheduler would look for any available node with enough CPU and memory and place pods there — potentially landing on x86 nodes instead of your ARM Axion nodes. The multi-arch Manifest List would handle this gracefully (the right binary would run regardless), but you'd lose the cost efficiency you provisioned Axion nodes for in the first place.

To guarantee that pods land on ARM nodes and only ARM nodes, we use a nodeSelector.

How nodeSelector Works

A nodeSelector is a set of key-value pairs in your pod spec. Before the Kubernetes scheduler places a pod, it checks every available node's labels. If a node doesn't have all the labels in the nodeSelector, the scheduler skips it — the pod will remain in Pending state rather than land on the wrong node.

This is a hard constraint, which is exactly what we want for cost optimization. Contrast this with Node Affinity's soft preference mode (preferredDuringSchedulingIgnoredDuringExecution), which says "try to use ARM, but fall back to x86 if needed." Soft preferences are useful for resilience, but they undermine the whole point of dedicated ARM pools. We want the hard constraint.

Write the Deployment Manifest

Create k8s/deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-axion
  labels:
    app: hello-axion
spec:
  replicas: 3
  selector:
    matchLabels:
      app: hello-axion
  template:
    metadata:
      labels:
        app: hello-axion
    spec:
      nodeSelector:
        kubernetes.io/arch: arm64

      containers:
      - name: hello-axion
        image: us-central1-docker.pkg.dev/PROJECT_ID/multi-arch-repo/hello-axion:v1
        ports:
        - containerPort: 8080
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 3
          periodSeconds: 5
        resources:
          requests:
            cpu: "250m"
            memory: "64Mi"
          limits:
            cpu: "500m"
            memory: "128Mi"

Replace PROJECT_ID with your project ID. Here's what the key sections do:

replicas: 3 — tells Kubernetes to keep three instances of this pod running at all times. If one crashes or a node goes down, the scheduler spins up a replacement. Three replicas also means one pod per ARM node in us-central1, which distributes load across availability zones.

selector.matchLabels and template.metadata.labels — these two blocks must match. The selector tells the Deployment which pods it "owns," and the template.metadata.labels is what those pods will be tagged with. If they don't match, Kubernetes won't be able to manage the pods.

nodeSelector: kubernetes.io/arch: arm64 — this is the pin. The Kubernetes scheduler filters out every node that doesn't carry this label before considering resource availability. Since GKE automatically applies kubernetes.io/arch=arm64 to all ARM nodes, our pods will schedule only onto the axion-pool nodes.

livenessProbe — periodically calls GET /healthz. If this check fails a certain number of times in a row (indicating the container has deadlocked or is otherwise unresponsive), Kubernetes restarts the container. initialDelaySeconds: 5 gives the server 5 seconds to start up before the first check.

readinessProbe — similar to the liveness probe, but with a different purpose. While the readiness probe is failing, Kubernetes removes the pod from the service's load balancer, so no traffic is sent to it. This is important during startup — the pod won't receive traffic until it signals it's ready.

resources.requests — reserves 250m (25% of a CPU core) and 64Mi of memory on the node for this pod. The scheduler uses these numbers to decide whether a node has enough room for the pod. Setting requests is required for sensible bin-packing. Without them, nodes can be silently overcommitted.

resources.limits — caps the container at 500m CPU and 128Mi memory. If the container exceeds these limits, Kubernetes throttles the CPU or kills the container (for memory). This prevents a single misbehaving pod from starving other workloads on the same node.

A Note on Taints and Tolerations

Once you're comfortable with nodeSelector, the next step in production clusters is adding a taint to your ARM node pool. A taint is a repellent — any pod without an explicit toleration for that taint is blocked from landing on the tainted node.

This means other workloads in your cluster can't accidentally consume your ARM capacity. You'd add the taint when creating the pool:

# Add --node-taints to the pool creation command:
--node-taints=workload-type=arm-optimized:NoSchedule

And a matching toleration in the pod spec:

tolerations:
- key: "workload-type"
  operator: "Equal"
  value: "arm-optimized"
  effect: "NoSchedule"

We're not doing this in the tutorial to keep things simple, but it's the pattern production multi-tenant clusters use to enforce hard separation between workload types.

Write the Service Manifest

We also need a Kubernetes Service to expose the pods over the network. Create k8s/service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: hello-axion-svc
spec:
  selector:
    app: hello-axion
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080
  type: LoadBalancer

• selector: app: hello-axion — the Service discovers pods using labels. Any pod with app: hello-axion on it will be added to this Service's load balancer pool.

• port: 80 — the port the Service is reachable on from outside the cluster.

• targetPort: 8080 — the port on the pod that traffic gets forwarded to. Our Go server listens on port 8080, so this must match.

• type: LoadBalancer — tells GKE to provision an external Google Cloud load balancer and assign it a public IP. This is what makes the Service reachable from the internet.

Apply Both Manifests

kubectl apply -f k8s/deployment.yaml
kubectl apply -f k8s/service.yaml

kubectl apply reads each manifest file and creates or updates the resources described in it. If the resources don't exist yet, they're created. If they already exist, Kubernetes only applies the diff — it won't restart pods unnecessarily.

Watch the pods come up in real time:

kubectl get pods -w

The -w flag watches for changes and prints updates as they happen. You should see pods transition from Pending → ContainerCreating → Running. Once all three show Running, press Ctrl+C to stop watching.

Step 9: Verify the Deployment

Everything is running. Now we need evidence — not just that pods are up, but that they're on the right nodes and serving the right binary.

Confirm Pod Placement

kubectl get pods -o wide

The -o wide flag adds extra columns to the output, including the name of the node each pod was scheduled on. Look at the NODE column:

NAME                          READY   STATUS    NODE
hello-axion-7b8d9f-abc12      1/1     Running   gke-axion-tutorial-axion-pool-a-...
hello-axion-7b8d9f-def34      1/1     Running   gke-axion-tutorial-axion-pool-b-...
hello-axion-7b8d9f-ghi56      1/1     Running   gke-axion-tutorial-axion-pool-c-...

All three pods should show node names containing axion-pool. None should show default-pool.

Confirm the Nodes Are ARM

Take one of those node names and verify its architecture label:

kubectl get node NODE_NAME --show-labels | grep kubernetes.io/arch

Replace NODE_NAME with one of the node names from the previous command. You should see:

kubernetes.io/arch=arm64

That's the automatic label GKE applied when it provisioned the ARM hardware. Our nodeSelector matched on this label to pin the pods here.

Ask the Application Itself

This is the most satisfying verification step. Our Go server reports the architecture of the binary that's running. Let's ask it directly.

Use kubectl port-forward to create a secure tunnel from port 8080 on your local machine to port 8080 on the Deployment:

kubectl port-forward deployment/hello-axion 8080:8080

This command stays running in the foreground — open a second terminal window and run:

curl http://localhost:8080

You should see:

Hello from freeCodeCamp!
Architecture : arm64
OS           : linux
Pod hostname : hello-axion-7b8d9f-abc12

Architecture : arm64. That's our Go binary confirming that it was compiled for ARM64 and is executing on an ARM64 CPU. The single image tag we built does the right thing automatically.

The Bonus: See the Manifest List in Action

Want to see the multi-arch image indexing at work? Stop the port-forward, then run:

docker buildx imagetools inspect \
  us-central1-docker.pkg.dev/PROJECT_ID/multi-arch-repo/hello-axion:v1

Replace PROJECT_ID with your actual Google Cloud project ID.

You'll see four entries in the manifest list. Two are real images — Platform: linux/amd64 and Platform: linux/arm64. The other two show Platform: unknown/unknown with an attestation-manifest annotation. These are build provenance records that Docker Buildx automatically attaches to every image — a supply chain security feature (SLSA attestation) that proves how and where the image was built.

You may notice that if you check the image digest recorded in a running pod:

kubectl get pod POD_NAME \
  -o jsonpath='{.status.containerStatuses[0].imageID}'

Replace POD_NAME with one of the pod names from earlier.

The digest returned matches the top-level manifest list digest, not the arm64-specific one. This is expected behaviour. Modern Kubernetes (using containerd) records the manifest list digest, not the resolved platform digest. The platform resolution already happened when the node pulled the correct image variant.

The definitive proof that the right binary is running is what you already have: the node labeled kubernetes.io/arch=arm64 and the application reporting Architecture: arm64.

Step 10: Cost Savings and Tradeoffs

The hands-on work is done. Let's talk about why any of this is worth the effort.

The Cost Math

At the time of writing, here's how ARM compares to equivalent x86 machines on Google Cloud (prices are approximate and change over time — check the official pricing page before making decisions):

That's a raw 25–30% reduction in compute cost per node. Factor in Google's published claim of up to 65% better price-performance for Axion on relevant workloads — meaning you may need fewer nodes to handle the same traffic — and the savings compound further.

Here's how that looks at scale, for a service running 20 nodes continuously for a year:

• 20 × n2-standard-4 × \(0.19 × 8,760 hours = \)33,288/year

• 20 × t2a-standard-4 × \(0.14 × 8,760 hours = \)24,528/year

That's roughly $8,760 saved annually on compute, before committed use discounts (which further widen the gap).

When ARM Is the Right Choice

ARM works best for:

• Stateless API servers and web applications — like the app we built. ARM excels at high-throughput, low-latency network workloads.

• Background workers and queue processors — long-running services that don't depend on x86-specific binaries.

• Microservices written in Go, Rust, or Python — these languages have excellent ARM64 support and are built cross-platform by default.

When to Proceed Carefully

• Native library dependencies — some older C libraries, proprietary SDKs, or compiled ML model-serving runtimes don't have ARM64 builds. Always audit your dependency tree before migrating.

• CI pipelines need ARM too — your automated tests should run on ARM, not just x86. An image that silently fails only on ARM is harder to debug than one that never claimed ARM support.

• Profile before optimizing — the cost savings are real, but measure your actual workload behavior on ARM before committing. Not every workload benefits equally.

Cleanup

When you're done, clean up to avoid ongoing charges:

# Remove the Kubernetes resources from the cluster
kubectl delete -f k8s/

# Delete the ARM node pool
gcloud container node-pools delete axion-pool \
  --cluster=axion-tutorial-cluster \
  --zone=us-central1-a

# Delete the cluster itself
gcloud container clusters delete axion-tutorial-cluster \
  --zone=us-central1-a

# Delete the images from Artifact Registry (optional — storage costs are minimal)
gcloud artifacts docker images delete \
  us-central1-docker.pkg.dev/PROJECT_ID/multi-arch-repo/hello-axion:v1

Conclusion

Let's recap what you built and why each part matters.

You started with a Go application, a Dockerfile, and a docker buildx build command that produced two images — one for x86, one for ARM64 — wrapped in a single Manifest List tag. Any server that pulls that tag gets the right binary automatically, without you maintaining separate pipelines or separate tags.

You provisioned a GKE cluster with two node pools running different CPU architectures, then used nodeSelector to make sure your ARM-optimized workload lands only on the ARM Axion nodes — not on x86 by accident. The result is a deployment that's both architecture-correct and cost-efficient.

The patterns you practiced here don't stop at this demo. The same Dockerfile technique works for any language with cross-compilation support. The same nodeSelector approach works for any workload you want to pin to ARM. As more teams migrate services to ARM over the coming years, having these skills will be a real asset.

Where to go from here:

• Add a GitHub Actions workflow that runs docker buildx build --platform linux/amd64,linux/arm64 on every push, automating this entire process in CI.

• Audit one of your existing stateless services for ARM compatibility and try migrating it.

• Explore Node Affinity as a softer alternative to nodeSelector for workloads that can run on either architecture but prefer ARM.

• Look into GKE Autopilot, which now supports ARM nodes and handles node pool management automatically.

Happy building.

Project File Structure

hello-axion/
├── app/
│   ├── main.go          — Go HTTP server
│   ├── go.mod           — Go module definition
│   └── Dockerfile       — Multi-stage Dockerfile
└── k8s/
    ├── deployment.yaml  — Deployment with nodeSelector and probes
    └── service.yaml     — LoadBalancer Service

All source files for this tutorial are available in the companion GitHub repository: https://github.com/Amiynarh/multi-arch-docker-gke-arm

In this tutorial, you’ll learn how to build and deploy a production-ready WhatsApp bot using:

• FastAPI

• Evolution API

• Docker

• EasyPanel

• Google Cloud Platform (GCP)

By the end of this guide, you will have a fully working WhatsApp bot connected to your own WhatsApp account and deployed on a cloud virtual machine.

Table of Contents

• How the Architecture Works

• How Your WhatsApp Bot Works

• Prerequisites

• Step 1: Create Firewall Rules on GCP

• Step 2: Create a Virtual Machine (Ubuntu 22.04)

• Step 3: SSH into the VM

• Step 4: Install Docker

• Step 5: Install EasyPanel

• Step 6: Open the EasyPanel Dashboard

• Step 7: Deploy Evolution API

• Step 8: Connect WhatsApp

• Step 9: Deploy the FastAPI Bot

• Step 10: Connect the Webhook - Telling Evolution API Where to Send Messages

• Step 11: Final Test

• Production Considerations

• Conclusion

How the Architecture Works

Before we start installing anything, let’s understand how the system works.

How Your WhatsApp Bot Works

Before we continue setting things up, let's make sure you understand what's actually happening behind the scenes. Don't worry – no technical experience needed here.

Imagine a postal service

Think of your WhatsApp bot like a very fast, automated postal service:

• Someone sends you a letter (a WhatsApp message)

• A postal worker (Evolution API) picks it up and brings it to your office

• Your office manager (FastAPI bot) reads it and writes a reply

• The postal worker takes the reply back and delivers it

That's it. That's the whole system.

The 7 steps

1. Someone sends a message to your WhatsApp number – just like texting a friend.

2. Evolution API notices the message – it's constantly watching your WhatsApp number for new messages, like a receptionist sitting by the phone.

3. Evolution API passes the message to your bot – it sends the message content to your app and says "hey, you've got a new message!"

4. Your bot reads the message and decides what to say – this is where your code does its job.

5. Your bot sends the reply back to Evolution API – "okay, send this response."

6. Evolution API delivers the reply through WhatsApp.

7. The user sees the reply on their phone – usually within seconds.

One line summary

User → WhatsApp → Evolution API → Your Bot → Evolution API → WhatsApp → User

Every step in this guide is just setting up one piece of that chain. Once they're all connected, the whole thing runs on its own automatically.

This architecture allows you to automate replies while keeping full control of your infrastructure.

Why These Tools?

Let’s briefly understand why we’re using each tool.

FastAPI

FastAPI is a modern Python framework for building APIs. It is fast, lightweight, and ideal for handling webhook requests from Evolution API.

Evolution API

Evolution API is a self-hosted WhatsApp automation server built on top of Baileys. It connects your personal WhatsApp account without requiring official WhatsApp Business API approval.

Docker

Docker allows us to run applications in containers. This makes deployments consistent, portable, and production-ready.

EasyPanel

EasyPanel is a graphical platform for managing Docker services. Instead of writing Docker Compose files manually, we use EasyPanel’s UI to deploy and manage our services easily.

Google Cloud Platform (GCP)

GCP provides the virtual machine that hosts our infrastructure. We will use an Ubuntu 22.04 server to run Docker, EasyPanel, Evolution API, and our FastAPI bot.

I chose these tools because they are practical, lightweight, and suitable for real-world production deployments.

Prerequisites

Before starting, make sure you have:

• A Google Cloud, AWS, or Azure account

• Billing enabled

• A project selected

• Access to Cloud Shell

• Basic Linux and Docker knowledge

Step 1: Create Firewall Rules on GCP

We need to allow traffic to specific ports on our VM. So, we run this command in GCP Cloud Shell:

gcloud compute firewall-rules create easypanel-whatsapp-fw \
 --network default \
 --direction INGRESS \
 --priority 1000 \
 --action ALLOW \
 --rules tcp:22,tcp:80,tcp:443,tcp:3000,tcp:8080,tcp:9000,tcp:5000-5999 \
 --source-ranges 0.0.0.0/0 \
 --description "SSH, EasyPanel, Evolution API, Bot"

This command:

• Creates a firewall rule named easypanel-whatsapp-fw

• On the default network

• Allows incoming internet traffic (INGRESS)

• Opens these ports:

  • 22 → SSH (server access)

  • 80 → HTTP

  • 443 → HTTPS

  • 3000, 8080, 9000 → App panels / APIs

  • 5000–5999 → Custom app range

• Allows access from any IP address (0.0.0.0/0)

Basically It opens your server so people (and you) can access your apps and services from the internet. This firewall rule allows external traffic to reach your VM.

Step 2: Create a Virtual Machine (Ubuntu 22.04)

Now we'll create the server that hosts everything. Run the following command in the GCP Cloud Shell to set up a virtual machine with Ubuntu 22.04.

gcloud compute instances create whatsapp-vm \
  --zone=asia-south1-a \
  --machine-type=e2-medium \
  --image-family=ubuntu-2204-lts \
  --image-project=ubuntu-os-cloud \
  --boot-disk-size=30GB \
  --tags=easypanel

This command creates a new virtual machine (VM) on Google Cloud:

• Name: whatsapp-vm

• Location (zone): asia-south1-a (India region)

• Machine size: e2-medium (2 vCPU, 4GB RAM)

• Operating System: Ubuntu 22.04 LTS

• Disk size: 30GB

• Tag: easypanel (used to apply firewall rules)

This creates a Linux server in Google Cloud that you can use to host EasyPanel, WhatsApp bot, or your APIs.

Note: Wait about one minute for the instance to start.

Step 3: SSH into the VM

Connect to your server by using SSH to access the virtual machine you just created on Google Cloud.

gcloud compute ssh whatsapp-vm --zone=asia-south1-a

This command connects to your virtual machine named whatsapp-vm in the zone asia-south1-a using SSH (secure remote login).

It logs you into your Google Cloud server so you can start installing software and running commands. After running this, you will see a terminal prompt – that means you are now inside your Ubuntu server and ready to go.

Step 4: Install Docker

Docker is needed to run EasyPanel and the Evolution API.

First update the system:

sudo apt update -y
sudo apt install -y curl

This does two things:

1. sudo apt update -y→ Updates your server’s package list (refreshes available software info).

2. sudo apt install -y curl→ Installs curl, a tool used to download things from the internet using the terminal.

It prepares your server and installs a tool needed to download and install other software.

Then install Docker:

curl -fsSL https://get.docker.com | sudo sh

This command uses curl to download Docker’s official installation script. The | (pipe) sends it directly to sudo sh, which runs the script as administrator.

It automatically installs Docker on your server.

After this finishes, Docker should be installed.

Enable Docker:

sudo systemctl enable docker
sudo systemctl start docker

This command does two things:

1. enable docker→ Makes Docker start automatically every time the server reboots.

2. start docker→ Starts Docker right now.

It turns Docker ON now and makes sure it stays ON after restart.

Allow the Ubuntu user to run Docker:

sudo usermod -aG docker ubuntu

This command adds the user ubuntu to the Docker group.

This is important: By default, you must use sudo before every Docker command.After running this, the Ubuntu user can run Docker without needing sudo every time.

Note: This command assumes your username is ubuntu, which is the default on Google Cloud VMs. If your username is different, replace Ubuntu with your actual username.

Exit the session and reconnect:

exit
gcloud compute ssh whatsapp-vm --zone=asia-south1-a

1. exit→ Logs you out of your current server session.

2. gcloud compute ssh whatsapp-vm --zone=asia-south1-a→ Logs you back into your Google Cloud VM.

Why we do this: After adding the ubuntu user to the Docker group, you must log out and log back in for the permission changes to work.

Test Docker:

docker run hello-world

This command downloads a small test image called hello-world, runs it inside Docker, and prints a success message if Docker is working correctly.

It checks if Docker is installed and working properly. If you see “Hello from Docker!”, Docker is working correctly.

Step 5: Install EasyPanel

EasyPanel provides a user interface for deploying Docker services. Run this command in the VM:

curl -sSL https://get.easypanel.io | sudo bash

This command:

• Downloads the official EasyPanel installation script

• Runs it with administrator (sudo) permission

• Automatically installs and configures EasyPanel on your server

It installs EasyPanel on your VM so you can manage apps using a web dashboard instead of commands. Installation takes about one minute.

Step 6: Open the EasyPanel Dashboard

Once you have your IP address, open a new tab in your browser and type it in like this:

http://<YOUR_PUBLIC_IP>:3000

For example, if your IP was 34.123.45.67, you would type:

http://34.123.45.67:3000

EasyPanel runs on port 3000 by default – that's why we add :3000 at the end. Without it, your browser won't know which service to open on the server.

Create an admin account and log in; the EasyPanel login page will appear.

Click “Create Admin Account”.

Fill in:

• Username (choose something you’ll remember)

• Email

• Password (make it strong!)

• Submit the form.

You are now logged in as the admin and can start managing apps, APIs, and bots through the EasyPanel dashboard.

You will see a page like the one below:

Step 7: Deploy Evolution API

1. Create a new project (for example: whatsapp-1)

2. Go to Services → Templates

3. Select Evolution API

4. Deploy the latest version

Wait until all services turn green. You will see a page like the one below.

Next, open Environment Variables and locate:

AUTHENTICATION_API_KEY

Copy the AUTHENTICATION_API_KEY.

Open the Evolution API dashboard

Inside EasyPanel, find your Evolution API service. You will see a clickable domain link – it usually looks something like:

https://evolution-api.easypanel.host

Click that link to open it in your browser. You will see a JSON response confirming the service is running.

Once you open the link, you’ll see a JSON response confirming success. To proceed with login, copy the Manager link displayed in the response. This link opens the management dashboard where you can authenticate and begin using the Evolution API. The screenshot below highlights the manager URL along with version details for easy reference

Copy the manager link and open it in a new tab, then copy the AUTHENTICATION_API_KEY, which you did in the previous step. This is how it looks, as you can see below:

Create a new instance:

• Choose channel: Baileys

• Leave phone number blank

• Give your instance a name

Save the instance.

Step 8: Connect WhatsApp

Inside your instance dashboard:

1. Click Get QR

2. Scan it using WhatsApp on your phone

Once connected, your chats and contacts will sync automatically. If syncing fails, disconnect and reconnect the session.

Step 9: Deploy the FastAPI Bot

Now we’ll deploy the bot service.

1: Go to EasyPanel

You’re opening the EasyPanel dashboard you just installed. This is where you can manage apps, servers, and services using a graphical interface instead of terminal commands.

2: Create a new project

A “project” is like a container or folder for your bot service. It organizes all files, settings, and deployments for this app.

3: Add an App service

“App service” means a running instance of your application. In this case, it will be the WhatsApp bot.

4: Choose Git deployment

Git deployment lets you connect a code repository to EasyPanel.This will automatically download your code from GitHub and run it inside Docker.

5: Paste your repository URL

https://github.com/rajumanoj333/wabot

This is the GitHub repository containing the WhatsApp bot code. EasyPanel will clone this repo and prepare the app automatically.

6: Domains in EasyPanel

This section lets you assign a URL or domain name to your app service. Even if you don’t have a custom domain, you can use your server’s public IP. Your WhatsApp bot app runs on port 9000 inside the server.

7: Set the port to 9000

By setting the domain to use port 9000, EasyPanel knows where to send traffic.

Example URL after this step:

https://your-project.easypanel.host

This is the public address people (and other services) will use to reach your bot.

You’re telling EasyPanel:

“Whenever someone accesses this project, forward them to the bot service running on port 9000.”

Without this step, the bot service would run but you wouldn’t be able to access it from your browser or other apps.

Configure Environment Variables

Set the following variables:

EVOLUTION_API_URL=http://evolution-api:8080
EVOLUTION_API_KEY=YOUR_AUTHENTICATION_API_KEY
INSTANCE_NAME=your_instance_name

Note: You might notice two different names here – AUTHENTICATION_API_KEY (used in EasyPanel) and EVOLUTION_API_KEY (used in your bot code). They are the same key. Just copy the value from EasyPanel and paste it into both places.

Step 10: Connect the Webhook – Telling Evolution API Where to Send Messages

At this point, you have two separate things running:

1. Evolution API: the service that connects to WhatsApp and handles messages

2. Your app (fastapi bot): the chatbot brain you deployed in the previous steps

Right now, these two don't know each other exists. They're like two people in different rooms with no way to pass notes between them. A webhook fixes that.

So what exactly is a webhook?

A webhook is simply a URL (a web address) that you hand to one service so it can automatically notify another service when something happens.

You're going to tell Evolution API "whenever a WhatsApp message arrives, forward it to this address." Your app will be sitting at that address, waiting to receive it, read it, and send a reply.

Think of it like a forwarding address at the post office. When mail (a WhatsApp message) arrives, it gets automatically redirected to your app's door.

Let's set it up

1. Open your Evolution API dashboard.

You should already have this open from earlier steps. In the left sidebar, click on Events, then click on Webhook. This is where you control how Evolution API sends data to your app.

2. Turn the webhook on.

At the top of the page, you'll see a toggle next to the word "Enabled". Click it so it turns green. This tells Evolution API that you want to start using a webhook.

3. Enter your app's webhook URL.

In the URL field, type your app's address with /webhook added to the end, like this:

https://your-domain.easypanel.host/webhook

Replace your-domain with the actual domain name you set up when you deployed your app. The /webhook part at the end is important: it's a specific page your app has set up just for receiving these messages. Without it, Evolution API would be knocking on the wrong door.

4. Leave "Webhook by Events" and "Webhook Base64" turned off for now.

These are advanced options you won't need for a basic chatbot.

5. Scroll down to the Events section and enable these two events:

• MESSAGES_UPSERT: This triggers every time someone sends your WhatsApp number a message. Without this, your app would never know a message arrived.

• SEND_MESSAGE: This triggers when a message is sent out. It helps your app confirm that replies are going through correctly.

You can leave all the other events (like APPLICATION_STARTUP) turned off. They handle things like group chats and contact updates, which aren't needed for what we're building.

6. Click Save.

Quick recap of what you just did

You created a direct line between Evolution API and your app. Now, the moment someone messages your WhatsApp number, Evolution API will instantly pass that message along to your app. Your app reads it, figures out a response, and sends one back all automatically.

This is the step that brings your chatbot to life. Without it, nothing would happen when someone sent you a message. With it, the whole system clicks into place.

Step 11: Final Test

Send a message from a different WhatsApp number (not the connected one).

Send:

Hi

If everything is configured correctly, your bot should reply:

👋 Hello! Bot is working.

Congratulations! Your WhatsApp bot is now live.

Production Considerations

For real-world deployments, consider:

• Restricting firewall rules instead of allowing 0.0.0.0/0

• Using HTTPS with a custom domain

• Securing API keys with a secret manager

• Monitoring logs and container health

• Setting up automatic backups

This tutorial demonstrates the core working system, but these improvements will make your deployment more secure and scalable.

Conclusion

You now have a fully self-hosted WhatsApp bot running on a cloud VM using FastAPI, Evolution API, Docker, EasyPanel, and GCP.

This setup gives you:

• Full control over infrastructure

• No dependency on expensive SaaS platforms

• Production-ready container deployment

• Scalable architecture

From here, you can extend your bot with:

AI integrations : connect your bot to ChatGPT or Gemini or Claude so it can answer questions intelligently instead of just sending fixed replies.

Database storage: save incoming messages, user details, or conversation history to a database like PostgreSQL or MongoDB.

Custom automation workflows trigger actions based on keywords, like sending a PDF when someone types "menu" or booking an appointment when they type "schedule".

CRM integrations :connect your bot to tools like HubSpot or Notion to automatically log leads and customer conversations.Building your own infrastructure is one of the best ways to deeply understand how modern backend systems work together.

Happy building!

Like, what in the world – out of every other possible name, they decided to name a database after a literal cockroach? 🤣

I mean, I get it: cockroaches are known for being resilient, and the devs were probably trying to say “our database never dies”… but still…a cockroach?

The name aside, out of all the databases out there, you might be wondering why would you choose CockroachDB? And if you did choose it, where would you even start when trying to host and deploy it? Would you go for a managed cloud service? Or could you actually self-manage it?

If you ever thought of doing it yourself – maybe in a dev environment, or even introducing it to your company – how would you go about it?

Well, just calm your nerves 😄

In this book, we’ll explore everything you need to know about deploying and managing CockroachDB on Kubernetes. We’ll dive deep into:

• Understanding how CockroachDB’s masterless (multi-primary) architecture actually works

• Setting up and deploying CockroachDB on a Kubernetes cluster

• Automating backups to Google Cloud Storage using just a few queries in the CockroachDB cluster

• Managing service accounts and authentication securely

• Tuning CockroachDB’s memory settings for stable performance

• Scaling the cluster horizontally and vertically without downtime

• Monitoring and maintaining the database like a pro

By the end, you’ll not only understand how CockroachDB works, you’ll be confident enough to deploy and manage your own resilient, production-ready instance. 🚀

Table of Contents

1. What Even Is CockroachDB? 🤔

   • Simple Definition

   • Who Made CockroachDB? When Was it Released?

   • What Problems Does CockroachDB Try to Solve?

   • Key Terms You Should Know (in plain language):

   • Why the name “CockroachDB”? 😅

2. Why Choose CockroachDB Over PostgreSQL or MongoDB 🤷🏾‍♂️?

   • How Fault Tolerance is Handled in PostgreSQL and MongoDB

   • How CockroachDB Handles It Differently

3. How CockroachDB Works Behind the Scenes ⚙️

   • Ranges: The Small Pieces of Data

   • Replication: Many Copies for Safety

   • Raft Consensus: How All Copies Agree

   • MultiRaft: Keeping Raft Efficient When Things Scale

   • Rebalancing: Movement for Balance

   • Distributed Transactions: Doing Work Across Multiple Ranges

   • How It All Fits Together: Read + Write Flow (What Happens When You Use It)

   • Why This All Matters (Putting It in Plain English)

4. Where (and How) Should You Host CockroachDB? ☁️

   • Option 1: CockroachDB Cloud (fully managed by Cockroach Labs)

   • Option 2: Bring Your Own Cloud (BYOC)

   • Option 3: Use Cloud Marketplaces (AWS, GCP, Azure)

   • Option 4 (My Favorite 😁): Self-Hosting — Especially Using Kubernetes

5. Setting Up Your Local Environment 🧑‍💻

   • Why these tools?

   • Step 1: Install Minikube

   • Step 2: Install kubectl

   • Step 3: Install Helm

6. Deploying CockroachDB on Minikube (The Fun Part Begins 😁!)

   • Step 1: Visit ArtifactHub

   • Step 2: Explore the Helm Chart

   • Step 3: Copy the Default Values

   • Step 4: Create a Folder for Our Project

   • Step 5: Understanding the Key Configurations

   • Step 6: Create a Simplified Values Config for the CockroachDB Helm Chart

   • Overview of the YAML values

   • 🚀 Step 7: Install the CockroachDB Cluster Using Helm

7. Accessing the CockroachDB Console & Viewing Metrics

   • Step 1: Locate the CockroachDB Public Service

   • Step 2: Learn More About the Service

   • Step 3: Access the CockroachDB Dashboard

   • Step 4: Visit the Dashboard

   • Step 5: Exploring the Metrics Dashboard

   • Step 6: Creating a Little Load on the CockroachDB Cluster

   • Step 7: Viewing the Metrics from the Load

   • Step 8: View the List of Created Items in the Database

8. Backing Up CockroachDB to Google Cloud Storage ☁️

   • Why Backups Are Absolutely Critical

   • Connecting to Our DB – Installing Beekeeper Studio

   • How to Install Beekeeper Studio

   • Connecting Beekeeper Studio to CockroachDB

   • Exposing the Cluster for Local Access

   • 🐝 Connecting via Beekeeper Studio

   • Verify the Connection

   • Creating a Google Cloud Account

   • Creating a Google Cloud Storage Bucket

   • Giving CockroachDB Access to the Bucket

   • Attaching the Key to Our CockroachDB Cluster

   • Testing Our Backup — Disaster Recovery Time

9. Managing Resources & Optimizing Memory Usage

   • How CockroachDB Uses Memory

   • The Memory Usage Formula You Must Follow

   • Where You Find These Settings

   • Concrete Example (Step-by-Step)

   • ⚠️ On Requests vs Limits in Kubernetes

   • Overriding the Default Fractions

10. Scaling CockroachDB the Right Way

    • Key Metrics to Understand

    • When (and What) to Scale Based on Your Metrics

    • Disk-Bound Situations — What to Do When Your Disk Is the Limiting Factor

    • Memory Pressure — What to Do When Your Database Hits the Limit

    • When Queries Are Slow but Everything Else (CPU, Memory & Disk) Looks “Fine”

    • Understanding Disk Speed (IOPS & Throughput) Across Cloud Providers

    • Downsizing the Cluster (Reducing Replicas)

    • ⚠️ The Wrong Way to Downscale

    • Decommissioning a Node Before Scaling Down the Cluster

11. What to Consider When Deploying CockroachDB on Google Kubernetes Engine (GKE) ☁️

    • Creating Your GKE Cluster

    • Connecting to your GKE cluster

    • Deploying CockroachDB in Production (on GKE)

    • Understanding the Configuration

    • Installing the CockroachDB Cluster on GKE

    • Connecting to Our CockroachDB Cluster (Now That TLS + mTLS Are Enabled)

    • Connecting via Mutual TLS (mTLS) — Why We Need a Certificate for Our root User

    • Let’s Explore Our Cluster’s Certificate

    • Understanding the Certificate Sections (Explained Super Simply)

    • Creating a Client Certificate (So We Can Finally Connect to CockroachDB)

    • Connecting to Our CockroachDB Cluster Securely (Using mTLS)

    • Restoring Our Previous Database into the New GKE CockroachDB Cluster (without SA keys)

    • Restoring Our Previous Database from Google Cloud Storage

    • Now, Let’s Restore the Data 🎉

    • Connecting to the Database with a New User

    • Connecting with Passwordless Authentication (Mutual TLS)

    • Connecting via Mutual TLS (mTLS) from Our Apps on Kubernetes

12. How to Get a CockroachDB Enterprise License for FREEE!

    • Three Types of Licenses

    • How to Apply for the Free Enterprise License

    • Adding Your License to the CockroachDB Cluster

13. Conclusion & Next Steps ✨

    • About the Author 👨🏾‍💻

What Even Is CockroachDB? 🤔

Hey! before we jump into setting up our Kubernetes cluster and deploying our CockroachDB cluster, let’s get grounded in what CockroachDB really is. (Because if you don’t understand the why and how, the implementation and practical session will just feel like magic 😅.)

Simple Definition

CockroachDB is a distributed SQL database. This means it gives you the features of a relational database (tables, SQL queries, JOINS, transactions) but copies data across multiple replicas (servers, nodes, instances). No need for sharding manually. 😃

It’s built to survive failures, scale easily (compared to other SQL databases), and keep your data consistent no matter what (across all the instances).

Who Made CockroachDB? When Was it Released?

CockroachDB was created by Cockroach Labs, founded by Spencer Kimball, Peter Mattis, and Ben Darnell. The idea first started taking shape around 2014, and by 2015 Cockroach Labs was formally founded.

Its 1.0 “production-ready” version was announced in 2017, marking its transition from beta to being suitable for real-world use.

What Problems Does CockroachDB Try to Solve?

Traditional relational databases are great, but they run into real challenges when your app grows. CockroachDB was built to solve those. Here are the key pain points and how CockroachDB addresses them:

So instead of fighting your database as it grows, CockroachDB handles much of the hard work for you.

Key Terms You Should Know (in plain language):

• Node: Duplicates or copies of your database. These are also known as replicas. They can be read-only (databases from which data can only be read, for example using SELECT statements), OR read-write (databases from which data can be read, created, updated, and deleted).

• Replication: making copies of data on multiple nodes. If one node fails, others still have the data.

• Raft (consensus algorithm): a system that ensures copies (replicas) agree on changes in a safe, reliable way. For example, when you want to write data, Raft ensures that most copies agree before it’s accepted.

• Sharding / Ranges: Instead of putting all your data in one big blob, CockroachDB splits it into smaller chunks called ranges. Each range is replicated and can move between nodes.

• Distributed transaction: a transaction (series of operations) that might touch data stored in different nodes. CockroachDB manages this, so you still get ACID (atomic, consistent, isolated, durable) properties.

Why the name “CockroachDB”? 😅

You might wonder: Why name a database after a cockroach? It sounds weird at first, but there's a reason:

Cockroaches are known for surviving harsh conditions: radiation, natural disasters, and so on. The founders wanted a database that feels almost “impossible to kill,” that can survive node failures, outages, and network splits. The name is a tongue-in-cheek nod to resilience.

Why Choose CockroachDB Over PostgreSQL or MongoDB 🤷🏾‍♂️?

Let’s compare the classic setup (Postgres / MongoDB) to CockroachDB, especially why you might want to go with CockroachDB, and how it helps ease scaling. I’ll also explain some terms to make sure you’re following.

In many setups, when you use Postgres or MongoDB, you’ll often have one “primary” node that handles all writes (that is, inserts, updates, deletes).

Then you have multiple “read replicas” that copy the primary’s data and serve read requests (selects). That works okay – reads can be spread out – but all write traffic goes to that one primary node.

Usually, the primary eventually gets stressed when the write volume grows (for example, more customers create accounts and products on your platform).

You can add more read replicas (horizontal scaling for reads, for example customers trying to view their accounts, or previously created products on your site), but scaling the primary is much harder.

To scale the primary, you often resort to upgrading its resources (CPU, RAM, disk) – that’s vertical scaling – which often needs downtime (shut down the primary database, increase its CPU and RAM, then spin it back up).

Or you’d have to manually shard (split) your data across multiple primaries, route traffic carefully, and manage complexity.

How Fault Tolerance is Handled in PostgreSQL and MongoDB

When you try to make Postgres (or MongoDB) highly available and fault tolerant in a self-managed setup, you often need two+ read replicas and one primary.

The tricky part is handling what happens when the primary fails (or is taken down temporarily for an upgrade). You need something that can promote a replica to a primary automatically.

In Postgres land, that’s often handled by Patroni or repmgr (tools that handle cluster management, failover, leader election, and so on).

In MongoDB, such logic is part of the replica set behavior: it does automatic elections among replicas.

Here are some of the core challenges with that classic model:

• Every write must go to a single primary. If that primary fails or is overloaded, your whole system suffers.

• Scaling reads is easy (add more replicas), but scaling writes is hard.

• Vertical scaling (give more resources to one server) has its cons. If the primary node needs more resources, you might experience some downtime when it’s being scaled up.

• Manual sharding is messy: you decide which piece of data goes to which shard, handle cross-shard queries, and build routing logic. That’s a lot of maintenance and can lead to unexpected issues if not handled properly.

• One service (or load balancer/proxy) points to the primary (for ALL write queries).

• Another service or routing logic handles read queries and can share reads across replicas.

• You might use HAProxy, pgpool-II, or pgBouncer for Postgres to route traffic, do read/write splitting, or manage connection pooling. These are external (not part of the database core) tools you have to configure.

So when the primary fails, Patroni (or repmgr, and so on) will detect it and promote one of the read replicas to be the new primary.

But that promotion, reconfiguration, and traffic rerouting often cause a brief window of downtime (when your primary database node becomes unavailable).

How CockroachDB Handles It Differently

CockroachDB changes the rules:

• All replicas are equal for reads and writes. You don’t have a special “primary” that handles writes. Every node in the cluster can accept write requests.

• CockroachDB breaks your data into small chunks (ranges) and replicates them across nodes. If you add a new node, data moves around automatically to balance the load.

• Every write is automatically copied to other replicas, and consistency is managed by a protocol (Raft), so you don’t have to build this yourself.

• No manual sharding needed. Because the database handles how data is split and moved, you don’t need to decide how to shard by hand.

• You don’t need a special service to route writes vs reads queries. Any node can accept both reads and writes.

• During scaling, you don’t have to worry about which node is the primary – because there is no primary.

• You can scale your nodes one at a time (rollout style). When one node is being upgraded, the others continue to serve traffic. You won’t hit a downtime window just because you're scaling the “primary.”

• Because there's no replica promotion logic to fight with, there's no moment where a replica needs to be “elevated” to primary – it’s all just nodes continuing to serve.

How CockroachDB Works Behind the Scenes ⚙️

In CockroachDB, there are many moving parts behind the scenes. But they work together, so you don’t have to babysit them. The core ideas, which we’ve mostly already touched on, are:

• Splitting data into pieces (ranges)

• Keeping multiple copies of each piece (replicas/replication)

• Making sure all copies agree via Raft consensus

• Moving pieces around to balance the load (automatic rebalancing/distribution)

• Coordinating transactions that might touch many pieces

Let’s go through each of those, one by one.

Ranges: The Small Pieces of Data

Imagine you have a giant book of recipes. If you try to carry the whole thing, it’s heavy. So you split the book into smaller booklets, each covering recipes for a certain range of meals: breakfasts, lunches, dinners, desserts.

In CockroachDB, data is split into ranges, which are like those smaller booklets:

• Each range covers a certain block of data (like “all users whose ID is 1-1000”)

• When a range gets too big (like having too many recipes in one booklet) it’s cut/split into two smaller ones. That makes each piece easier to manage.

• If two neighboring ranges have become very small (few recipes), they might be merged (joined) back together so you’re not keeping too many tiny booklets.

• These splits and merges happen automatically, behind the scenes, so the database stays smooth as things grow or shrink.

This chopping helps the system in many ways: moving pieces, copying them, balancing load, recovering from node failures becomes easier.

Replication: Many Copies for Safety

Nobody likes losing their work, so you keep backup copies. CockroachDB does this for data as well.

For each range, there are usually 3 copies (replicas) stored on different machines (nodes). If one machine dies, you still have others. (cockroachlabs.com). And these copies are always kept in sync: when you write something (for example, insert or update), the change is propagated to the other copies.

The database also tolerates failures. If one node goes down, the system detects it and eventually makes a new copy elsewhere to replace it. So the target number of copies is maintained. This gives you fault tolerance: your data stays safe even when parts of your system fail.

Raft Consensus: How All Copies Agree

Having copies is useful, but you also need them to agree with each other – like all your recipe booklets have the same content in each copy. The Raft protocol is a way to make sure that happens reliably.

Here’s how Raft works in simple terms:

• Each range has a group of replicas. One of these replicas acts as the leader. Others are followers.

• All write requests for that range go through the leader. The leader gets the request, then tells followers to record the same change.

• Once most of the copies (a majority) say “yep, we got it,” the change is considered final (committed). Then the leader tells the client, “Done.”

• If the leader stops working (the machine dies or the network fails), the followers notice it (they stop getting regular “I’m alive” messages), then they hold an election to pick a new leader, and the show goes on.

• This way, the system ensures everyone has the same final data and no conflicting changes happen.

So Raft is the agreement protocol that keeps all copies in sync and safe.

MultiRaft: Keeping Raft Efficient When Things Scale

When you have many ranges (many pieces of the booklets), each range has its own Raft group. That can mean a lot of “are you alive?” messages between nodes, and a lot of overhead. MultiRaft is the trick CockroachDB uses to make this efficient.

MultiRaft groups together Raft work for many ranges that share nodes, so overhead is reduced. Instead of sending separate heartbeat (are you alive?) messages for each range, some of the messages are bundled.

This reduces network chatter and resource waste and helps the database scale smoothly when you have tons of data and many pieces.

Rebalancing: Movement for Balance

When your ranges are not evenly spread across nodes (machines), some machines are doing way too much work, and some hardly any. That’s not good. So CockroachDB automatically moves pieces around to balance things.

• The system watches how busy each node is (how many ranges it holds, how much data, how much read/write traffic).

• If one node is overloaded, it will move some ranges to other nodes.

• If a node dies, the system notices and makes sure that ranges that were on that node get copied somewhere else so safety (replica count) is maintained.

• If you add a new node, the system starts moving ranges to the new node so its resources are used.

This happens without you having to manually decide “move this here, move that there.”

Distributed Transactions: Doing Work Across Multiple Ranges

Often, an operation touches multiple ranges. For example, “transfer money from account A (in range 1) to account B (in range 2)”. That must be handled carefully so that either both parts succeed, or neither do.

CockroachDB supports distributed transactions, meaning a single transaction can work across many ranges. It uses “intent” writes (temporary placeholders) and once everything is ready, it commits the transaction so it becomes permanent. If something fails, it aborts (cancels) the whole thing. The system ensures atomic behavior: all or nothing.

How It All Fits Together: Read + Write Flow (What Happens When You Use It)

Let’s picture a write, step by step:

1. Your app sends a write (for example, “add new user”) to any node in the CockroachDB cluster.

2. That node figures out which range(s) are involved (which pieces hold the data you want to write).

3. For each range, the write goes to that range’s leader.

4. The leader writes the change to their own copy, then tells followers to do the same.

5. Once most copies confirm they have the change, the leader declares it “committed” and tells your app, “yes, write done.”

6. If a node is busy or down, others still handle traffic.

Read flow:

• Your app sends a read (for example “get user by ID”) to any node.

• That node checks its copies. If it has a fresh copy, it answers. If not, it asks the node that does.

Everything works so data is correct, up to date, and reliably available even if machines fail or network lags.

Why This All Matters (Putting It in Plain English)

All these tweaks are important for several key reasons. First of all, because data is chopped into ranges and replicated, no single node is a bottleneck. Also, Raft ensures consensus, so you can trust that data is consistent across all working replicas.

Beyond this, rebalancing is automatic, you don’t have to micromanage shards or worry about nodes drowning in load. And because transactions that touch multiple ranges are coordinated, you can trust ACID properties even in a distributed setup.

Where (and How) Should You Host CockroachDB? ☁️

There isn’t just one “right” way to host CockroachDB. There are a few paths you can pick, each with pros and cons. What you pick depends on cost, control, ease of use, and your risk tolerance.

In this section, we’ll explore:

• Cockroach Labs’ own managed cloud (CockroachDB Cloud)

• “Bring Your Own Cloud” (BYOC) – letting Cockroach Labs manage it inside your cloud account

• Hosting via cloud marketplaces (AWS, GCP, Azure)

• Self-hosting / Kubernetes / your own infrastructure

• And notes on DigitalOcean support

Let’s dive in.

Option 1: CockroachDB Cloud (fully managed by Cockroach Labs)

This is the easiest option if you want to offload operations. You don’t manage nodes (computers, Virtual machines, and so on), upgrades, or backups, as Cockroach Labs handles all that.

What it offers:

• You sign up and click “create cluster.”

• Automatic scaling, zero-downtime upgrades, and managed backups.

• It supports multiple cloud providers behind the scenes (you pick region(s)).

• You get tools, APIs, and Terraform integration to automate it.

• They often give free credits to get started.

Tradeoffs:

• You have less control over underlying infrastructure, for example Virtual Machines, networking, disks, and so on (you trade control for convenience).

• You pay for the managed service premium.

• You rely on Cockroach Labs’ SLAs, uptime, and support.

If you want, you can check it out here: CockroachDB Cloud (managed by Cockroach Labs).

Option 2: Bring Your Own Cloud (BYOC)

This is a middle ground: you keep your cloud environment, but let Cockroach Labs manage the database. It gives you control over infrastructure, billing, network, and so on, while still offloading operational complexity.

How it works:

• You run CockroachDB Cloud inside your cloud account (AWS, GCP, and so on).

• Cockroach Labs still handles provisioning, upgrades, backups, and observability. You manage roles, networking, and logs.

• Useful for complying with regulations, keeping data within your cloud folder/account, and using your cloud discounts.

Tradeoffs:

• You still need to set up cloud aspects (VPCs, IAM, roles) correctly.

• There’s more complexity than pure managed, but more control as well.

• Cockroach Labs needs access to certain parts of your account (permissions).

If you want to explore BYOC, you can read more here: CockroachDB Bring Your Own Cloud.

Option 3: Use Cloud Marketplaces (AWS, GCP, Azure)

If you already use a cloud provider, sometimes the easiest way is to deploy via their marketplace offerings. It gives you familiarity, billing simplicity, and so on.

• GCP Marketplace – CockroachDB is available on the Google Cloud Marketplace, making it easier to deploy within your GCP environment. You can learn more here: GCP Marketplace.

• AWS Marketplace – CockroachDB is listed there: AWS Marketplace.

• Azure Marketplace – Also supported for Azure deployments (SaaS/managed listings): Azure Marketplace.

• DigitalOcean – There is support for CockroachDB deployment on DigitalOcean using their infrastructure: Deploy CockroachDB on DigitalOcean.

These options let you stay in your cloud console, use your existing cloud accounts, and integrate with other resources you already have.

But you're still responsible for certain operational tasks (networking, security, monitoring, backups) depending on how the marketplace offering is configured.

Option 4 (My Favorite 😁): Self-Hosting — Especially Using Kubernetes

If you self-host CockroachDB, you get full control. You’re the boss of everything: the machines, storage, networking, backups, upgrades, monitoring – all of it.

What’s even better is that using Kubernetes means your setup isn’t tied to one cloud provider. You can run it on AWS, GCP, Azure, or even on-premises later, with very little change. Kubernetes gives you a “portable infra” layer.

Managed CockroachDB services charge you extra for “maintenance, upgrades, backup, etc.” – those are baked into the price. But when you self-host, you accept the burden, but also avoid paying that extra margin. You pay for compute, disks, network, and your time/ops work.

You can also self-host in the cloud (using cloud VMs) but still manage every layer: disks, network, security, and so on. Using Kubernetes, there is a sweet middle ground: you get cloud reliability for VMs, but you fully control everything above that.

Why Kubernetes Beats Tools Like Docker Swarm or Hashicorp Nomad for Databases

Because CockroachDB is a stateful system (it holds data), you need strong support for “data that stays even when a pod restarts or moves.” Kubernetes is designed with good primitives for that. Other tools don’t always shine there.

Here’s the comparison in simple terms:

• Docker Swarm / Docker Compose: Great for stateless apps (web servers, APIs), but when it comes to databases, it struggles. Swarm doesn’t natively support persistent volume claims at a cluster level, so if a container (database replica) moves to a different node (VM), it might lose access to its storage. Devs often pin containers to specific nodes manually to avoid this.

• Nomad: More flexible and simpler in some ways, but it’s not as rich in features around connectivity, storage management, and built-in tooling for containers. It works well in mixed workloads, but handling complex databases usually means you need to build extra layers.

• Kubernetes: It has built-in support for stateful workloads:

  • StatefulSets (Properly managing data for each database): This ensures that each CockroachDB replica (pod) keeps its identity and storage intact even if the pod restarts. So the database replica doesn’t lose its “name” or data when things change.

  • Persistent volumes and persistent volume claims (external disks): These are like dedicated hard drives or disks attached to pods (database replicas). Even if a pod moves, crashes, or restarts, the disk (data) stays. Kubernetes makes sure the data stays safe.

  • StorageClasses (choose your disk): You can customize the disks in which your data will be stored, that is:

    • HDD (most affordable, but slower),

    • Balanced Disk (SSD enabled, a balance between costs and speed),

    • Fast SSD (Very fast, recommended by the CockroachDB team, but a bit more expensive than a Balanced Disk).

    • Rolling updates, anti-affinity, (No Downtime, High Availability, Fault tolerance).
      Anti-affinity means you can tell Kubernetes, “don’t put more than one CockroachDB replica on the same VM or physical machine.” This protects you if one VM goes bad, other replicas are safe.

    • Rolling updates let you update one replica at a time (configuration, version, resources) without bringing down the whole cluster. While one replica updates, others serve traffic. That helps avoid downtime.

    • Kubernetes also has ordered start/stop for replicas (via StatefulSets) so things are predictable and safe

  • Vertical vs horizontal scaling (earlier talk – reminder)
    You remember we talked about scaling in prior sections:

    • Horizontal scaling means adding more replicas (more pods, more nodes) so load spreads out.

    • Vertical scaling means increasing the resources (CPU, RAM, disk) of existing nodes/replicas.

In tools like Nomad or Docker Swarm, vertical scaling tends to be harder, often involves stopping services, shutting things down, and restarting VMs, which causes downtime.

Kubernetes makes vertical and horizontal scaling easier at the pod level (you can resize one pod CPU + RAM) and manage rolling upgrades so you don’t take everything down at once.

You can also add more database replicas to the cluster easily (to balance load and make the database process queries faster), and the data is automatically copied to the new database replica (replication), especially when you use the official CockroachDB Helm Chart.

Why Other Tools (Swarm / Nomad / Docker Compose) Don’t Match Up Here

Docker Swarm and Docker Compose are simpler to use and are good when you don’t have much complexity. But they lack robust features for stable storage, default support for replication, vertical scaling, horizontal scaling of stateful services, and so on. For example, Swarm doesn’t have built-in StatefulSets or dynamic volume provisioning like Kubernetes.

Nomad is more flexible than Swarm in some ways, but many users say storage plugins (CSI) are weaker than what Kubernetes has. Also, less built-in for ordering things, rolling updates for stateful apps.

So while these work fine for simpler apps (stateless services, small apps), when you have a distributed stateful SQL database like CockroachDB, Kubernetes gives you more safety, more control, less chance of data loss or misconfiguration.

Because of all this, running CockroachDB on Kubernetes gives you the tools you need baked in, reducing how much custom plumbing you must write yourself.

Trade-offs (things to watch out for)

• You have to manage everything: backups, monitoring the ENTIRE CockroachDB cluster, withstanding failures (fault tolerance), and upgrades. That’s work 🥲.

• You need to know your way around infra (VMs, disks, networking, and inter-node connections) and operations (or have teammates who do – DevOps Engineers, Cloud Architects, Site Reliability Engineers).

• Using managed Kubernetes (like GKE, EKS, AKS) helps as you offload the control plane. You still manage the nodes, storage, and higher layers.

• But even with that, you avoid paying for “database management as a service” markup – you're only paying for infrastructure plus your time.

Setting Up Your Local Environment 🧑‍💻

Alright, we’ve learned quite a bit so far: what CockroachDB is, how it works behind the scenes, and where you can host it. Now, it’s time to roll up our sleeves and get our hands dirty with some practical setup.

Before we deploy CockroachDB, we need a safe “playground” where we can test and experiment without touching the cloud or spending a dime.

Why these tools?

Before we jump into running commands, here’s a quick lookup of what tools we’ll use and why:

• Minikube: A tool that runs a small Kubernetes cluster on your computer. It gives you a local “mini cloud” where you can deploy and experiment.

• Kubectl: The command line tool you’ll use to talk to your Kubernetes cluster to deploy apps, check status, and manage resources.

• Helm: A package manager for Kubernetes. It helps you install complex applications (like CockroachDB) with fewer manual steps.

Step 1: Install Minikube

What is Minikube?
Minikube is a lightweight tool that helps you run a small Kubernetes cluster on your personal computer.

Think of it as your own mini-cloud environment where you can test, deploy, and learn Kubernetes (and in our case, CockroachDB) locally. It’s perfect for learning and experimenting before deploying on the cloud.

Here’s how to get it on different operating systems:

🪟 Windows

1. Make sure you have a hypervisor (VirtualBox, Hyper-V) or Docker installed.

2. Open PowerShell as Administrator.

3. Run:

    choco install minikube
   

   or use:

    winget install minikube
   

4. After installation, check the version:

    minikube version
   

   If it returns a version number, you’re good 👍🏾

If you don’t have the choco or winget package manager, you can install Minikube via PowerShell by following the steps in the docs.

🍎 macOS

1. Ensure you have Homebrew installed.

2. In Terminal, run:

    brew install minikube
   

3. Start the cluster:

    minikube start
   

4. Verify:

    minikube version
   

🐧 Linux

1. Ensure you’re on a supported distribution (Ubuntu, Fedora, and so on) and virtualization (Docker, KVM, and so on) is enabled.

2. Run:

    curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
    sudo install minikube-linux-amd64 /usr/local/bin/minikube
    rm minikube-linux-amd64
   

3. Start the cluster:

    minikube start
   

4. Verify:

    minikube status
   

✅ At this point you should have a local Kubernetes cluster up and running on your machine! Next, we’ll install Kubectl so you can talk to the cluster from your command line.

Step 2: Install kubectl

What kubectl does:
kubectl is the command-line tool that lets you talk to your Kubernetes cluster. Using it, you can deploy applications, check your cluster’s health, and manage resources inside your cluster.

You’ll use it a lot when working with Kubernetes on Minikube and later when you deploy CockroachDB.

Here’s how to install it on Windows, macOS, and Linux:

🪟 Windows

1. Open PowerShell as Administrator.

2. Run:

    choco install kubernetes-cli
   

   or if you prefer:

    choco install kubectl
   

3. Then check the version:

    kubectl version --client
   

   If it prints a version number, you’re good.

🍎 macOS

1. Open Terminal.

2. If you have Homebrew installed, run:

    brew install kubectl
   

3. Check the version:

    kubectl version --client
   

   That should show something like “Client Version: v1.x.x”.

🐧 Linux

1. Open your terminal.

2. Download the latest kubectl binary:

    curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
   

3. Make it executable and move it into your PATH:

    chmod +x ./kubectl
    sudo mv ./kubectl /usr/local/bin/kubectl
   

4. Verify:

    kubectl version --client
   

After this, you’ll have kubectl installed and ready to use with your local Minikube cluster. Next up we’ll install Helm, which will make deploying CockroachDB much easier.

Step 3: Install Helm

Helm is basically the package manager for Kubernetes. Think of it like how you use apt, yum, or brew to install software on your computer. Helm does something similar for Kubernetes apps.

With Kubernetes, deploying a full app often means writing lots of configs (manifests – Deployments, Services, PersistentVolumes, ConfigMaps, and so on). Helm lets us bundle all of that into a single “package” (called a chart) so we don’t have to manually create the resources one-after-the-other (which could be hectic to manage btw 😖).

Because our goal is to deploy a pretty complex system (CockroachDB) on Kubernetes – which includes stateful nodes, persistent storage, networking, SSL/TLS, and so on – using a Helm chart makes it so much easier than crafting dozens of YAML files from scratch.

So before we install CockroachDB, we’ll install Helm. This gives us the toolkit to deploy and manage our cluster much more easily.

Let’s install Helm on each platform. After this, you’ll have the helm command ready to deploy apps into your Kubernetes cluster.

🪟 Windows

1. Open PowerShell as Administrator.

2. If you have Chocolatey installed, run:

    choco install kubernetes-helm
   

   Alternatively:

    choco install helm
   

3. Confirm installation:

    helm version
   

   You should see something like version.BuildInfo{Version:"v3.x.x",…}.

🍎 macOS

1. Open Terminal.

2. With Homebrew installed, run:

    brew install helm
   

3. Verify:

    helm version
   

   If you see version info, you’re good.

🐧 Linux

1. Open your terminal.

2. Download and install the binary (example for the latest version):

    curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
    chmod 700 get_helm.sh
    ./get_helm.sh
   

   Or you can directly download the binary and move it into your PATH.

3. Check version:

    helm version
   

✅ After this, you have helm installed and you’re ready to use it.

In the next part, we’ll use Helm to install CockroachDB into your local Minikube cluster. We’ll add the CockroachDB chart, configure it, and spin up a multi-node replica setup right on your PC.

Deploying CockroachDB on Minikube (The Fun Part Begins 😁!)

Before we go to the cloud, we’ll deploy CockroachDB locally on Minikube using Helm.

This process will help us:

• Understand how CockroachDB runs in a cluster

• Learn how Kubernetes manages database replicas

• Gain hands-on experience before deploying to the cloud

Step 1: Visit ArtifactHub

ArtifactHub is like an App Store for Kubernetes Helm Charts – a huge collection of open-source Helm charts and packages you can easily install.

1. Go to https://artifacthub.io

2. In the search bar, type CockroachDB

3. Click the CockroachDB Helm chart result (you’ll see it published by Cockroach Labs).

You’ll see something like this 👇🏾

Step 2: Explore the Helm Chart

You’ll notice a lot of information on the page:

• README – the documentation for installing and customizing CockroachDB

• Default Values – all the settings that define how the database runs

Don’t worry if it looks overwhelming. We’ll walk through it together 😉

Step 3: Copy the Default Values

Every Helm chart has a default configuration file. These defaults are usually too advanced or too heavy for local setups, so we’ll create our own lighter version. But first, let’s copy the original for reference.

1. On the CockroachDB chart page, click the Default Values button.

2. A modal window will pop up showing a long YAML file.

3. Click the Copy icon in the top-right corner to copy all the default values.

Step 4: Create a Folder for Our Project

We’ll keep everything organized in a single folder.

mkdir cockroachdb-tutorial
cd cockroachdb-tutorial

Inside this folder, create a new file called:

nano cockroachdb-original-values.yml

Now paste all the default values you copied earlier (use Ctrl+V or right-click → Paste), then save and exit (Ctrl+O, then Ctrl+X in nano).

If you’re on Windows, just open Notepad/VSCode, paste the content, and save the file in the same folder.

Step 5: Understanding the Key Configurations

Let’s break down a few important values you’ll notice in the file.

🧩 statefulset.replicas

This tells CockroachDB how many database nodes (replicas) to run in the cluster. By default, it’s set to 3, meaning you’ll have 3 independent database instances that can all read and write data.

⚙️ statefulset.resources.requests and statefulset.resources.limits

These settings tell Kubernetes how much CPU and memory to give CockroachDB.

• requests: the minimum guaranteed amount

• limits: the maximum allowed amount

CockroachDB can be a bit greedy with memory 😅, so limits make sure it doesn’t take everything and leave no room for other apps.

💾 storage.persistentVolume.size

This defines how much disk space each CockroachDB node gets. For example, if you set it to 10Gi and you have 3 replicas, total usage = 30Gi.

💽 storage.persistentVolume.storageClass

This defines the type of disk to use:

• standard: HDD (cheap but slow)

• standard-rwo: SSD (faster and affordable)

• pd-ssd or fast-ssd: NVMe (super fast but pricey)

You can check available storage classes in your Minikube cluster using:

kubectl get sc

On Minikube, the default storage class is usually standard.

You can learn more about Google Cloud storage classes here.

🔐 tls.enabled

This controls whether CockroachDB requires TLS certificates for secure connections.

If true, you’ll need to generate certificates for any app or client that connects to your cluster (instead of using a username and password). This is strongly recommended for production, but for our local Minikube setup, we’ll disable it so it’s easier to play around and test connections.

Step 6: Create a Simplified Values Config for the CockroachDB Helm Chart

We’ll now create a new config file with lighter resource settings for our local test environment.

In the same folder, create:

nano cockroachdb-values.yml

Then paste this:

statefulset:
  replicas: 3
  podSecurityContext:
    fsGroup: 1000
    runAsUser: 1000
    runAsGroup: 1000
  resources:
    requests:
      memory: "1Gi" # You should have 3GB+ of RAM free on your device; else, you can reduce this to 500Mi (this will result in your PC needing just 1.5 GB of RAM free)
      cpu: 1  # The same with this, you can reduce it to 500m CPU if you don't have up to 3 CPU cores (1 CPU core * 3 replicas)
    limits:
      memory: "1Gi"
      cpu: 1
  podAntiAffinity:
    type: ""
  nodeSelector:
    kubernetes.io/hostname: minikube

storage:
  persistentVolume:
    size: 5Gi # Make sure you have 15GB+ of free storage on your local machine, if not, you can reduce it to 2 - 3 Gi
    storageClass: standard

tls:
  enabled: false

init:
  jobs:
    wait:
      enabled: true

Setting the requests and limits to the same value ensures Kubernetes won’t terminate CockroachDB pods due to high memory or CPU usage.

You can read more about this here.

Overview of the YAML values

Now, let’s understand the content of the cockroachdb-values.yml file together

podSecurityContext – why you needed it on Minikube:

podSecurityContext:
  fsGroup: 1000
  runAsUser: 1000
  runAsGroup: 1000

This block sets the Linux user and group IDs that the CockroachDB process runs as inside the container, and the group ownership for mounted files.

Why this matters, simply:

• The CockroachDB process runs as UID 1000 inside the container. If the disk mount (the persistent volume) is owned by a different UID, Cockroach can’t create files there and fails with permission denied.

• runAsUser and runAsGroup make the container process run as UID/GID 1000.

• fsGroup makes the mounted volume be accessible to that group, so the process can write to /cockroach/cockroach-data.

In short, these lines make sure the DB process has permission to create and write files on the mounted disk (volume), which is especially important on Minikube and other local setups where host-mounted storage can have odd permissions.

podAntiAffinity and nodeSelector – what they do:

podAntiAffinity:
  type: ""

nodeSelector:
  kubernetes.io/hostname: minikube

podAntiAffinity is the default behavior. Normally this tells Kubernetes to spread pods across different nodes (VMs), so replicas don’t run on the same physical machine. This is good for high availability, because one node failing won’t kill multiple replicas.

By setting type: "" (empty), you disabled that spreading rule, so Kubernetes can place multiple CockroachDB replicas on the same node.

nodeSelector tells Kubernetes to schedule pods only on nodes that match the label you set (here kubernetes.io/hostname: minikube). That forces all pods to run on the node named minikube.

Quick summary of the effect:

• Good for local testing on a multi-node Minikube cluster, when only one node has properly mounted writable storage.

• Not recommended for production, because it places all replicas on the same machine (single point of failure).

PS: If you’re using another Kubernetes cluster provider, for example K3s, Kind, and so on… this might not get deployed due to the nodeSelector property targeting minikube nodes. So, I'd advise removing the nodeSelector property entirely.

...
nodeSelector:
    kubernetes.io/hostname: minikube
...

✅ At this point, we’ve:

• Copied the default CockroachDB Helm chart configuration

• Created a lightweight version for Minikube

• Learned what each key property means

🚀 Step 7: Install the CockroachDB Cluster Using Helm

Great job so far! You’ve created your cockroachdb-values.yml file and set up your custom configuration for Minikube. Now we’ll actually deploy the cluster.

What we’re going to do:
We’ll use Helm to install the official CockroachDB Helm chart using our custom values. This will spin up your 3-node cluster locally so you can play with it.

Command to run:

helm install crdb cockroachdb/cockroachdb -f cockroachdb-values.yml

Here:

• crdb is the name we’re giving this release (you can pick something else if you like).

• cockroachdb/cockroachdb tells Helm which chart to use.

• -f cockroachdb-values.yml tells Helm to use our custom file instead of default values.

After the command runs:

After a little while the command completes, and you’ll see output telling you what resources were created (pods, services, persistent volume claims, and so on).

Now to check if everything is working, do this:

kubectl get pods | grep -i crdb

This filters pods with “crdb” in the name (our release prefix).

You should see something like:

The three primary pods (0, 1, 2) should be in Running state. The init job or pod (crdb-cockroachdb-init-xxx) should show Completed. This means the initialization tasks (cluster bootstrap) succeeded.

If you see that, congratulations! You’ve got your local CockroachDB cluster up and running! 🎉

Accessing the CockroachDB Console & Viewing Metrics

Alright! Now that our CockroachDB cluster is up and running, let’s take a peek behind the scenes and explore the CockroachDB Admin Console. It’s a beautiful web dashboard that helps us visualize everything happening in our database cluster.

In this section, we’ll learn how to:

• Access the CockroachDB admin console right from your browser 🖥️

• Understand what each built-in dashboard shows (CPU, memory, disk, SQL performance)

• Confirm that our cluster is healthy and that all 3 nodes are working together perfectly

Step 1: Locate the CockroachDB Public Service

CockroachDB automatically creates a public service that allows us to connect to the database and also access its dashboard.

Let’s check it out by running:

kubectl get svc | grep -i crdb

You should see a line similar to:

crdb-cockroachdb-public   ClusterIP   10.x.x.x   <none>   26257/TCP,8080/TCP   ...

This service (crdb-cockroachdb-public) is what we’ll use to connect to both:

• The database itself (via port 26257)

• The dashboard UI (via port 8080)

Step 2: Learn More About the Service

Let’s dig a little deeper to understand it:

kubectl describe svc crdb-cockroachdb-public

Here’s what you’ll notice:

• Port 26257 is used for gRPC connections (this is how applications connect to send and receive SQL queries).

• Port 8080 is used for the web dashboard, where we can view metrics and monitor performance.

Step 3: Access the CockroachDB Dashboard

Now, let’s make the dashboard available on your local computer. Run this command:

kubectl port-forward svc/crdb-cockroachdb-public 8080:8080

This command simply tells Kubernetes:

“Hey, please open a tunnel from my local computer’s port 8080 to the CockroachDB service’s port 8080 in the cluster.”

Once you see something like:

...you’re good to go!

Step 4: Visit the Dashboard

Now, open your browser and go to http://localhost:8080.

You’ll see the CockroachDB Admin Console. This is your central command center for monitoring your cluster

Here, you’ll be able to view:

• Number of replicas (nodes): You should see 3 in our setup.

• RAM usage per node: Helps track how much memory each CockroachDB instance is using.

• CPU usage: Useful to know when your database is getting busy.

• Disk space: Shows how much data your cluster is storing and how much free space remains.

Here’s what your dashboard might look like 👇🏾

Step 5: Exploring the Metrics Dashboard

Now that you’re inside the CockroachDB Admin Console (http://localhost:8080), let’s take things a step further by exploring the Metrics section. This is where CockroachDB really shines.

On the left-hand side, click on “Metrics.” Here, you’ll find a collection of dashboards showing how your database is performing behind the scenes, things like query activity, performance, memory use, and much more.

These metrics help you understand what’s happening inside your cluster and make data-driven decisions – like when to scale up, optimize queries, or add more nodes.

We’ll start by focusing on some of the most insightful ones, such as:

• SQL Queries Per Second – how busy your database is

• Service Latency (SQL Statements, 99th percentile) – how fast or slow your queries are

Then, we’ll also look at others like SQL Contention, Replicas per Node, and Capacity to get a complete view of your CockroachDB cluster’s health.

Here’s what each of these metrics means in simple, everyday terms 👇🏾

SQL Queries Per Second

This metric shows the number of SQL commands (like SELECT, INSERT, UPDATE, DELETE) your database cluster is handling every second. In simpler words, it’s how busy your database is. Imagine cars passing through a toll booth – this is the count of cars per second.

This is useful to know because if this number is steadily climbing, your system is getting more traffic or work. You may need to scale up (more nodes, more resources) or optimize queries. If it drops suddenly, something might be wrong (traffic drop, and so on).

Look for a stable or expected value for your workload. Spikes or sustained high values mean you should check performance.

Service Latency: SQL Statements, 99th percentile

This metric shows the time it takes (for the slowest ~1 % of queries) from when the database gets the request until it finishes executing it. Think of waiting in a queue: 99% percentile is what the slowest people (1 in 100) experienced.

You’ll want to know this because if the slowest queries are taking too long, it might signal a bottleneck (CPU, disk, network, and so on). Low latency = good user experience.

So keep an eye out: if this value rises (gets worse) over time, investigate what’s slowing down. If it stays low and stable, you’re in good shape.

SQL Statement Contention

Statement contention demonstrates the number of SQL queries that got “stuck” or had to wait because other queries were using the same data or resources. This is like if two people were trying to grab the same book – one has to wait. That waiting is contention.

High contention means your database is chasing conflicts, waiting for locks or resources. This slows things down overall. So you’ll want to keep this number as low as possible. If it starts rising, you might need to revisit your schema, queries, or scale differently.

Replicas per Node

This tells you how many copies (“replicas”) of data ranges live on each database node. If you imagine your data is like documents saved in several safes (nodes), this shows how many copies are in each safe.

This matters, because you want balanced replicas so no node is overloaded with too many copies (which can slow it down or put it at risk).

To check on this, make sure nodes have roughly equal replica counts. If one node has many more replicas, you might need to rebalance or add nodes.

Capacity

Capacity shows how much disk/storage your cluster has (total), how much is used, and how much is free. Imagine a warehouse: it’s like how many boxes you can store, how many you’ve filled, and how much empty space remains.

You’ll need to know this, because if capacity is nearly full, you risk running out of space which can cause downtime or performance issues.

Free space should stay healthy (for example less than ~80% used). If it crosses that, plan to add storage or nodes.

Why These Matter Together

When you combine these metrics, you get a clear picture:

• High Queries Per Second + high latency = maybe you're under-powered.

• High contention = your workload design might be fighting itself.

• Imbalanced replicas or full capacity = infrastructure issues.

• Stable low latency + balanced replicas + plenty of capacity = sounds like a healthy cluster.

So by keeping an eye on these, you make data-driven decisions: when to scale, when to optimize, when to tweak configs.

Step 6: Creating a Little Load on the CockroachDB Cluster

So far, we’ve explored the CockroachDB dashboard and understood what each metric means. Now, let’s make things a bit more fun. 🎉

In this part, we’ll run a simple Python app that connects to our CockroachDB cluster and performs a few database operations (creating, updating, deleting, and retrieving some records). This will help us generate a small load on the database so we can actually see the metrics in action.

Here’s what we’ll be doing step-by-step 👇🏾

Step 6.1: Create a ConfigMap for Our Books Data

We’ll first create a list of 20 books that our Python script will interact with. Each book will have basic info like name, author, genre, pages, and price.

1. Create a new file called books.json

   • On Linux:

       nano books.json
     

     Paste the below JSON content into it.

       [
         {
           "name": "The Bright Signal",
           "author": "Ava Hart",
           "isbn": "9783218196000",
           "published_year": 2020,
           "pages": 234,
           "genre": "Fantasy",
           "price": 10.99
         },
         {
           "name": "The Hidden Library",
           "author": "Liam Stone",
           "isbn": "9783863794026",
           "published_year": 1993,
           "pages": 358,
           "genre": "Romance",
           "price": 30.2
         },
         {
           "name": "The Shadow Archive",
           "author": "Maya Chen",
           "isbn": "9781615594078",
           "published_year": 2001,
           "pages": 404,
           "genre": "History",
           "price": 16.21
         },
         {
           "name": "The Bright Voyage",
           "author": "Noah Rivers",
           "isbn": "9785931034133",
           "published_year": 1987,
           "pages": 507,
           "genre": "Fantasy",
           "price": 13.14
         },
         {
           "name": "The Shadow Garden",
           "author": "Zara Malik",
           "isbn": "9785534192834",
           "published_year": 2004,
           "pages": 404,
           "genre": "Sci-Fi",
           "price": 28.13
         },
         {
           "name": "The Crystal Signal",
           "author": "Ethan Brooks",
           "isbn": "9785030564135",
           "published_year": 2009,
           "pages": 508,
           "genre": "Self-Help",
           "price": 20.79
         },
         {
           "name": "The Atomic Atlas",
           "author": "Iris Park",
           "isbn": "9787242388493",
           "published_year": 2025,
           "pages": 442,
           "genre": "Romance",
           "price": 18.5
         },
         {
           "name": "The First Library",
           "author": "Caleb Nguyen",
           "isbn": "9787101226911",
           "published_year": 2017,
           "pages": 528,
           "genre": "Romance",
           "price": 24.47
         },
         {
           "name": "The Crystal River",
           "author": "Sofia Diaz",
           "isbn": "9781845146276",
           "published_year": 2004,
           "pages": 599,
           "genre": "Fiction",
           "price": 31.15
         },
         {
           "name": "The Crystal Archive",
           "author": "Jude Bennett",
           "isbn": "9784893252883",
           "published_year": 1996,
           "pages": 632,
           "genre": "Fiction",
           "price": 40.47
         },
         {
           "name": "The Last Compass",
           "author": "Nina Volkova",
           "isbn": "9784303911713",
           "published_year": 2018,
           "pages": 451,
           "genre": "History",
           "price": 29.53
         },
         {
           "name": "The Crystal Garden",
           "author": "Omar Haddad",
           "isbn": "9784896383461",
           "published_year": 1988,
           "pages": 251,
           "genre": "Thriller",
           "price": 36.38
         },
         {
           "name": "The Silent Signal",
           "author": "Priya Kapoor",
           "isbn": "9781509839308",
           "published_year": 2008,
           "pages": 649,
           "genre": "Fantasy",
           "price": 28.05
         },
         {
           "name": "The Hidden Compass",
           "author": "Felix Romero",
           "isbn": "9781834738291",
           "published_year": 2025,
           "pages": 180,
           "genre": "Self-Help",
           "price": 19.15
         },
         {
           "name": "The Lost Signal",
           "author": "Tara Quinn",
           "isbn": "9781165667017",
           "published_year": 2010,
           "pages": 368,
           "genre": "Fiction",
           "price": 41.37
         },
         {
           "name": "The Last Signal",
           "author": "Hana Sato",
           "isbn": "9783387262476",
           "published_year": 2005,
           "pages": 467,
           "genre": "Nonfiction",
           "price": 42.01
         },
         {
           "name": "The Crystal Archive",
           "author": "Leo Fischer",
           "isbn": "9780801326776",
           "published_year": 1984,
           "pages": 573,
           "genre": "Nonfiction",
           "price": 42.31
         },
         {
           "name": "The Hidden Atlas",
           "author": "Mila Novak",
           "isbn": "9784746872343",
           "published_year": 2005,
           "pages": 180,
           "genre": "Nonfiction",
           "price": 16.58
         },
         {
           "name": "The Hidden Compass",
           "author": "Arthur Wells",
           "isbn": "9780097882086",
           "published_year": 1983,
           "pages": 713,
           "genre": "Fantasy",
           "price": 39.42
         },
         {
           "name": "The Silent Atlas",
           "author": "Selene Ortiz",
           "isbn": "9781939909169",
           "published_year": 1991,
           "pages": 190,
           "genre": "Self-Help",
           "price": 33.79
         }
       ]
     

     To save and close the file in nano:

     • Press CTRL + O → then ENTER (to save)

     • Press CTRL + X (to exit the editor)

2. Then create a ConfigMap from the file:

    kubectl create configmap books-json --from-file=books.json
   

Step 6.2: Create the Python Script ConfigMap

Next, we’ll create a simple Python script that:

• Creates a new table for books

• Inserts 20 records

• Updates 7 of them

• Deletes 5

• Retrieves 15 books from the database

It’s like simulating a small library app. 📚

Create a new file called books-script.yml and paste the content below:

apiVersion: v1
kind: ConfigMap
metadata:
  name: books-script
data:
  run.py: |
    #!/usr/bin/env python3
    import argparse
    import json
    import os
    import sys
    import time
    from typing import List, Dict

    import psycopg
    from psycopg.rows import dict_row

    DDL = """
    CREATE TABLE IF NOT EXISTS books (
        id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
        name STRING NOT NULL,
        author STRING NOT NULL,
        isbn STRING UNIQUE,
        published_year INT4,
        pages INT4,
        genre STRING,
        price DECIMAL(10,2),
        created_at TIMESTAMPTZ NOT NULL DEFAULT now()
    );
    """

    INSERT_SQL = """
    INSERT INTO books (name, author, isbn, published_year, pages, genre, price)
    VALUES (%s, %s, %s, %s, %s, %s, %s);
    """

    UPDATE_SQL = """
    UPDATE books
    SET price = %s, pages = %s
    WHERE isbn = %s;
    """

    DELETE_SQL = """
    DELETE FROM books
    WHERE isbn = %s;
    """

    GET_SQL = """
    SELECT id, name, author, isbn, published_year, pages, genre, price, created_at
    FROM books
    WHERE isbn = %s;
    """

    def load_books(path: str) -> List[Dict]:
        with open(path, "r") as f:
            return json.load(f)

    def connect_with_retry(dsn: str, attempts: int = 30, delay: float = 2.0):
        last_exc = None
        for _ in range(attempts):
            try:
                conn = psycopg.connect(dsn, autocommit=False)
                return conn
            except Exception as e:
                last_exc = e
                time.sleep(delay)
        raise last_exc

    def main():
        ap = argparse.ArgumentParser()
        ap.add_argument("--dsn", required=True, help="Postgres/CockroachDB DSN")
        ap.add_argument("--json", default="/app/books.json", help="Path to books JSON")
        args = ap.parse_args()

        books = load_books(args.json)
        print(f"Loaded {len(books)} books")

        conn = connect_with_retry(args.dsn)
        conn.row_factory = dict_row
        try:
            with conn:
                with conn.cursor() as cur:
                    print("Creating table...")
                    cur.execute(DDL)

                    print("Inserting 20 books...")
                    for b in books[:20]:
                        cur.execute(INSERT_SQL, (
                            b["name"], b["author"], b["isbn"],
                            b.get("published_year"), b.get("pages"),
                            b.get("genre"), b.get("price"),
                        ))

                    print("Updating 7 books...")
                    for b in books[:7]:
                        new_price = round(float(b.get("price", 10)) + 1.23, 2)
                        new_pages = int(b.get("pages", 100)) + 5
                        cur.execute(UPDATE_SQL, (new_price, new_pages, b["isbn"]))

                    print("Deleting 5 books...")
                    for b in books[-5:]:
                        cur.execute(DELETE_SQL, (b["isbn"],))

                    print("Performing 15 retrievals...")
                    for b in books[:15]:
                        cur.execute(GET_SQL, (b["isbn"],))
                        row = cur.fetchone()
                        if row:
                            print(f"GET {b['isbn']}: {row['name']} by {row['author']} (${row['price']})")
                        else:
                            print(f"GET {b['isbn']}: not found (possibly deleted)")

            print("All operations completed.")
        finally:
            conn.close()

    if __name__ == "__main__":
        main()

This script connects to the CockroachDB cluster, creates a table (if it doesn’t exist), and performs all those operations in sequence.

It runs around 50 SQL queries in total – a mix of INSERT, UPDATE, DELETE, and SELECT statements.

Now apply it:

kubectl apply -f books-script.yml

Step 6.3: Create the Job to Run the Script

Next, let’s create a Kubernetes Job that will actually run our Python script inside a container.

Create a file called books-job.yml and paste the manifest below:

apiVersion: batch/v1
kind: Job
metadata:
  name: books-job
spec:
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: runner
          image: python:3.12-slim
          env:
            - name: CRDB_DSN
              value: "postgresql://root@crdb-cockroachdb-public:26257/defaultdb?sslmode=disable"
          command: ["bash", "-lc"]
          args:
            - |
              pip install --no-cache-dir "psycopg[binary]>=3.1,<3.3" && \
              python /app/run.py --dsn "$CRDB_DSN" --json /app/books.json
          volumeMounts:
            - name: script
              mountPath: /app/run.py
              subPath: run.py
            - name: books
              mountPath: /app/books.json
              subPath: books.json
      volumes:
        - name: script
          configMap:
            name: books-script
            defaultMode: 0555
        - name: books
          configMap:
            name: books-json

Here’s what’s happening:

• The Job runs a container based on Python 3.12-slim.

• It connects to CockroachDB using the connection string postgresql://root@crdb-cockroachdb-public:26257/defaultdb?sslmode=disable. Notice how sslmode=disable: this is because we disabled TLS in our Helm values earlier.

• The Job mounts the two ConfigMaps we created earlier (books-json and books-script) as volumes inside the container. Think of volumes like small external drives that the container can read from.

Apply it:

kubectl apply -f books-job.yml

Step 6.4: Check if the Job Ran Successfully

After a minute or two, check your pods:

kubectl get po

If you see books-job-xxx with the status Completed, then your script ran successfully 🎉

That means our database just got a nice little workout – some records were created, updated, deleted, and read.

Step 7: Viewing the Metrics from the Load

Now that we’ve generated a small load, let’s jump back to the CockroachDB dashboard.

Head to the Metrics section, and under SQL Queries Per Second, you should see a little spike: this shows the activity from our Python job.👇🏾

Hover your mouse over the graph lines to see exact numbers.

Do the same for Service Latency: SQL Statements (99th percentile). You’ll notice a few bumps showing how long some of the queries took.👇🏾

This small experiment gives you a real feel for how CockroachDB reacts under activity, even a tiny one.

To explore more metrics and dashboards, check out the official CockroachDB documentation here.

Step 8: View the List of Created Items in the Database

Now that our Python job ran and touched the database (creating, updating, deleting, retrieving records), let’s check the content of our books table just to verify everything really happened.

First, we’ll create another Kubernetes job (or pod) that connects to our CockroachDB cluster and runs a simple SQL query SELECT * FROM books;. This pulls out all the remaining records in the table.

Here’s the manifest to use. Create a file named view-books.yml and paste the below content inside it:

apiVersion: batch/v1
kind: Job
metadata:
  name: view-books
spec:
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: client
          image: cockroachdb/cockroach:v25.3.2
          command: ["bash", "-lc"]
          args:
            - |
              cockroach sql \
                --insecure \
                --host=crdb-cockroachdb-public:26257 \
                --database=defaultdb \
                --format=records \
                --execute="SELECT * FROM public.books;"

Note: We use sslmode=disable because we turned off TLS in our Minikube config. This job mounts nothing fancy. It just spins up, connects to the database, runs the SELECT, and displays the result.

Run the job:

kubectl apply -f view-books.yml

Wait a minute, then check the pod status:

kubectl get po

Look for something like books-client-job-xxx in Completed state.

Finally, view the job logs to see the actual records:

kubectl logs view-books

You’ll see output similar to the below:

Backing Up CockroachDB to Google Cloud Storage ☁️

In this section we’ll explain how you can automate backups of your CockroachDB cluster using simple SQL commands, service accounts (for authenticating to Google Cloud), and Google Cloud Storage (where the data will be stored).

Why Backups Are Absolutely Critical

Imagine you’ve built your cluster on Kubernetes, and everything’s humming along for weeks or months. You’ve got tens or hundreds of gigabytes of data and 10k+ users relying on it.

Then BAM! Something happens. Maybe someone accidentally overwrote the Helm release (helm upgrade --install … with the same release name, for example crdb), or a cloud disk got deleted, or a critical node failed and you lose the majority of data replicas. That’s the nightmare we all dread 😭.

Mistakes happen, even if you’re super careful. What matters most is: How fast and easily could you recover?

That’s why we’ll set up daily backups of our CockroachDB cluster, targeting a Google Cloud Storage bucket. (Quick note: Google Cloud Object Storage is a service where you can store large amounts of data in the cloud as “objects”. You can grab, store, and retrieve data from it, just like Google Drive or Apple Storage. 😃)

With your backups going into a storage bucket, if disaster strikes, you can restore the entire cluster (or specific databases/tables) in minutes or hours – instead of days or losing data forever.

Connecting to Our DB – Installing Beekeeper Studio

So far, we’ve been connecting to our database programmatically, running commands from pods or jobs inside Kubernetes. But what if there was a more visual and user-friendly way to explore our data?

Well, meet my friend Beekeeper Studio. 🙂

Beekeeper Studio is a sleek, open-source database management tool that lets you connect to a wide range of databases like PostgreSQL, MySQL, SQLite, and (most importantly for us) CockroachDB.

It comes with a simple, modern interface for running queries, browsing tables, and viewing data – no need to jump into pods or remember command-line flags 😄

How to Install Beekeeper Studio

1. Visit the official Beekeeper Studio download page here: https://www.beekeeperstudio.io/get

2. Click the “Skip to the download” link. You’ll see something like this:

   

3. You’ll be redirected to a page listing download options for different operating systems.

   

4. Choose your OS and download the correct installer.

5. Afterwards, install the downloaded Beekeeper Studio software according to your OS

Connecting Beekeeper Studio to CockroachDB

Now that we’ve installed Beekeeper Studio, it’s time to connect it to our CockroachDB cluster running inside Minikube

But before we jump in, here’s something important to note:👇🏾

Our CockroachDB cluster is running INSIDE Kubernetes, and by default, it’s not accessible from outside the cluster.

To confirm this, run:

kubectl get svc crdb-cockroachdb-public

You should see something like this 👇🏾

Notice the CLUSTER-IP column. That means the service can only be accessed by other pods INSIDE the Minikube cluster – not from your laptop or external apps

Exposing the Cluster for Local Access

To make our database accessible from your local machine (so Beekeeper Studio can reach it), we’ll use Kubernetes Port Forwarding.

In a new terminal tab, run:

kubectl port-forward svc/crdb-cockroachdb-public 26257

This command tells Kubernetes to forward your local port 26257 to CockroachDB service’s port 26257 inside the cluster.

Once it’s running, your CockroachDB instance will now be accessible from localhost:26257.
(Note: it’s not accessible via your browser because this isn’t an HTTP endpoint 😅)

🐝 Connecting via Beekeeper Studio

1. Open Beekeeper Studio.

2. Click on the dropdown that says “Select a connection type…”.

3. Choose CockroachDB from the list.

   

4. In the connection window that pops up:

   • Disable the Enable SSL option.

   • Set User to root

   • Set Default Database to defaultdb

   • Host to localhost

   • Port to 26257

5. Now click Test (bottom right corner). You should see a success message like Connection looks good.

Your setup should look like this:👇🏾

Finally, click Connect (right beside the Test button).

Verify the Connection

Once connected, you’ll land on a clean workspace where you can run SQL queries.

To confirm you’re connected to the right cluster, run:

SELECT * FROM books;

You should see a table containing about 15 books (the same ones we inserted earlier):

And there you go. You’ve now connected Beekeeper Studio to your CockroachDB running inside Minikube! 🚀

Creating a Google Cloud Account

Before we can back up our CockroachDB data to Google Cloud Storage, we need to have a Google Cloud account ready.

Step 1: Visit the Google Cloud Console

Head over to 👉🏾 https://console.cloud.google.com

If you don’t have a Google account yet, don’t worry. The process is simple and self-explanatory once you visit the site :). You’ll be guided to create a Google account first, and then your Google Cloud account.

Step 2: Create or Use a Project

Once you’re in the Google Cloud Console, you’ll either:

• Use the default project that was automatically created for you, or

• Create a new one by clicking on “New Project” and naming it crdb-tutorial.

Projects are like folders that contain all your Google Cloud resources: compute instances, storage buckets, databases, and more.

Step 3: Link a Billing Account (Optional but Recommended)

If you already have a billing account, link it to your project.

If not, you can easily create one by following Google’s instructions here. (You’ll need a valid Debit or Credit card.)

Don’t worry if your card doesn’t link right away. Sometimes Google’s billing system can be picky. 😅

Here’s a quick fix that usually works:

1. Add your card to Google Pay first.

2. Then go to Google Subscriptions in your Google account, and link it to your Google Billing Account.

To add your card via Google Subscriptions, visit here. (You need to have a Google account first. Don’t worry, the site will direct you on what to do if you don’t.)

You’ll see a page like this:👇🏾

Click Manage payment methods, then add your card details.

Once you’ve done that, refresh your Google Billing Account page – you should now see your card as one of the available options.

Creating a Google Cloud Storage Bucket

Now that we’ve set up our Google Cloud account and enabled billing, let’s create a Cloud Storage Bucket. This is simply a location (like an online folder) where our CockroachDB backup files will be stored.

In your Google Cloud console, type “storage” in the search bar at the top. From the dropdown results, click on “Cloud Storage”:

On the new page, click on the “Buckets” link in the side menu, then click the “Create Bucket” button.

Give your bucket a unique name, like cockroachdb-backup-. For example, cockroachdb-backup-i8wu, cockroachdb-backup-7gw8u. The random characters ensure your bucket name is unique globally (no other Google Cloud user will have the same name).

Scroll to the bottom and click “Create” to create your bucket.

You’ll see a pop-up asking you to confirm public access prevention. This means that only you (and people you explicitly give access to) can view or edit your bucket. Make sure the “Enforce public access prevention on this bucket” checkbox is checked, then click “Confirm.”

Perfect! 🎉 You’ve now created a storage bucket where your CockroachDB backups will live.

Giving CockroachDB Access to the Bucket

Our next goal is to let the CockroachDB cluster upload and read files from this bucket. To do this, we’ll create something called a Service Account using Google IAM.

What’s IAM?
IAM stands for Identity and Access Management. It’s basically Google Cloud’s way of managing who can access what in your project.

With IAM, we can create a service account (like a “digital employee”) and give it permission to interact with our bucket instead of using our personal Google account.

Creating a Service Account

Type “service account” in the search bar and click on “Service Accounts” in the results.

Click “Create Service Account” at the top of the page. On the new page, type: cockroachdb-backup as the service account name, then click ‘Create and Continue’

Now we’ll give this service account permission to work with our storage bucket. In the Permissions section, type “storage object creator” in the filter box and select it from the dropdown.

Repeat the same for “storage object viewer”, and “storage object user”.

At the end, you should see three roles assigned:

• Storage Object Creator

• Storage Object Viewer

• Storage Object User

Click “Continue”, then “Done.”

You’ve now created a service account that can create and read files in your bucket.

Downloading the Service Account Key

To let our CockroachDB cluster use this service account, we’ll generate a key file.

What’s a key file?
It’s just a small JSON file containing secret information your app (CockroachDB) can use to authenticate securely with Google Cloud – like an ID card.

But be careful ⚠️ If this key gets into the wrong hands, anyone could use it to access your Google Cloud resources. Never share or upload this file to your GitHub, BitBucket, or GitLab repository, or any other online repositories.

In the Service Accounts page, find your cockroachdb-backup account, click the three dots (⋮) under the Action column, then select “Manage Keys.”

On the new page, click “Add Key” then “Create new key.”

A dialog box will pop-up, choose JSON as the key type, and click “Create.”

Google will automatically download a file named something like cockroachdb-backup-1234567890abcdef.json

We’ll use this key soon when we configure our CockroachDB backup job.

Attaching the Key to Our CockroachDB Cluster

Now that we’ve downloaded the service account key, we need to attach it to our CockroachDB cluster so that the DB can upload and read backups from our Google Cloud Storage bucket.

Why this is needed:
Our Minikube cluster (and even any managed Kubernetes cluster like GKE, EKS, or AKS) doesn’t have direct access to the files on your computer. So, we’ll upload the key file to Kubernetes as a Secret, and then mount it inside our CockroachDB pods as a volume.

Step 1: Create a Kubernetes Secret

Run the command below in your terminal👇🏾 Replace <PATH_TO_KEY> with the path to your downloaded key file:

kubectl create secret generic gcs-key --from-file=key.json=<PATH_TO_KEY>

This command creates a Kubernetes Secret named gcs-key that securely stores your Google Cloud key.

Step 2: Mount the Secret to the CockroachDB Cluster

Now, let’s tell Kubernetes to use this secret inside our CockroachDB cluster.

Open your cockroachdb-values.yml file and scroll to the statefulset: section. Add the following lines under it:👇🏾

statefulset:
  ...
  env:
    - name: GOOGLE_APPLICATION_CREDENTIALS
      value: /var/run/gcp/key.json

  volumes:
    - name: gcp-sa
      secret:
        secretName: gcs-key

  volumeMounts:
    - name: gcp-sa
      mountPath: /var/run/gcp
      readOnly: true

Here’s what this does:

• The volumes section tells Kubernetes to create a volume from the secret we just made.

• The volumeMounts section attaches that volume inside the CockroachDB container.

• The GOOGLE_APPLICATION_CREDENTIALS environment variable points CockroachDB to our key file so it knows where to find it when connecting to Google Cloud.

Your final file should look like this:👇🏾

statefulset:
  replicas: 3
  podSecurityContext:
    fsGroup: 1000
    runAsUser: 1000
    runAsGroup: 1000
  resources:
    requests:
      memory: "1Gi"
      cpu: 1
    limits:
      memory: "1Gi"
      cpu: 1
  podAntiAffinity:
    type: ""
  nodeSelector:
    kubernetes.io/hostname: minikube
  env:
    - name: GOOGLE_APPLICATION_CREDENTIALS
      value: /var/run/gcp/key.json
  volumes:
    - name: gcp-sa
      secret:
        secretName: gcs-key
  volumeMounts:
    - name: gcp-sa
      mountPath: /var/run/gcp
      readOnly: true

storage:
  persistentVolume:
    size: 5Gi
    storageClass: standard

tls:
  enabled: false

init:
  jobs:
    wait:
      enabled: true

Now, apply the update using Helm:👇🏾

helm upgrade crdb cockroachdb/cockroachdb -f cockroachdb-values.yml

Step 3: Confirm the Key Exists in the Cluster

Once the upgrade is complete, run this command to confirm the key is now inside your CockroachDB pods:

kubectl exec -it crdb-cockroachdb-1 -- cat /var/run/gcp/key.json

You should see something similar to this:👇🏾

prince@DESKTOP-QHVTAUD:~/programming/cockroachdb-tutorial$ kubectl exec -it crdb-cockroachdb-1 -- cat /var/run/gcp/key.json
{
  "type": "service_account",
  "project_id": ***,
  "private_key_id": ***,
  "private_key": ***,
  "client_email": ***,
  "client_id": ***,
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": ***,
  "universe_domain": "googleapis.com"
}

Nice! That means our cluster now has access to the Google Cloud key.

Step 4: Creating the Backup Schedule

CockroachDB makes backups super convenient. It can automatically back up your database on a schedule (without you needing to manually create Kubernetes CronJobs).

To create an automatic backup schedule, run this SQL command inside the CockroachDB SQL shell 👇🏾(Replace the BUCKET_NAME placeholder with the name of your Google Cloud Storage bucket):

CREATE SCHEDULE backup_cluster
FOR BACKUP INTO 'gs://<BUCKET_NAME>/cluster?AUTH=implicit'
WITH revision_history
RECURRING '@hourly'
FULL BACKUP '@daily'
WITH SCHEDULE OPTIONS first_run = 'now';

Here’s what each part means:

• AUTH=implicit tells CockroachDB to use the Google key we mounted (GOOGLE_APPLICATION_CREDENTIALS) for authentication.

• FULL BACKUP '@daily' creates a complete backup of the entire database every day.

• RECURRING '@hourly' creates smaller, incremental backups every hour, capturing just the changes since the last backup.

• WITH SCHEDULE OPTIONS first_run = 'now' starts the first backup immediately after running the command.

After running it, CockroachDB will return two rows:

• The first is for the recurring incremental backup (hourly updates)

• The second is for the full backup (daily snapshot)

You can read more about full and incremental backups in the official docs here 👉🏾CockroachDB Backups Guide.

Step 5: Checking Backup Status

To see the status of your backups, copy the Job ID from the second row (the id column) and run this command:

SHOW JOBS FOR SCHEDULE <YOUR_JOB_ID>;

Replace <YOUR_JOB_ID> with the ID you copied.

You’ll see output similar to this:👇🏾

Now, do the same for the recurring backup job (the ID on the 1st row of the previous result)

If both statuses show succeeded, that means your full and recurring backups worked perfectly! If either is still running, just give it a few minutes – backups can take a bit of time :)

Testing Our Backup — Disaster Recovery Time

Woohoo! We’ve successfully created a backup of our CockroachDB cluster to Google Cloud Storage. That’s a huge milestone. But let’s be honest: how can we be sure it works if we’ve never tried restoring it?

So, in true brave-developer fashion, we’re going to do the unthinkable: destroy our entire database...yes, everything! 😬

Why would we do that?! Because in real life, disasters happen. A node crashes, data gets wiped, or an upgrade goes sideways. The question is: Can we recover? Let’s find out.

Step 1: Uninstall the Helm Chart

First, let’s remove the CockroachDB Helm release. This deletes the cluster resources like StatefulSets, pods, and secrets:

helm uninstall crdb

This removes the running cluster, but not the actual data, which is stored on Persistent Volumes (PVs).

Step 2: Delete Persistent Volume Claims (PVCs)

Each CockroachDB node stores its data in a Persistent Volume Claim (PVC). These PVCs remain even after uninstalling the Helm release, so let’s manually delete them:

kubectl delete pvc datadir-crdb-cockroachdb-0
kubectl delete pvc datadir-crdb-cockroachdb-1
kubectl delete pvc datadir-crdb-cockroachdb-2

Step 3: Delete the Persistent Volumes (PVs)

Next, list all the Persistent Volumes:

kubectl get pv

You’ll see a list of volumes similar to this 👇🏾

Look for the PVs that are bound to the PVCs you just deleted. Then delete them manually using:

kubectl delete pv <PV_NAME>

At this point, you’ve completely wiped out your database like it never existed 🥲. Don’t worry: this is all part of the plan.

Step 4: Reinstall the Cluster

Let’s bring CockroachDB back to life (an empty one for now):

helm install crdb cockroachdb/cockroachdb -f cockroachdb-values.yml

Once the installation is done, expose the cluster locally again:

kubectl port-forward svc/crdb-cockroachdb-public 26257

Step 5: Check What’s Left

Connect to the Beekeeper Studio to your DB if your not, and try running the query below:

SELECT * FROM books;

You’ll get an error saying the books table doesn’t exist, because this is a brand new database.

Step 6: Restore from Google Cloud Storage

Now for the magic part, let’s bring our data back from the backup we created earlier 😃!

Run this query the new cluster:

RESTORE FROM LATEST IN 'gs://<BUCKET_NAME>/cluster?AUTH=implicit';

Replace <BUCKET_NAME> with your actual Google Cloud Storage bucket name (for example: cockroachdb-backup-7gw8u).

CockroachDB will begin restoring your data. This can take a few seconds or minutes depending on your backup size. When it’s done, you’ll see a response showing a success status:

Step 7: Confirm the Restoration

Now, run the same query again:

SELECT * FROM books;

Boom 💥 your books are back 😁! That means your backup and restore process works perfectly. You just performed a full disaster recovery test.

Congrats! You’ve done something many real-world teams fail to test: a full backup and restore cycle. You’ve now proven that your database setup is resilient, even in a worst-case scenario.

Managing Resources & Optimizing Memory Usage

In this section, we’ll learn how CockroachDB handles memory internally (for things like caching and SQL query work), and how to tune these settings so you avoid OOM kills or Eviction – Kubernetes crashing/stopping the database due to it using too much memory than what was allocated to it.

How CockroachDB Uses Memory

When you deploy CockroachDB nodes (each replica) via Kubernetes, each pod (node) needs memory for multiple things. At a high level, there are two major internal uses:

• Cache (conf.cache): This is the space CockroachDB uses to keep frequently accessed data in memory so queries can run faster without hitting the disk.

• SQL Memory (conf.max-sql-memory): This is the memory used when running SQL queries (things like sorting, joins, buffering numbers, and temporary data).

Together, they need to be sized appropriately relative to the total memory you give the pod, so there’s room for these internal operations plus other overhead (networking, logging, background tasks).

The Memory Usage Formula You Must Follow

Here’s the golden rule you should never forget:

(2 × max-sql-memory) + cache  ≤  80% of the memory limit

What this means:

• You take the max-sql-memory value and multiply by 2 (because SQL work may need space for both input and output, etc)

• Add your cache value

• That total must be less than or equal to 80% of the pod’s memory limit (statefulset.resources.limits.memory)

• The remaining ~20% (or more) is free space for other internal CockroachDB processes like background jobs, metrics, network, and so on

If you give CockroachDB too little “free” memory beyond these two settings, you risk OOM kills (pod gets killed by Kubernetes because it used more memory than allowed) or performance issues.

Where You Find These Settings

If you go to the Helm chart docs on ArtifactHub, CockroachDB Helm Chart on ArtifactHub, and scroll down to the Configuration section (or press Ctrl-F for conf.cache), you’ll see:

• conf.cache (cache size)

• conf.max-sql-memory (SQL memory size)

• It states that each of these is by default set to roughly 25% of the memory allocation you set in the resources.limits.memory for the statefulset.

Concrete Example (Step-by-Step)

Let’s do the math with numbers in our Minikube environment.

• In our case we set statefulset.resources.limits.memory = 2 GiB for each CockroachDB pod.

• The Helm default of ¼ (25%) rule means:

  • conf.cache = ¼ × 2 GiB = 512 MiB

  • conf.max-sql-memory = ¼ × 2 GiB = 512 MiB

• Apply the formula: (2 × 512 MiB) + 512 MiB = 1,536 MiB

• Calculate 80% of the memory limit: 80% of 2 GiB = 1,638 MiB (approximately)

• Compare: 1,536 MiB ≤ 1,638 MiB – so we’re within the safe zone ✅

• That means in this configuration, CockroachDB expects to use ~1,536 MiB for its cache + SQL memory. This leaves ~512 MiB (20%) of the 2 GiB limit for other internal processes.

That leftover memory is for things like internal bookkeeping (range rebalancing, replication metadata), communication among database replicas, metric collection, logging, garbage collection, and temporary or unexpected memory spikes.

If you don’t leave this free space, your node might struggle when “normal operations”. And on Kubernetes, if the pod uses more memory than the limits.memory says, it can get OOM-killed which causes downtime or restarts.

⚠️ On Requests vs Limits in Kubernetes

Important nuance: Kubernetes schedules pods based on requests (what you ask for) but enforces limits based on limits (what you allow).

• statefulset.resources.requests.memory = what the scheduler guarantees the pod will have.

• statefulset.resources.limits.memory = the maximum the pod can use before Kubernetes will kill it for excess memory.

Because CockroachDB’s internal memory computations (cache + SQL memory) use the limit value to calculate sizing, if you set requests < limits you’ll get a mismatch. Example:

• Suppose requests = 1 GiB, limits = 2 GiB

• Kubernetes may schedule the pod on a node that has (at least) 1 GiB free

• But internally, CockroachDB will plan for ~1.5 GiB usage (based on the 2 GiB limit)

• The node may not actually have that much free memory available

• The pod might try to use more memory than the node reserved and risk eviction due to less memory for other pods

✅ Best practice: Set requests = limits for memory and CPU for CockroachDB pods. That way the scheduler reserves enough space for what CockroachDB will use internally.

Overriding the Default Fractions

If you want to set static conf.cache or conf.max-sql-memory values (rather than relying on 25% of limit) you can – but you must still obey the memory usage formula.

For example, if you set:

...
conf:
  cache: "1Gi"
  max-sql-memory: "1Gi"
statefulset:
  resources:
    requests:
      memory: "3Gi"
      cpu: 1
    limits:
      memory: "3Gi"
      cpu: 1

According to the above configuration your pod memory request and limit is 3 GiB, then calculate:

(2 × 1Gi) + 1Gi = 3Gi
80% of 3Gi = ~2.4Gi

Here 3Gi > 2.4Gi, so you’d be violating the rule. This is a risky setup.

So you’ll need to either reduce cache or SQL memory, for example to 768Mi (or increase the memory limit, for example 4Gi) so that your formula results in ≤ 80% of the limit.

Scaling CockroachDB the Right Way

In this section we’ll look at when and how you should grow your CockroachDB cluster – whether that means adding more replicas (horizontal scale), giving each node more CPU/RAM (vertical scale), or giving them more storage.

I’ll explain everything in simple terms and cover what metrics to watch, what decisions to make, and how to scale safely.

What we’ll discuss:

• How you can tell it’s time to “grow” your cluster

• How to safely add more nodes or upgrade what you already have

• How to decide whether you need more nodes, bigger nodes, or bigger disks

• How to do all this without causing downtime or stress

Key Metrics to Understand

Before we dive into how to scale our cluster, we need to understand what certain metrics mean. Because, these metrics will help us make calculated decisions, knowing what and and when to scale certain resources.

Read bytes/second & Write bytes/second (Throughput)

Read bytes/second is how much data (in bytes) the disk is reading every second from itself to the database, that is, passing from the disk to the database app.

Write bytes/second is how much data is being written to the disk per second, that is, moving from the database to the disk.

This matters because your database is an application that stores data on disk. If your app needs to read a lot of data (reads) or write a lot of data (writes), this metric shows the volume of data flowing to/from disk.

To keep an eye on it, go to your CockroachDB dashboard and navigate to the “Metrics” link on the sidebar. Under the “Metrics” title, click the “Dashboard:…” drop-down and select “Hardware” from the options.

Now, scroll down a bit till you see “Disk Read Bytes/s” and “Disk Write Bytes/s”.

Read IOPS & Write IOPS

IOPS = “Input/Output Operations Per Second”. Here, Read IOPS = how many read operations the disk is performing per second. Write IOPS = how many write operations per second.

This is different from throughput because throughput is about how many bytes (data) are being transferred. IOPS, on the other hand, is about how many operations are happening (regardless of size).

Here’s an example: 10 read operations/sec of 1 MiB each = 10 MiB/sec throughput, 10 IOPS. Another scenario: 100 reads/sec of 10 KiB each = ~1 MiB/sec throughput, but 100 IOPS (higher operations count though lower data size.

Scroll down a bit more to view the IOPS metrics:

SQL p99 Latency (99th percentile latency)

P99 latency is the time it takes for the slowest 1% of queries to finish.

For example, let’s say you run 1,000 queries. How long the slowest 10 of them took is what p99 shows.

This matters because it’s not about the average query, but about the tail (worst cases). If your p99 is high, it means some queries are seriously lagging. All other queries might be fine, but some are dragging.

So if p99 jumps up (for example, from 10 ms → 300 ms), you should investigate: maybe big joins, missing indexes, contention, or data takes too much time to get stored in the disk.

To access the SQL P99 Latency metrics, simply click the “Dashboard:…” select field, and choose the “Overview” option from the dropdown.

PS: The higher the p99 latency, the more problem there is (slower queries).

Disk Ops In Progress (Queue Depth)

This shows how many disk reads and writes are waiting in line (queued) because the storage system is busy.

A queue depth of 0–5 is generally OK. If it frequently goes into double-digits (10+), that means storage is struggling and latency may spike. If you see this number high and staying high, you may need faster storage or more database replicas.

Simple rule: if “Ops In Progress” > ~9 for extended time, this is a bad sign. Time to check disks and I/O.

To access the “Disk Ops In Progress“ metric, return to the “Hardware“ dashboard, and scroll down:

By monitoring these, you can choose:

• “I need more nodes” (horizontal scale)

• “I need bigger nodes or faster storage” (vertical scale)

• “I need better query/index tuning” (optimize rather than scale)

When (and What) to Scale Based on Your Metrics

So, let’s imagine you’re watching your CockroachDB dashboard and notice this pattern:

• The SQL P99 latency (the slowest 1% of your queries) is high, meaning your queries are taking too long.

• The CPU usage for your CockroachDB pods (under Cockroach process CPU%) is above 80% consistently.

That’s a classic sign your cluster is running out of CPU power and the database is struggling to process queries fast enough because the CPU is maxed out.

Here’s how to fix it 👇🏾

Step 1: Add More CPU Power

You can scale up your CPUs directly through the Helm chart values file, cockroachdb-values.yml.

In that file, look for the section where CPU and memory requests/limits are defined under statefulset.resources. Then, increase the CPU allocations. For example:

statefulset:
  resources:
    requests:
      cpu: "3"
      memory: "6Gi"
    limits:
      cpu: "3"
      memory: "6Gi"

This means each CockroachDB pod (replica) will now request 3 vCPUs (guaranteed). Save the file, then apply the update with the Helm command:

helm upgrade crdb cockroachdb/cockroachdb -f cockroachdb-values.yml

Once the upgrade is done, give it 30 minutes to 1 hour to stabilize. The CockroachDB dashboard will automatically start showing you updated metrics.

If you see that the CPU usage drops below 70% and the SQL P99 latency improves, you’re good. 👍🏾

Step 2: Add Another Replica (New Node)

But…what if the latency is still high even after adding more CPU? That likely means the cluster is still overloaded, and it’s time to add another node (replica) to distribute the load.

Here’s why that works: CockroachDB is horizontally scalable, meaning it automatically spreads out your data (remember ranges?) and balances reads/writes across all replicas. So, the more nodes you add, the more evenly your cluster can share the work.

To add another replica, simply increase the replicas value in your Helm config:

statefulset:
  replicas: 4  # If it was 3 before

Then, redeploy again:

helm upgrade crdb cockroachdb/cockroachdb -f cockroachdb-values.yml

This adds a new pod (a new CockroachDB node) to your cluster. CockroachDB will automatically rebalance your data across nodes – no manual migration needed

💡 Tip: Try to keep one CockroachDB pod (replica) per VM. For example, if you have 3 replicas, you should ideally have 3 separate VMs (worker nodes). This ensures better fault tolerance and performance.

Luckily, the official CockroachDB Helm chart already helps with this by managing Pod anti-affinity rules, so pods are automatically spread across nodes safely.

Disk-Bound Situations — What to Do When Your Disk Is the Limiting Factor

If you’re seeing this kind of pattern in your CockroachDB dashboard and Kubernetes cluster:

• SQL P99 latency is high (queries are slow)

• “Disk Ops In Progress” (queue depth) stays above ~9-10 – meaning many disk I/O operations are waiting to be processed

• Disk “Read bytes/sec” or “Write bytes/sec” (throughput) are high or “Read IOPS” or “Write IOPS” are high (even though CPU looks okay)

Then you’re very likely disk-bound, meaning your storage is the bottleneck.

Here’s how to fix it (and yes, it’s a bit more complex than just “add more RAM”)…

Step 1: Increase Disk Size in Your Helm Values

Often the first problem is that the disk size is too small. Here’s how you can increase it:

1. Open your cockroachdb-values.yml (the Helm chart values file)

2. Look for the storage section, for example:

storage:
  persistentVolume:
    size: 5Gi  # current size

3. Update it to a larger size, like:

storage:
  persistentVolume:
    size: 15Gi  # increased size

4. Save the file and run:

helm upgrade crdb cockroachdb/cockroachdb -f cockroachdb-values.yml

N.B. If this doesn’t work or you receive an error from the Helm chart concerning not being able to modify some values (this is normal), just upsize the disk this way:👇🏾 (just replace the PVC_NAME and SIZE placeholders accordingly)

kubectl patch pvc <PVC_NAME> \
  -p '{"spec":{"resources":{"requests":{"storage":"<SIZE>"}}}}'

Do that for each PVC (datadir-crdb-cockroachdb-0, datadir-crdb-cockroachdb-1, and so on).

Important: Increasing size may help, but often alone is not enough because your disk speed (IOPS/throughput) also depends on factors beyond just size.

Let’s break down why that’s the case, and what really affects your disk performance (especially on Google Cloud, which is what I’m using, too).

Why Disk Speed Can Vary

Your CockroachDB cluster uses external disks provided by your cloud provider (like Google, AWS, or Azure). The speed of those disks – that is, how fast they can read/write data – isn’t fixed. It depends on a few key factors.

On Google Cloud, disk performance depends on three main things:

1. Disk type: HDD, SSD, or fast SSD (pd-ssd) (the faster the disk type, the faster it can handle data operations)

2. Disk size: larger disks usually come with higher speed limits (the bigger, the faster)

3. VM’s vCPU count: more CPUs mean higher quotas for both

   • read/write operations per second (IOPS), and

   • how much data can flow to/from the disk per second (throughput)

The Recommended Disk Type for CockroachDB

The pd-ssd (Google’s fast SSD) is the recommended type for CockroachDB.

• Each pd-ssd disk starts with a minimum of 6,000 IOPS (read or write operations per second).

• It also has around 240 MiB/s (~252 MB/s) of read/write throughput.

In simple terms, that means your CockroachDB disk can handle up to 6,000 read/write operations EVERY SECOND, and move 250+ MB of data in and out every second. That’s pretty impressive!

But here’s the catch: those numbers can still vary depending on your VM family and CPU count.

How VM Family Affects Disk Speed (E2 Example)

If your CockroachDB is running on an E2 VM family (one of Google Cloud’s general-purpose VM types):

• A VM with 2–7 vCPUs can handle up to:

  • 15k IOPS (read/write operations per second)

  • 250+ MiB/s throughput (which is already far more than many databases ever use 😅)

• A VM with 8–15 vCPUs still allows 15k IOPS, but throughput jumps up to ~800 MiB/s 😮 –
  meaning your disk can push nearly 0.8 GB per second of data in/out IN A SECOND.

The more vCPUs you have, the higher these limits grow, both for IOPS and throughput.

Putting It All Together

So, if you notice high SQL P99 latency (queries taking long), and disk read and write IOPS or throughput (read & write bytes) usage close to their limits, then your disk may be maxing out, not your database itself.

Here’s what you can do:

• Check your current VM’s vCPU count and disk performance limit for that CPU.

• If you’re using E2 with low vCPUs (for example, 2–4), try increasing it to 8 vCPUs or more. That’ll immediately lift your IOPS and throughput ceiling.

Example: E2 VM Family IOPS/Throughput Table

E2 per-VM caps (pd-ssd):

e2-medium:     10k write / 12k read IOPS, 200/200 MiB/s
2–7 vCPUs:     15k / 15k IOPS, 240/240 MiB/s
8–15 vCPUs:    15k / 15k IOPS, 800/800 MiB/s
16–31 vCPUs:   25k / 25k IOPS, 1,000 write / 1,200 read MiB/s
32 vCPUs:      60k / 60k IOPS, 1,000 write / 1,200 read MiB/s

The rule is simple — the higher the CPU tier (2–7, 8–15, and so on), the higher the disk speed cap.

⚠️ But What If You’re Still Seeing Slow Queries?

If your CockroachDB queries are still slow, but your metrics show that you’re not fully using your disk capacity (based on your VM’s CPU range), then your disk size might be the actual limitation.

In that case:

• Gradually increase your disk size, for exaxmple from 50Gi to 70Gi to 100Gi.

• Each increase enables your disk to pass more amount of data in and out (especially with pd-ssd).

• Remember: once you increase disk size on Google Cloud, you can’t shrink it back down, so grow it slowly and observe improvements before scaling again.

This step helps you pinpoint exactly whether the slowdown is coming from insufficient IOPS, throughput, or just a disk that’s too small for CockroachDB’s workload 💪🏾

Memory Pressure — What to Do When Your Database Hits the Limit

There are some signs in your cluster you can look out for that’ll tell you your database is getting close to its limit. Pods (database replicas) might be getting OOMKilled (out of memory) or being evicted by Kubernetes, or your memory usage might be staying above ~ 75–80% for a while.

If either these is the case, you’re often dealing with memory pressure (you can check memory usage on the CockroachDB overview dashboard).

Why this happens

If you didn’t set memory requests and limits properly for each replica, the pod might not have enough head-room for all of its internal work (cache, SQL memory, background jobs) and Kubernetes kills it or it crashes.

Also, as you increase load (lots of queries, many users), your database needs more memory for two internal areas:

• --cache (or conf.cache): in-memory data caching

• --max-sql-memory (or conf.max-sql-memory): memory for running SQL queries (joins, sorts, and so on).
  And yes, we covered the formula earlier (2 × max-sql-memory) + cache ≤ ~ 80% of RAM limit.

What to do:

First, you can increase the DB memory. In your Helm chart values (cockroachdb-values.yml), bump up the statefulset.resources.limits.memory and statefulset.resources.requests.memory. Or you can modify conf.cache and conf.max-sql-memory values (if you’re comfortable) but only if the total RAM limit is sufficient to support them.

Because the defaults (when you installed) set each to ~25% of RAM limit, they will scale automatically when you increase RAM.

For example:

• If RAM limit per pod = 5 GiB, then cache ≈ 1.25 GiB, max-sql-memory ≈ 1.25 GiB

• If you raise RAM limit to 8 GiB, these become ≈ 2 GiB each. This keeps you inside the formula and avoids memory crashes.

Quick YAML snippet example:

statefulset:
  resources:
    requests:
      memory: "8Gi"
    limits:
      memory: "8Gi"
conf:
  cache: "25%"
  max-sql-memory: "25%"

After editing your values file, remember to apply it:

helm upgrade crdb cockroachdb/cockroachdb -f cockroachdb-values.yml

When Queries Are Slow but Everything Else (CPU, Memory & Disk) Looks “Fine”

Sometimes you’ll see that your resource metrics (CPU, memory, disk I/O) all seem healthy. But your queries are still slow.

What then? One important cause: hotspots – especially “hot ranges” or “hot nodes” in CockroachDB.

A hot range is a portion of data (in CockroachDB, a range is a section of data from a table) that’s receiving much more traffic (reads or writes) than others.

A hot node, on the other hand, is a node/replica in the cluster which has significantly more load compared to the other nodes – often because it holds one or more hot ranges.

Because most of the traffic (queries) go to a range which is on a specific node, even though your overall CPU / memory / disk metrics might look “okay”, performance still suffers locally: queries are funneled into that specific range, making a “hotspot”.

Learn more about Hotspots here.

Why A High Write Workload Can Slow Reads

When you have lots of write queries, they may overload specific ranges or nodes (especially if the keyspace is skewed). Writes tend to:

• Acquire locks or latches on rows or ranges

• Cause contention among transactions

• Require coordination (for example, via Raft consensus) which impacts performance.

When writes dominate a range, read queries that hit the same ranges may get queued behind these write operations, or suffer longer wait times.

Since reads and writes are sharing the same underlying data/ranges, too much writes can delay reads by creating bottlenecks. The docs call this part of “write hotspots”.

Key Signs You Might Have a Hotspot

• One node’s CPU % is much higher than the others (even though overall resources seem fine)

• On the Hot Ranges page in the CockroachDB UI, some ranges show very high QPS (queries per second) compared to others.

  

• You observe that increasing overall resources (more CPU, more nodes) didn’t resolve the slowness. This suggests the problem isn’t “not enough resources” but “resource imbalance”.

What You Can Do

There are a few things you can do to prevent hotspots:

• Use the Hot Ranges UI page (go to the Database Console and then to Hot Ranges) to identify the range IDs and table/indexes causing the issue.

• Examine how the key space is being used. If your table/index primary key is monotonically increasing (for example, timestamps or serial IDs), the writes may target a narrow portion of the data, causing a hotspot. The docs suggest using hash-sharded indexes or distributing writes across the key-space.

• Ensure load is balanced across nodes: avoid “one node doing most of the work”. If needed, add nodes or ensure range distribution/lease-holder movement is happening.

• Monitor write-versus-read workload. if writes are heavy, they may cause queuing for reads even when resources appear OK. So look at write heavy traffic patterns and try reducing the amount of writes (if possible).

⚠️ Note

Learning everything about hotspots, key visualizers, and range splitting is a bit advanced. For those wanting to dive deeper: see the CockroachDB Performance Recipes page.

Understanding Disk Speed (IOPS & Throughput) Across Cloud Providers

So far, we’ve talked about how disk speed affects CockroachDB’s performance – especially how Google Cloud measures it. But it’s important to know that each cloud provider has its own way of measuring and limiting disk performance (IOPS and throughput).

So, while our earlier examples focused on Google Cloud, similar logic applies to AWS, Azure, and even DigitalOcean, just with different formulas and limits.

For Google Cloud:

These guides break down how disk performance works:

• Persistent Disk performance overview: explains how baseline IOPS and throughput are calculated and the per-instance caps.

• About Persistent Disks: quick definitions of pd-standard (HDD), pd-balanced (SSD), and pd-ssd (SSD).

• Optimize PD performance: shows how disk size, machine series, and tuning can affect performance.

For AWS (EBS):

AWS’s Elastic Block Store (EBS) has several disk types:

• EBS volume types: overview of all SSD and HDD types (gp3, gp2, io2, and so on).

• General Purpose SSD (gp3): lets you provision custom IOPS and throughput for your disks (about 0.25 MiB/s per IOPS, up to 2,000 MiB/s).

For Azure (Managed Disks):

Azure disks also vary by type and size:

• Disk types overview: compares Standard HDD, Standard SSD, Premium SSD, Premium SSD v2, and Ultra Disk.

• Premium SSD v2: lets you independently set IOPS and throughput for your disks.

• VM & disk performance: lists per-VM IOPS and throughput caps.

For DigitalOcean:

DigitalOcean offers simpler storage setups:

• Volumes overview: explains block storage and NVMe details.

• Volume Limits: shows per-Droplet IOPS and throughput caps (including burst windows).

Downsizing the Cluster (Reducing Replicas)

Now that we’ve seen how to scale up our CockroachDB cluster, let’s look at how to scale it down safely and correctly.

Let’s assume we scaled our cluster from 3 replicas to 5 replicas earlier (to handle more workload).

PS: If your CockroachDB pods were crashing often, you might need to increase the CPU and memory limits in the Helm chart configuration, like this:

statefulset:
  replicas: 5
  resources:
    requests:
      memory: "2Gi"
      cpu: 1
    limits:
      memory: "3Gi" # We can keep the memory requests and limits inconsistent for now, since we're in a development environment
      cpu: 1
...

Then, you update the cluster using:

helm upgrade crdb cockroachdb/helm-chart -f cockroachdb-values.yml

After a few minutes, you can confirm the newly added replicas kubectl get pods. You should now see five CockroachDB pods running.

Also, check your CockroachDB Admin UI – the new nodes should now appear in the cluster overview.

P.S: You might experience some issues when upscaling your cluster, especially if you don’t have sufficient memory and CPU on your PC or wherever you’re running your Kubernetes cluster.

⚠️ The Wrong Way to Downscale

Now, what if your workload reduces and you’d like to cut costs by scaling down from 5 replicas back to 3?

You might think, “Oh, I’ll just reduce the number of replicas in the Helm chart from 5 to 3 and redeploy.” But hold on, that’s very wrong! 😅

Scaling up CockroachDB is simple…but scaling down must be done carefully, because of certain factors which will explain.

Decommissioning a Node Before Scaling Down the Cluster

Before you go ahead and reduce the number of replicas in your CockroachDB cluster, it’s important to follow the right process.

You can’t just go from 5 replicas down to 3 and expect everything to go smoothly. There are steps you must take.

Why you can’t just scale from 5 to 3 instantly

If you reduce your cluster size too quickly, you might:

• Lose data redundancy or fail to meet the required replication factor.

• Cause data rebalancing to happen under heavy load, which can slow queries.

• Put your cluster into a state where certain ranges or data replicas don’t have enough copies to remain fault-tolerant.

✅ The correct approach: Decommission first, then scale down one node at a time

Here’s the safe way to downscale:

1. Decommission the node you plan to remove.

2. Once decommissioning is complete, reduce the replica count (for example, from 5 to 4).

3. Delete the disk/PVC tied to that removed node.

4. Repeat the process (remove one node at a time) until you reach your target size (for example, down to 3 replicas).

Step-by-step: Decommission the 5th node (before scaling 5 to 4)

1. Create a client pod to run CockroachDB commands.
   Create a file named cockroachdb-client.yml with this content:

    apiVersion: v1
    kind: Pod
    metadata:
      name: cockroachdb-client
    spec:
      serviceAccountName: <SA>
      containers:
        - name: cockroachdb-client
          image: cockroachdb/cockroach:v25.3.1
          imagePullPolicy: IfNotPresent
          command:
            - sleep
            - "2147483648"
      terminationGracePeriodSeconds: 300
   

   Replace <SA> with your CockroachDB service account name (find it via kubectl get sa -l app.kubernetes.io/name=cockroachdb).

   

2. Apply the manifest:

    kubectl apply -f cockroachdb-client.yml
   

3. Confirm the pod is running:

    kubectl get pods
   

   You should see cockroachdb-client.

4. Exec into the client pod:

    kubectl exec -it cockroachdb-client -- bash
   

5. Get the list of nodes and IDs:

    ./cockroach node status --insecure --host <SERVICE_NAME>
   

   Find your service name: kubectl get svc -l app.kubernetes.io/component=cockroachdb. In our case it’s crdb-cockroachdb-public.

   You’ll see nodes with IDs 1, 2, 3, 4, 5. Each maps to a replica pod like crdb-cockroachdb-0, -1, -2, -3, -4.

   

6. Decommission the node with the highest index (since Kubernetes will remove the highest-numbered replica when scaling down).
   For example, if you’re removing the pod crdb-cockroachdb-4…, and the node ID is 5:

   

   Run the command below to decommission the 5th node.

    ./cockroach node decommission 5 --host crdb-cockroachdb-public --insecure
   

7. Navigate to the CockroachDB dashboard, and monitor until the node status shows as decommissioned.
   In the CockroachDB Console’s Cluster Overview page, you’ll see formerly removed nodes under “Recently Decommissioned Nodes”.

   

8. Scale down the replicas in your Helm values file:

    statefulset:
      replicas: 4
    ...
   

   Then run:

    helm upgrade crdb cockroachdb/cockroachdb -f cockroachdb-values.yml
   

9. Verify pods:

    kubectl get pods
   

   You should now see 4 CockroachDB replica pods.

10. Delete the PVC for the removed node (to avoid paying for storage you’re no longer using):

kubectl delete pvc datadir-crdb-cockroachdb-4

11. Repeat the process for the next node if you want to go from 4 to 3 replicas: decommission node #4 next, scale to 3, delete its PVC, and so on.

After you’re done, you’ll have the target state (for example, 3 nodes) safely and cleanly without causing cluster instability or data loss.

To learn more about scaling down your CockroachDB nodes, visit the official CockroachDB docs.

Note that you should NOT use Horizontal Pod Autoscalers for scaling up and down your CockroachDB cluster.

Remember, before scaling down, you need to DECOMMISSION THE NODES FIRST, and scale down ONE AT A TIME!

However, the Horizontal Pod Autoscalers do NOT obey this. So if you intend to auto-scale your CockroachDB cluster, it's best to have a fixed size of replicas, for example, 3, 5, 7.

Then set up a Vertical pod Autoscaler to scale their CPU and RAM (Remember to set the Memory and CPU requests and limits to the same quantity to prevent eviction as explained earlier).

What to Consider When Deploying CockroachDB on Google Kubernetes Engine (GKE) ☁️

Up until now we’ve been working in a development environment (using Minikube, local setups), testing and learning.

Now we’re ready to move into production mode 🤓. And one of the best places to host CockroachDB in production is on GKE.

In this section, we’ll cover GKE-specific considerations, such as storage classes, load balancers, networking, and how to secure our CockroachDB cluster on GKE using mTLS for authenticating our clients and encrypting any data sent to and from our CockroachDB cluster.

Creating Your GKE Cluster

To get started, head over to the Google Cloud Console.

In the search bar at the top, type “Kubernetes” and click on “Kubernetes Engine” from the dropdown.

You’ll be taken to the Kubernetes Engine page. On the left sidebar, click “Clusters.” Then click the “Create” button at the top.

💡 Note: You’ll need to enable the Compute Engine API before you can create a GKE cluster. If you haven’t done that yet, Google Cloud will automatically redirect you to a page where you can enable it. Just click “Enable”, then return to the cluster page.

You can also learn more about enabling APIs in Google Cloud here: Enable APIs in Google Cloud.

Once you’re back, you’ll see the cluster creation page. If it defaults to Autopilot, click “Switch to Standard cluster” in the top-right corner. This gives you more control over node settings.

Under Cluster basics, give your cluster a name – something like cockroachdb-tutorial works great! Then, set Location type to Zonal (that’s fine for now).

On the left sidebar, go to “Node pools.” You’ll see a default pool already added.

• Keep the name as is.

• Set the Number of nodes to 1.

• Enable the Cluster autoscaler option (so it can scale up automatically later).

• Set the Maximum number of Nodes to 10, and the minimum to 0.

  

Next, click the dropdown arrow beside “default-pool” and select “Nodes.” Here, set up your node specifications:

• VM family: E2

• Machine type: Custom

• vCPUs: 2

• Memory: 7 GB

• Boot disk type: Standard persistent disk

• Disk size: 50 GB

  

  

When all that’s set, click “Create.” Your cluster will start provisioning.

Connecting to your GKE cluster

Once your GKE cluster creation is complete (this might take a few minutes), you’ll see something like this in the console:

Next, click the “Connect” link at the top of the page. A modal will pop up. Copy the CLI command you see.

It’ll look something like:

gcloud container clusters get-credentials cockroachdb-tutorial --zone us-central1-a --project <PROJECT_NAME>

📌 Note: To run this command successfully, you need to have the gcloud CLI tool installed. If you don’t have it yet, visit Install Google Cloud SDK and pick the steps for your OS.

After installing the gcloud CLI, run:

gcloud auth login

This authenticates your terminal with your Google Cloud account so you can access the cluster securely.

After authenticating your terminal with access to Google Cloud, run the command you copied earlier. You should see something like this:

Now run the command to retrieve your pods, kubectl get po. This will retrieve the pods from your new cluster on Google Kubernetes Engine, not Minikube.

For now, we’ve not deployed anything yet, so the namespace should be empty.

But we should have at least 1 worker node available. Run the kubectl get nodes command to view it. You should see something similar to this (GKE takes care of our control plane for us, so when we view the nodes, we’ll only see the worker nodes).

Deploying CockroachDB in Production (on GKE)

Now that we’ve successfully created our Google Kubernetes Engine (GKE) cluster, it’s time to deploy our CockroachDB cluster in it – this time, in production mode.

Unlike our earlier Minikube setup (which we used for local development), deploying to GKE introduces new considerations like security, storage classes, and authentication methods – all tailored for a real-world production environment.

To get started, create a new file called cockroachdb-production.yml, and paste the following configuration inside:

statefulset:
  replicas: 3
  resources:
    requests:
      memory: "3Gi"
      cpu: 1
    limits:
      memory: "3Gi"
      cpu: 1
  serviceAccount:
    create: true
    name: "crdb-cockroachdb"
    annotations:
      iam.gke.io/gcp-service-account: <GOOGLE_SERVICE_ACCOUNT>

storage:
  persistentVolume:
    size: 10Gi
    storageClass: premium-rwo

tls:
  enabled: true

init:
  labels:
    app.kubernetes.io/component: init
  jobs:
    wait:
      enabled: true

Replace the placeholder <GOOGLE_SERVICE_ACCOUNT> with the CockroachDB backup service account you created earlier (in the “Backing Up CockroachDB to Google Cloud Storage” section). It should look something like this cockroachdb-backup@<PROJECT_ID>.iam.gserviceaccount.com.

Understanding the Configuration

Let’s break down what’s happening in this production Helm values configuration and how it differs from the one we used in Minikube.👇🏽

1. Modified the statefulset Configuration

We’re allocating 3 GiB of RAM and 1 vCPU to each replica, both as requests and limits.

This ensures that each node has enough guaranteed resources and avoids Kubernetes evicting it due to it using more than its requested resources.

We also defined a service account and annotated it with a GCP service account using the iam.gke.io/gcp-service-account annotation.

This annotation allows CockroachDB to securely access Google Cloud services (like Google Cloud Storage) without using static JSON key files (key.json), thanks to a GKE feature called Workload Identity.

In production, we let GKE handle authentication to Google services instead of mounting key files.

2. Removed podSecurityContext

In Minikube, we included this section:

...
podSecurityContext:
  fsGroup: 1000
  runAsUser: 1000
  runAsGroup: 1000
...

We did that to give CockroachDB permission to access our local disk for persistent storage. But in GKE, this isn’t needed. Google Cloud handles storage mounting securely on our behalf, so we can safely omit this part.

3. Removed podAntiAffinity and nodeSelector

In our Minikube deployment, we used:

...
podAntiAffinity:
  type: ""
nodeSelector:
  kubernetes.io/hostname: minikube
...

That was just to force all CockroachDB instances to run on the same node on Minikube.

But in production, we want each replica on a different VM. This ensures high availability, even if one VM fails, only one CockroachDB replica is affected, and the cluster stays active.

Since our cluster uses a replication factor of 3, at least 2 replicas (a quorum) need to be active for the database to stay online, else, it will crash 🥲.

4. Removed env, volumes, and volumeMounts

In Minikube, we had to manually mount the Service Account key:

...
env:
  - name: GOOGLE_APPLICATION_CREDENTIALS
    value: /var/run/gcp/key.json
volumes:
  - name: gcp-sa
    secret:
      secretName: gcs-key
volumeMounts:
  - name: gcp-sa
    mountPath: /var/run/gcp
    readOnly: true
...

This was needed so CockroachDB could access our Google Cloud Storage bucket for backups.

But in production, we don’t use key files. Instead, we use a GKE feature called Workload Identity.

It securely binds a Kubernetes Service Account to a Google Service Account, giving our CockroachDB pods the same permissions as the GCP account: no keys, no secrets, and much safer 🔒

5. Updated storage.persistentVolume.storageClass

In Minikube, we used a standard disk:

...
storage:
  persistentVolume:
    size: 5Gi
    storageClass: standard
...

But for production, we’re switching to a faster SSD:

...
storage:
  persistentVolume:
    size: 10Gi
    storageClass: premium-rwo
...

This uses Google Cloud’s pd-ssd disk type which is the recommended choice for CockroachDB due to its high IOPS (read/write operations per second) and throughput. This gives our cluster faster read and write speeds under load, leading to better performance.

6. Enabled TLS for Secure Communication

In development, we disabled TLS:

tls:
  enabled: false

That made it easier and simpler to connect without dealing with certificates.

But in production, security is non-negotiable. We’re enabling TLS to ensure that all communication with CockroachDB is encrypted in transit, and that only clients with valid certificates (signed by the same authority) can connect. This is mutual TLS (mTLS) authentication.

mTLS ensures that both sides (client and server) prove who they are, preventing impersonation or man-in-the-middle attacks. It’s one of the strongest ways to secure a production database connection.

To learn more about TLS and mTLS encryption, check out:

• Understanding Website Encryption (FreeCodeCamp)

• Mutual TLS Deep Dive (Medium)

Installing the CockroachDB Cluster on GKE

We’ll use the values file you created (cockroachdb-production.yml) and deploy our CockroachDB cluster in our GKE cluster using Helm.

Deploy the cluster

Run the following command:

helm install crdb cockroachdb/cockroachdb -f cockroachdb-production.yml

This command tells Helm to install a release named crdb using the cockroachdb/cockroachdb chart with your custom production-values file.

This step will take a few minutes. GKE will spin up 3 (or more) worker nodes to host the CockroachDB replicas.

Thanks to pod anti-affinity rules, you’ll typically see one replica pod per VM (which improves fault tolerance).

Verify the pods

Once provisioning is done, check the pods:

kubectl get pods

You should see three CockroachDB replica pods (for example: crdb-cockroachdb-0, crdb-cockroachdb-1, crdb-cockroachdb-2) in Running status.

Verify the storage class (SSD)

Now check the persistent volume claims to confirm they’re using the fast SSD storage class you requested:

kubectl get pvc

Look for your PVCs (persistent volume claims) and check the STORAGECLASS column. You should see something like premium-rwo instead of standard or standard-rwo. This confirms that your replicas are using the high-performance disk type you configured.

📌 This is important, because in production you want good disk IOPS and throughput. Slower disks can bottleneck the database.

Connecting to Our CockroachDB Cluster (Now That TLS + mTLS Are Enabled)

Now that we’ve enabled TLS encryption and mTLS authentication, let’s actually try connecting to the cluster so you can see what this security setup looks like in action.

We’ll break down in more detail what TLS and mTLS mean shortly. But for now, let’s jump straight into trying to connect – because once you see the behavior, the explanation becomes much easier to understand.

Step 1: Expose the CockroachDB Cluster to Your Local PC (Using Port Forwarding)

Just like we've been doing from the start, we’ll expose our CockroachDB cluster through port-forwarding.

Open a new terminal window and run:

kubectl port-forward svc/crdb-cockroachdb-public 26259:26257

What this means:

• The first port (26259) is the port on your computer.

• The second port (26257) is the port inside the CockroachDB cluster.

• Format is: <YOUR_COMPUTER_PORT> : <COCKROACHDB_PORT>

So now, CockroachDB will be reachable locally at localhost:26259.

Step 2: Open Beekeeper Studio and Create a Fresh Connection

If Beekeeper Studio is still connected to our old Minikube cluster, or you're not seeing the “new connection” screen, just press Ctrl + Shift + N. This opens a new connection window instantly.

Step 3: Enter the Connection Details

Now fill in these fields:

• Port: 26259

• User: root

• Default Database: defaultdb

Now click Test Connection.

And boom! You should see a message telling you something like:

“This cluster is running in secure mode. You must use SSL to connect.”

It’ll look similar to this:👇🏾

This is good: it means our CockroachDB cluster is officially in secure mode, and it’s rejecting any connection that doesn’t include proper TLS certificates.

Connecting via Mutual TLS (mTLS) — Why We Need a Certificate for Our root User

Now that our CockroachDB cluster is officially running in secure mode, we can’t just connect to it with a username and port anymore. CockroachDB won’t accept that.

To talk to it, we must connect using Mutual TLS (mTLS).

Why? Because TLS alone only protects the connection in one direction (you verifying the server). mTLS protects the connection in both directions (you verify the server, and the server also verifies you).

Let’s break this down in simple, everyday English 👇🏾

Why TLS Exists in the First Place

Whenever you send anything to CockroachDB, like a query, a connection, a password, whatever, it’s all data moving over a network – for example, the internet.

Without protection, anyone could intercept it and read the data being sent to your DB while it’s on its way
TLS fixes that :)

✔️ The CockroachDB cluster has its own public key + private key
✔️ It has a certificate that carries its public key
✔️ When you connect, the cluster sends you this certificate
✔️ Your database tool, for example Beekeeper, uses the public key in the process of encrypting all your traffic sent to the DB
✔️ Only CockroachDB can decrypt it with the help of its private key

This gives you encryption and proof you’re really talking to CockroachDB, not some fake service pretending to be it.

Why mTLS Exists (Mutual TLS)

TLS protects the server – CockroachDB. mTLS protects both sides – you and CockroachDB.

So CockroachDB also wants YOU to send your certificate.

But not just any certificate. Your certificate must be:

• Signed by THE SAME Certificate Authority (CA)

• Trusted by the CockroachDB cluster

• Mapped to a CockroachDB user (like root)

This is how CockroachDB says:

“Let me see your certificate so I know you’re someone I should allow in.”

And we reply:

“Here is my certificate, signed by the same CA that signed yours.”

At that point, both sides trust each other.

If this still feels abstract, watch this video. It explains TLS beautifully.

Let’s Explore Our Cluster’s Certificate

Remember that the Helm chart automatically created:

• The CockroachDB Certificate Authority

• The CockroachDB node certificates

• The keypairs used for encryption

You can list all the CockroachDB-related Kubernetes secrets with:

kubectl get secrets

The one we're interested in is:

crdb-cockroachdb-node-secret

If you inspect this secret, you’ll see three keys inside:

• ca.crt: the CA’s public certificate

• tls.key: the CockroachDB node’s private key

• tls.crt: the CockroachDB node certificate

Now let’s decode the CockroachDB node certificate.

Run this:

kubectl get secret crdb-cockroachdb-node-secret -o jsonpath='{.data.tls\.crt}' | base64 -d > crdb-node.crt

This gives you the raw certificate (which looks like gibberish):

-----BEGIN CERTIFICATE-----
MIIEGDCCAwCgAwIBAgIQWgOPJa4OLoZZjcXLgDF3bjANBgkqhkiG9w0BAQsFADAr
...
-----END CERTIFICATE-----

Let’s decode it into something readable:

openssl x509 -in ./crdb-node.crt -text -noout > crdb-node.crt.decoded

Open the crdb-node.crt.decoded file. This is the human-readable CockroachDB cluster certificate.

N.B.: You need to have the openssl tool installed in order to be able to make the certificate human-readable. If you don’t, install it following this tutorial.

Understanding the Certificate Sections (Explained Super Simply)

1. Issuer

You’ll see something like:

Issuer: O = Cockroach, CN = Cockroach CA

This tells us:

• The certificate was signed by a Certificate Authority created by the Helm chart

• The Organization (O) is “Cockroach”

• The Common Name (CN) is “Cockroach CA”

This basically means:

“This certificate comes from the CockroachDB internal CA.”

2. Subject

You’ll also see this:

Subject: O = Cockroach, CN = node

What does this mean?

Organization = Cockroach

• This simply groups all CockroachDB-generated certificates under one “organization label.”

• It doesn’t refer to the company. It’s just a logical grouping created by CockroachDB’s built-in toolset.

Common Name = node

• This tells CockroachDB that this certificate belongs to a cluster node, not a user or a client machine.

• In CockroachDB, node certificates are used for:

  1. DB-to-DB communication

  2. cluster gossip

  3. handling incoming connections from clients (you)

So this certificate is saying:

“Hi, I’m a CockroachDB node. Please trust me as part of the cluster.”

3. Extended Key Usage (EKU)

Scroll down and you’ll see:

X509v3 Extended Key Usage:
    TLS Web Server Authentication
    TLS Web Client Authentication

This is super important, because it defines how this certificate is allowed to be used.

Let’s simplify it:

TLS Web Server Authentication

This means:

“This certificate can be presented by a server to prove its identity.”

In our case, the CockroachDB node uses this certificate to prove to you (the client) that it is the real CockroachDB server. Think of it like flashing an ID card before letting you in.

TLS Web Client Authentication

This means:

“This certificate can also be used as a client certificate.”

Why would a server have a client certificate? Well, because in CockroachDB, nodes (DBs) talk to each other. When node A connects to node B, node A is a client, and node B is a server.

So the same certificate serves two roles. Your local machine will use a different certificate, created specifically for your root user. We’ll generate that soon.

Creating a Client Certificate (So We Can Finally Connect to CockroachDB)

Now that we’ve seen how the CockroachDB node certificate works, let’s generate our client certificate – the one we’ll use to connect from Beekeeper Studio.

Remember: CockroachDB is running in secure mode, so it won’t accept any connection that doesn’t come with a valid, signed certificate.

To fix that, let’s build a tiny Kubernetes pod whose only job is to create a certificate for our root SQL user.

Step 1: Create a File Called gen-root-cert.yml

Paste this into it:

apiVersion: v1
kind: Pod
metadata:
  name: gen-root-cert
spec:
  restartPolicy: Never
  volumes:
    - name: crdb-ca
      secret:
        secretName: crdb-cockroachdb-ca-secret
        items:
          - key: ca.crt
            path: ca.crt
          - key: ca.key
            path: ca.key
  containers:
    - name: gen
      image: cockroachdb/cockroach:v25.3.1
      command: ["sh", "-ec"]
      args:
        - |
          mkdir -p /out

          # Copy the CockroachDB cluster Certificate Authority certificate file `ca.crt` (for Mutual TLS authentication)
          cp /ca/ca.crt /out/ca.crt

          # Create the client certificate and key pair for the SQL user 'root' using the CockroachDB cluster Certificate Authority private key `ca.key`
          /cockroach/cockroach cert create-client root \
            --certs-dir=/out \
            --ca-key=/ca/ca.key \
            --lifetime=5h \
            --overwrite

          # List the generated files
          ls -al /out

          # Keep the pod alive so we can kubectl cp the files
          sleep 3600
      volumeMounts:
        - { name: crdb-ca, mountPath: /ca, readOnly: true }
      resources:
        requests:
          memory: "50Mi"
          cpu: "10m"
        limits:
          memory: "500Mi"
          cpu: "50m"

So how does this work?

We previously mentioned that the Helm chart created a secret, crdb-cockroachdb-ca-secret.

This secret contains:

• The Certificate Authority public certificate

• The private key (used for signing)

• The CA metadata

CockroachDB requires that the server certificate (node cert) and the client certificate (your root cert) be signed by THE SAME CA. Because this ensures both sides trust each other.

So what do we do?

We mount the CA secret into the pod:

volumes:
  - name: crdb-ca
    secret:
      secretName: crdb-cockroachdb-ca-secret

This gives the pod access to:

• /ca/ca.crt: CA public certificate

• /ca/ca.key: CA private key

And with these, we can sign new client certificates inside the cluster.

The important command inside the pod:

/cockroach/cockroach cert create-client root \
  --certs-dir=/out \
  --ca-key=/ca/ca.key \
  --lifetime=5h \
  --overwrite

What this does:

• Generates a brand new public/private key pair for the root SQL user

• Uses the CA private key to sign the client certificate

• Places everything inside /out

• Makes the certificate valid for 5 hours

If we passed demo instead of root, then the certificate CN would be demo, and CockroachDB would treat anyone using that certificate as the demo SQL user.

That’s how CockroachDB identifies and authenticates users when running in secure mode.

Step 2: Deploy the Pod

Run:

kubectl apply -f gen-root-cert.yml

Give it a minute to start and generate the files.

Step 3: Copy the Certificates to Your Local PC

We need three files:

• client.root.crt: client certificate

• client.root.key: private key

• ca.crt: CA certificate

Copy them from the pod to your machine:

kubectl cp default/gen-root-cert:/out/client.root.crt ./client.root.crt
kubectl cp default/gen-root-cert:/out/client.root.key ./client.root.key
kubectl cp default/gen-root-cert:/out/ca.crt             ./ca.crt

Now your folder should contain:

client.root.crt
client.root.key
ca.crt

These are the files Beekeeper Studio needs for mTLS.

Step 4: Decode the Client Certificate (Just Like We Did for the Node Certificate)

Run:

openssl x509 -in client.root.crt -text -noout > crdb-root.crt.decoded

Open the crdb-root.crt.decoded file and look at the contents.

Understanding the Client Certificate

1. Issuer

You'll see Issuer: O = Cockroach, CN = Cockroach CA

This is the same Issuer as the CockroachDB node certificate.

This confirms that both certificates were signed by the same Certificate Authority, that they trust each other, and that mTLS will work perfectly.

2. Subject

You’ll see: Subject: O = Cockroach, CN = root

This means that the Organization is just a label grouping CockroachDB identities, and that the Common Name is root. This is VERY important.

The CN of a client certificate literally tells CockroachDB:

“This connection belongs to the SQL user named root.”

If CN was demo, CockroachDB would authenticate you as the demo SQL user.

Extended Key Usage (EKU)

You should see: TLS Web Client Authentication.

This is exactly what we want. It tells CockroachDB:

“This certificate is only for clients connecting to the database.”

Unlike node certificates, you will NOT see: TLS Web Server Authentication.

Why?

Because:

• Server Authentication = for certificates the SERVER SHOWS TO THE CLIENT. For example: CockroachDB nodes proving they are legitimate.

• Client Authentication = for certificates THE CLIENT SENDS TO THE SERVER. For example: You proving you are the real root user.

Why your client certificate cannot be used as a server certificate

Because a server certificate says:

“Trust me, I AM the CockroachDB server.”

But your client certificate says:

“Trust me, I am an authenticated user.”

Two very different identities. And CockroachDB will reject any certificate used in the wrong role.

So having only TLS Web Client Authentication in your certificate is perfect for our use case. :)

Connecting to Our CockroachDB Cluster Securely (Using mTLS)

Now that we’ve successfully generated the certificates and key pairs we need, it's time to use them to securely connect to our CockroachDB cluster from Beekeeper Studio.

Remember: CockroachDB is running in secure mode, so without these certificates, it will reject all incoming connections, even if you enter the correct username and password.

Let’s walk through the steps.👇🏾

Step 1: Make Sure Port Forwarding Is Still Running

Before connecting, ensure that your CockroachDB cluster is still exposed to your PC.

If you already closed the previous terminal window, simply re-run this:

kubectl port-forward svc/crdb-cockroachdb-public 26259:26257

This makes your CockroachDB node reachable at: localhost:26259. If this step isn’t active, Beekeeper Studio will not be able to connect.

Step 2: Open Beekeeper Studio and Set Up the Connection

Launch Beekeeper Studio and open a fresh connection window (Ctrl + Shift + N if needed).

Now fill in the fields like this:

Now enable the “Enable SSL” option. Once enabled, expand the SSL section and set the following three fields:

• CA Cert: Set this to the location of: ca.crt. This is the root Certificate Authority file you copied earlier using: kubectl cp default/gen-root-cert:/out/ca.crt ./ca.crt. It should still be in your project’s root directory (for example, cockroachdb-tutorial/).

• Certificate: Set this to the location of: client.root.crt

• Key File: Set this to the location of: client.root.key

Step 3: Click “Connect”

Once all the fields are set properly, click Connect.

If everything was done correctly, you should now be connected to your CockroachDB cluster securely over Mutual TLS.

If the connection fails:

• Double-check your certificate paths

• Ensure port-forwarding is running

• Verify the user is root

• Confirm the selected connection type is CockroachDB.

Step 4: Run Your First Secure Query

Now that you're connected, let’s verify everything works by running:

SHOW users;

You should see two users automatically created by CockroachDB:

• admin

• root

In the next subsection, we’ll create a new SQL user and generate a certificate for that user (just like we did for the root user) so you’ll understand how CockroachDB handles user authentication in production environments.

Restoring Our Previous Database into the New GKE CockroachDB Cluster (without SA keys)

Now that our CockroachDB cluster is up and running on GKE – fully secured with TLS encryption and mTLS authentication – it’s time to bring back the data from our previous setup.

Remember how we backed up our CockroachDB database (running on Minikube) to Google Cloud Storage?

Well, now we’re going to restore that same backup into our new production cluster on GKE. But before CockroachDB can access our bucket, we must give it permission – securely.

And here’s the cool part: we don’t need to use Service Account keys anymore.

Why We Don’t Need Service Account Keys on GKE

Earlier, in the backup section, we generated a Service Account key on our PC and mounted it into our Minikube cluster.

But for GKE, we intentionally left out the following fields in our cockroachdb-production.yml:

• env

• volumes

• volumeMounts

The reason? GKE supports something called Workload Identity.

Workload Identity lets us securely connect Kubernetes Service Accounts (KSAs) to Google Cloud Service Accounts (GSAs), without storing or mounting any secret keys. The authentication happens “implicitly” thanks to Google’s metadata server.

💡 Workload Identity works easily when your cluster is running on GKE. It’s more complex to set up on Minikube, Kind, EKS, AKS, or any other non-GKE cluster.

Step 1: Linking the Google Service Account to Our Kubernetes Service Account

We already touched this when deploying our cluster, but let’s look at the specific line again.

Open your cockroachdb-production.yml Helm values file and scroll to the serviceAccount section. You should see something like this:

...
serviceAccount:
    create: true
    name: "crdb-cockroachdb"
    annotations:
      iam.gke.io/gcp-service-account: cockroachdb-backup@<PROJECT_ID>.iam.gserviceaccount.com
...

Replace the <PROJECT_ID> placeholder with your real Google Cloud project ID.

If you’re unsure of the ID, go to Google Cloud Console, then to IAM & Admin, and finally to Service Accounts. Search for cockroachdb-backup and copy the project ID from there.

This annotation instructs GKE to automatically authenticate our CockroachDB pods as the cockroachdb-backup Google Service Account – no keys needed.

Step 2: Binding KSA ↔️ GSA Using Workload Identity

Annotating the Service Account isn’t enough. We still need to explicitly allow our KSA to “impersonate" the GSA.

Run this command to set the active project:

gcloud config set project <PROJECT_ID>

Now, apply the IAM policy binding:

gcloud iam service-accounts add-iam-policy-binding \
  <GOOGLE_SERVICE_ACCOUNT> \
  --role roles/iam.workloadIdentityUser \
  --member "serviceAccount:<PROJECT_ID>.svc.id.goog[<NAMESPACE>/<KUBERNETES_SERVICE_ACCOUNT>]"

Replace the placeholders with:

• <GOOGLE_SERVICE_ACCOUNT> with cockroachdb-backup@<PROJECT_ID>.iam.gserviceaccount.com

• <PROJECT_ID> with your GCP project ID

• <NAMESPACE> with where CockroachDB runs (default)

• <KUBERNETES_SERVICE_ACCOUNT> with crdb-cockroachdb

After a few seconds, you should see something like:

Updated IAM policy for serviceAccount [cockroachdb-backup@<PROJECT_ID>.iam.gserviceaccount.com].
bindings:
- members:
  - serviceAccount:<PROJECT_ID>.svc.id.goog[default/crdb-cockroachdb]
  role: roles/iam.workloadIdentityUser
etag: ***
version: 1

Perfect. Your KSA can now access Google Cloud Storage automatically.

Restoring Our Previous Database from Google Cloud Storage

Now that authentication is set up, let’s restore the backup we previously created in the Minikube cluster.

Open Beekeeper Studio and reconnect to your CockroachDB cluster (the one running on GKE).

Before restoring anything, let’s check if the books table exists:

SELECT * FROM books;

You should see an error saying the table doesn’t exist. Don’t worry, that’s expected.

Now, Let’s Restore the Data 🎉

Run this command:

RESTORE FROM LATEST IN 'gs://<BUCKET_NAME>/cluster?AUTH=implicit';

Replace <BUCKET_NAME> with the name of the bucket you created earlier (for example: cockroachdb-backup-7gw8u).

CockroachDB will now:

• Authenticate using Workload Identity

• Find the latest backup inside your bucket

• Restore all tables, schemas, and data into your new GKE cluster

After a couple of minutes, you should get a Success message.

Now, run the query again:

SELECT * FROM books;

Boom! Your books from the Minikube cluster should now appear inside the new CockroachDB cluster running on GKE 😃.

Connecting to the Database with a New User

So far, we’ve been connecting to our CockroachDB cluster using the root user. While this is super convenient for tutorials, it’s not recommended for real apps.

This is because the root user has advanced privileges – basically, full access to your entire cluster. If an attacker got hold of these credentials, or your application was compromised, they could do A LOT of damage. 😬

Instead, it’s best practice to create a user with limited permissions for your apps. This way, even if the user is compromised, the damage is contained.

Authentication Options for Users

CockroachDB is flexible when it comes to authentication:

1. Password Authentication: Create a user with a password and connect using just username + password (no client certificates required).

2. Passwordless / Mutual TLS Authentication: Create a user without a password, then connect using client certificates signed by the same CA (like we did for root).

3. Both Password + Mutual TLS: Create a user with a password and also connect using client certificates. This adds an extra layer of security.

In this subsection, we’ll start simple and use password authentication.

Step 1: Create the New User

Open your current connection in Beekeeper Studio (signed in as root) and run:

CREATE USER password_auth WITH PASSWORD 'supersecret';

You should see a message confirming the user was created successfully.

Step 2: Connect as the New User

Open a new Beekeeper Studio window (Ctrl + Shift + N). DO NOT exit/close the old window, as we’ll need it later.

Fill in the connection fields:

Click Connect.

❌ You’ll see an error about SSL connection being required.

Even though we’re connecting with a password instead of certificates, enabling SSL is still important. It encrypts the data between Beekeeper Studio and CockroachDB.

Without it, sensitive info like passwords and queries could be intercepted (man-in-the-middle attacks).

Step 3 — Enable SSL & CA Verification

• Tick Enable SSL

• Click the CA Cert field and select the ca.crt file in your project root (cockroachdb-tutorial/)

This ensures that Beekeeper Studio verifies it’s really talking to our CockroachDB cluster and protects against attackers trying to intercept the connection.

Now, click Connect again.

❌ Initially, you’ll still see a Password authentication failed error because we intentionally entered the wrong password.

Step 4: Connect With the Correct Password

Replace the password with supersecret, then click Connect.

You are now signed in as the password_auth user!

Step 5: Check Permissions

Run:

SELECT * FROM books;

❌ You should see an error stating that password_auth does not have permission to access the books table.

This is expected, as it confirms that our limited-access user can only access what we explicitly grant it. Even if compromised, the attacker can’t modify our entire database.

Step 6: Granting Access to Specific Tables

To allow password_auth to work with the books table, switch back to the root connection Beekeeper Studio window and run:

GRANT USAGE ON SCHEMA defaultdb.public TO password_auth;
GRANT SELECT, INSERT, UPDATE, DELETE ON TABLE defaultdb.public.books TO password_auth;

This gives the user read and write access to the books table only.

Step 7: Verify the New User Access

Go back to the Beekeeper Studio window where you’re signed in as password_auth and run:

SELECT * FROM books;

Boom! You should now see the list of books from your restored database.

Our new user is fully functional with limited privileges, making it safe for use in real applications.

Connecting with Passwordless Authentication (Mutual TLS)

We’ve already seen how to connect to the database using a user that authenticates with a password, and without any client certificates.

Now, let’s look at the opposite scenario: passwordless authentication via Mutual TLS (mTLS).

This is one of the strongest forms of authentication because instead of a password, the database verifies you using a cryptographically signed certificate.

Let’s walk through it.

Step 1: Create the mtls_auth User

Navigate back to the Beekeeper Studio window where you're currently signed in as the root user. Run:

CREATE USER mtls_auth;

You should see a success message confirming that the user has been created.

N.B.: If this query fails, there’s a good chance your root client certificate has expired. Remember that we set a 5-hour lifetime when generating it earlier.

If this happens, delete the certificate-generation pod:

kubectl delete po/gen-root-cert

Then re-apply the gen-root-cert.yml manifest. Copy the newly generated client.root.crt, client.root.key, and ca.crt back to your PC. Then try creating the user again.

Step 2: Attempt Signing In as mtls_auth (Expect Failure)

Open a new Beekeeper Studio window (Ctrl + Shift + N).

Try filling in the connection settings using:

• User: mtls_auth

• SSL enabled

• CA Cert: ca.crt

• Client Cert: client.root.crt

• Client Key: client.root.key

Click Connect.

You’ll see an error message similar to this:

Why does this fail?

1. The user has no password, so password login is impossible.

2. You’re using the root certificate, not a certificate belonging to mtls_auth. CockroachDB is strict: each user must authenticate using their own certificate.

So let's fix that by generating a new certificate + key pair for the mtls_auth user.

Step 3: Create Certificate + Key for mtls_auth

Just like we generated certificates for the root user earlier, we’ll do the same for mtls_auth.

Create a new manifest named gen-mtls_auth-cert.yml.

Paste in this content:

apiVersion: v1
kind: Pod
metadata:
  name: gen-mtls-auth-cert 
spec:
  restartPolicy: Never
  volumes:
    - name: crdb-ca
      secret:
        secretName: crdb-cockroachdb-ca-secret 
        items:
          - key: ca.crt
            path: ca.crt
          - key: ca.key
            path: ca.key
  containers:
    - name: gen
      image: cockroachdb/cockroach:v25.3.1
      command: ["sh", "-ec"]
      args:
        - |
          mkdir -p /out

          # Copy the CA certificate
          cp /ca/ca.crt /out/ca.crt

          # Create the client certificate and key pair for user 'mtls_auth'
          /cockroach/cockroach cert create-client mtls_auth \
            --certs-dir=/out \
            --ca-key=/ca/ca.key \
            --lifetime=5h \
            --overwrite

          # List generated files
          ls -al /out

          # Keep pod alive for kubectl cp
          sleep 3600
      volumeMounts:
        - { name: crdb-ca, mountPath: /ca, readOnly: true }
      resources:
        requests:
          memory: "50Mi"
          cpu: "10m"
        limits:
          memory: "500Mi"
          cpu: "50m"

Apply this file, wait for the pod to start, then copy the generated files:

kubectl cp default/gen-mtls-auth-cert:/out/client.mtls_auth.crt ./client.mtls_auth.crt 
kubectl cp default/gen-mtls-auth-cert:/out/client.mtls_auth.key ./client.mtls_auth.key
kubectl cp default/gen-mtls-auth-cert:/out/ca.crt ./ca.crt

Now we have the correct certificate + key pair for our new user.

Step 4: Connect as mtls_auth

Go back to the new Beekeeper Studio window and update the SSL fields:

• CA Cert: ca.crt

• Certificate: client.mtls_auth.crt

• Key File: client.mtls_auth.key

Click Connect.

This time, it should succeed instantly

Step 5 — Inspect the Certificate

To understand how CockroachDB links certificates to users, decode the certificate:

openssl x509 -in client.mtls_auth.crt -text -noout > client.mtls_auth.crt.decoded

Open the file, scroll to the Subject field, and you’ll see:

...
Subject: O = Cockroach, CN = mtls_auth
...

The CN (Common Name) is the username CockroachDB uses to authenticate the session.

This is how CockroachDB knows you’re connecting as the mtls_auth user without any password at all. :)

Step 6: Try Reading the Books Table

Run:

SELECT * FROM books;

❌ You’ll get a permission error, just like we did earlier with the password_auth user.

This is expected because mtls_auth has no privileges yet. Perfect!

Step 7: Grant Permissions to mtls_auth

Switch to the Beekeeper Studio window where you're signed in as root, and run:

GRANT USAGE ON SCHEMA defaultdb.public TO mtls_auth;
GRANT SELECT, INSERT, UPDATE, DELETE ON TABLE defaultdb.public.books TO mtls_auth;

You should see a success message.

Now return to the mtls_auth session and run:

SELECT * FROM books;

Boom! You should now see your previously restored list of books.

You’ve successfully connected using passwordless, certificate-based authentication and granted controlled permissions to the new user. :)

Connecting via Mutual TLS (mTLS) from Our Apps on Kubernetes

So far, we’ve been connecting to our CockroachDB cluster securely using Beekeeper Studio thanks to our TLS certificates and mTLS authentication.

But…what happens when we have applications running inside our Kubernetes cluster that need to talk to CockroachDB as well?

Exactly: those apps also need to authenticate using client certificates

And that brings us to a very important point…

Why We Should Not Generate Client Certificates Using Pods (The Dangerous Way)

Up until now, we’ve been generating our client certificates using Kubernetes Pods like:

• gen-root-cert

• gen-mtls-auth-cert

They work, yes…but they’re not safe for production.

Why? Because these jobs mount our Certificate Authority (CA) key inside the pod:

...
volumes:
    - name: crdb-ca
      secret:
        secretName: crdb-cockroachdb-ca-secret
        items:
          - key: ca.crt
            path: ca.crt
          - key: ca.key
            path: ca.key
...

This is a big security risk!

If an attacker ever gains access to that pod?

🔥 Your CA key is exposed
🔥 They can generate their own trusted certificates
🔥 They can impersonate ANY client/user, including the root and admin users
🔥 They’ll have full access to your CockroachDB cluster

And they’ll keep that access forever, until you rotate the CA key (which is painful and disruptive).

This is why CockroachDB strongly advises against mounting CA keys into Pods.

The Right Way: Using Cert Manager (Recommended by CockroachDB)

CockroachDB’s official docs recommend managing client certificates using cert-manager.

This is because instead of YOU exposing your CA key inside Pods, cert-manager handles everything internally and securely:

• Cert-manager stores and protects your CA key

• It generates client certificates for you

• It issues private keys without ever exposing your CA key

• It auto-renews certificates before they expire

• And it gives you production-grade certificate lifecycle management

But Wait: Don’t We Need the CA Key to Generate Client Certificates?

Great question.

Yes, normally you need the CA key to sign client certificates…but cert-manager takes care of that for us.

You simply:

1. Create an Issuer (or ClusterIssuer)

2. Tell cert-manager to use your CockroachDB CA

3. Request a Certificate

Then cert-manager automatically:

1. Signs it

2. Stores it in a Kubernetes Secret (where its safe)

3. Rotates it before expiry

4. Keeps your CA key completely secure

No more exposing the CA key in Pods. No more writing custom Kubernetes Pods.

Certificate Rotation — Another Huge Win

Let’s talk about expirations.

Right now:

• The mtls_auth client cert we generated manually has 5 hours validity

• After 5 hours, it expires

• Your apps will fail all DB connections

• You’d need to regenerate a new certificate manually

• Or worse: create a CronJob to regenerate them every 4 hours

This is messy and unsafe.

With cert-manager?

• Certificates are automatically rotated

• Renewed before expiration

• No downtime

• No manual intervention

• Apps easily reload the new certificates

Alright — Let’s Install Cert Manager

To start using cert-manager, install it using the Helm chart:

helm repo add cert-manager https://charts.jetstack.io

helm install cert-manager cert-manager/cert-manager \
  --set crds.enabled=true \
  --create-namespace \
  -n cert-manager \
  --version 1.19.1

Once cert-manager is installed, we’ll:

1. Create a ClusterIssuer that uses our CockroachDB CA

2. Create a Certificate for our mtls_auth user

3. Mount that Certificate into our application Pods

4. Connect securely to CockroachDB via mTLS from inside Kubernetes

That’s what we’ll walk through next

Before cert-manager can issue our certificates, it needs an Issuer. And before creating an Issuer, we need a secret that contains our CA certificate and CA key using the correct key names.

Creating a CA Secret for the Issuer

cert-manager’s Issuer is a bit picky about the secret format. It expects the secret to contain two keys:

• tls.crt: the CA certificate

• tls.key: the CA private key

But\ the CockroachDB Helm chart automatically generates a secret named crdb-cockroachdb-ca-secret, which uses different key names:

• ca.crt

• ca.key

So even though this secret contains exactly what we need, cert-manager won’t accept it because the keys are not named the way it expects.

To fix this, we’ll re-create a new secret with the correct key names. First, copy the existing CA files from Kubernetes to your local machine:

kubectl get secret crdb-cockroachdb-ca-secret -o jsonpath='{.data.ca\.crt}' | base64 -d > ca.crt

If you get a “permission denied”, simply delete any existing ca.crt file in your project directory.

Now copy the key:

kubectl get secret crdb-cockroachdb-ca-secret -o jsonpath='{.data.ca\.key}' | base64 -d > ca.key

Next, create the properly formatted secret:

kubectl create secret tls crdb-ca-issuer-secret --cert=ca.crt --key=ca.key

If you describe it:

kubectl describe secret crdb-ca-issuer-secret

You should now see tls.crt and tls.key in the Data section – exactly what cert-manager needs.

Creating the Issuer

Now that we have a properly formatted CA secret, we can create the Issuer that cert-manager will use to sign our client certificates.

Create a file called crdb-issuer.yml:

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: crdb-issuer
spec:
  ca:
    secretName: crdb-ca-issuer-secret

Apply it:

kubectl apply -f crdb-issuer.yml

Confirm that it’s ready:

kubectl get issuer crdb-issuer

The Ready column should display True.

Creating the Certificate Manifest

Now we’ll define a Certificate object. This doesn’t create the client certificate instantly – instead, it tells cert-manager what kind of certificate we need. cert-manager then generates and stores the certificate automatically.

Create a file named crdb-mtls_auth-certificate.yml:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: crdb-mtls-auth-certificate
spec:
  secretName: crdb-mtls-auth-certificate # Secret that will hold the cert+key
  commonName: mtls_auth # MUST match Cockroach SQL role
  duration: 24h # 1 day
  renewBefore: 20h # renew 4 hours before expiry
  privateKey:
    algorithm: RSA
    size: 2048
    encoding: PKCS8
  usages:
    - client auth # important: client certificate
  issuerRef:
    name: crdb-issuer
    kind: Issuer
    group: cert-manager.io

Let’s look at the important properties so we can understand what the Certificate workload does:

• secretName: The Kubernetes secret where cert-manager will store the generated certificate, key, and CA certificate. This is where your apps will later mount the certificate files from.

• commonName: Very important! This must match the CockroachDB SQL user (mtls_auth), because CockroachDB uses the certificate’s Common Name to identify the connecting user.

• duration and renewBefore: duration defines how long the certificate is valid. renewBefore ensures cert-manager renews it early, preventing the certificate from getting expired before it gets renewed (to avoid downtime).

• usages: Tells cert-manager what the certificate is for. client auth ensures this certificate is only used by clients connecting to servers, not the other way around.

• issuerRef: Points to the Issuer we created earlier. This tells cert-manager who should sign the certificate.

Apply the manifest:

kubectl apply -f crdb-mtls_auth-certificate.yml

After a few seconds, cert-manager will generate the certificate.

Check the secret:

kubectl get secret crdb-mtls-auth-certificate

Describe it to view the keys:

kubectl describe secret crdb-mtls-auth-certificate

You should see:

• tls.crt

• tls.key

• ca.crt

These are the files the application will use.

If we copied the content of the tls.crt to our local machine and decoded it using the openssl x509... command, we'll see similar details to the content in the client.mtls_auth.crt client certificate we previously generated, with the Common Name (CN being mtls_auth).

Creating a Pod That Connects Using the Client Certificate

Now let’s create a simple Pod that uses our new client certificate to connect to CockroachDB.

Create a file called books-pod.yml:

apiVersion: v1
kind: Pod
metadata:
  name: books-pod
spec:
  restartPolicy: Never
  volumes:
    - name: crdb-certs
      secret:
        secretName: crdb-mtls-auth-certificate
        # Make secret files read-only for the user only: 0400 (Without this, the Python app will thow an error). Howevwe, this is not compulsory for all apps, just this one being used in this tutorial :)
        defaultMode: 0400
  containers:
    - name: books
      image: prince2006/cockroachdb-tutorial-python-app:new
      imagePullPolicy: Always
      env:
        - name: DATABASE_URL
          value: >-
            postgresql://mtls_auth@crdb-cockroachdb-public.default:26257/defaultdb?sslmode=verify-full&sslrootcert=/crdb-certs/ca.crt&sslcert=/crdb-certs/tls.crt&sslkey=/crdb-certs/tls.key
      volumeMounts:
        - name: crdb-certs
          mountPath: /crdb-certs
          readOnly: true
      resources:
        limits:
          memory: "100Mi"
          cpu: "50m"
        requests:
          memory: "50Mi"
          cpu: "10m"

Here’s what’s happening:

• We mount the generated certificate secret into /crdb-certs.

• The Python app uses those certificate files (tls.crt, tls.key, ca.crt) to authenticate.

• The connection string does NOT include a password. CockroachDB authenticates the user entirely via the certificate’s Common Name.

Apply the Pod:

kubectl apply -f books-pod.yml

After about a minute, view the logs:

kubectl logs books-pod

Or if the Pod already restarted:

kubectl logs -p books-pod

You should see a successful connection to CockroachDB using the mtls_auth user and a list of books

If you remove the certificate files or try connecting without them, the app will fail – as expected.

Congratulations!

You’ve officially built a fully secure, production-ready CockroachDB cluster on Kubernetes – complete with:

• End-to-end encryption (TLS)

• Mutual TLS authentication (mTLS) for users and apps

• Automated, daily backups to Google Cloud Storage

• Proper certificate rotation with cert-manager

How to Get a CockroachDB Enterprise License for Free

Okay, so here’s a thing: even though you’ve built a super professional CockroachDB cluster, there’s one small catch: without a license, your cluster might be “throttled.”

We know that because, when we access our dashboard, we get a message concerning our cluster getting throttled.

That means things slow down: queries take longer, performance gets worse, and scaling up won’t magically make it faster. Yeah, it’s real. 🥲

Why does this happen? Because CockroachDB’s “full feature set” is under a special license. If you don’t set a valid license, it limits how many SQL transactions you can run at a time.

Three Types of Licenses

Here’s a breakdown of the different kinds of CockroachDB licenses and what they mean for you:

1. Trial License

   • Valid for 30 days.

   • Lets you try all the “Enterprise” features.

   • You must send telemetry (more on that soon) while the trial is active.

2. Enterprise License (Paid)

   • This is CockroachDB’s “premium / fully paid” version.

   • You can pick the kind of license based on your environment: “Production”, “Pre-production”, or “Development.”

   • Companies with more than $10 million in annual revenue need to pay for this license.

   • There are discounts, startup perks, or “free” versions for smaller companies (more below).

3. Enterprise Free License

   • This is the magic one for early-stage companies or startups: it has exactly the same features as the paid Enterprise license. But it’s free if your business makes under $10 million per year.

   • You do need to renew it each year.

   • Support for this “Free” license is community-level (forums, docs), not paid enterprise.

N.B.: To keep your free license active and not get throttled, CockroachDB requires telemetry. Telemetry means your cluster sends some usage data back to Cockroach Labs. And no, they’re not “stealing your data”. Here’s what that actually means:

• Telemetry includes basic usage stats, cluster health info, and configuration metrics.

• It does NOT send your business data, queries, or personal customer data.

• It helps Cockroach Labs make sure the free license is used responsibly, and helps them build better features.

• If you stop sending telemetry, your cluster will eventually be throttled after 7 days (slowed down).

How to Apply for the Free Enterprise License

Here’s how you can try to get that free enterprise license:

1. Go to the CockroachDB Cloud Console (Sign up if you don’t have a account). Then go to the “Organization” link on the menu, click it, then click the “Enterprise Licenses” from the dropdown.

2. Click the Create License button → Enable the “Find out if my company qualifies for an Enterprise Free license” option.

3. Fill in the form: your name, company name, job function, and the intended use of the license.

4. Click “Continue”.

You should see this success message “Based on your company's intended use, you qualify for an Enterprise Free license.” Now agree to the terms and conditions, then click the “Generate License key“.

Learn more about CockroachDB licenses here 👉🏾 https://www.cockroachlabs.com/docs/stable/licensing-faqs

Adding Your License to the CockroachDB Cluster

Now that you’ve gotten your shiny new CockroachDB license (whether it’s the Free one or the Enterprise one), the next step is…actually using it.

Let’s add it to your CockroachDB cluster so it stops shouting “THROTTLED!” at you every time you open the dashboard :)

We’ll do this by updating our CockroachDB Helm configuration.

Step 1: Update Your cockroachdb-production.yml

Open your production Helm values file, and inside the init section, add the following:

init:
...
    provisioning:
        enabled: true
        clusterSettings:
          cluster.organization: "'<ORGANIZATION>'" # Enter the name of your organization here 
          enterprise.license: "'<LICENSE>'" # Enter your CockroachDB Enterprise license key here
...

Now replace:

• <ORGANIZATION> with the name of your startup, business, project, or company

• <LICENSE> with the exact license string CockroachDB gave you

That’s it – super simple.

Step 2: Apply the Changes With Helm

Run your usual Helm upgrade command:

helm upgrade cockroachdb -f cockroachdb-production.yml cockroachdb/cockroachdb

Step 3: Confirm the License Was Added Correctly

Now let’s double-check everything worked.

1. Connect as the root user: You can connect using Beekeeper Studio (like we’ve been doing).

2. Run this query to check your license:

SHOW CLUSTER SETTING enterprise.license;

If everything went well, you should see your license key printed out in the results.

Step 4: Make Sure Telemetry Is Enabled (Important!)

Remember: without telemetry enabled, your cluster will still get throttled, even if you have a valid license 🥲

Run:

SHOW CLUSTER SETTING diagnostics.reporting.enabled;

If the result says “true”, you're good! Telemetry is on, CockroachDB can verify your license, and your cluster will behave normally without slowing down.

Conclusion & Next Steps ✨

Throughout this book, you’ve gone from “What even is CockroachDB?” to actually running your own secure, production-ready database on Kubernetes – and that’s a BIG deal. 🎉

You learned why CockroachDB is special, how it avoids downtime, and why it’s different from the usual databases everyone talks about.

Then you set up your own local environment, practiced everything safely on Minikube, and gradually built your way to a full production setup on GKE.

You explored CockroachDB’s dashboard, checked your cluster’s health, backed up your data to the cloud, and even learned how to keep your database fast, stable, and ready to grow when needed.

Finally, you deployed it on Google Cloud, secured it with encryption and certificates, and connected to it from your own PC – all step-by-step.

By now, you’ve basically gone from curious learner to “I can actually run this thing in production.” 🚀

You’ve covered a lot – and you’ve built something powerful, modern, and production-worthy. Amazing job 👏🏾😁!! And thanks for reading.

About the Author 👨🏾‍💻

Hi, I’m Prince! I’m a DevOps engineer and Cloud architect passionate about building, deploying, architecting, and managing applications and sharing knowledge with the tech community.

If you enjoyed this book, you can learn more about me by exploring more of my blogs and projects on my LinkedIn profile. and reach out to me on Twitter (X). You can find more of my articles here or on my freeCodeCamp blog.

You can also visit my website. Let’s connect and grow together! 😊

But if you examine BigQuery's implementation more closely, you’ll find an approach that goes beyond a simple feature addition. Instead of bolting on a vector library, Google has deeply integrated vector search into its existing distributed, columnar architecture.

In this article, we’ll take a technical deep dive into the engineering decisions behind BigQuery's vector search. We’ll explore how foundational Google technologies like Dremel, Borg, and Colossus, combined with a proprietary columnar format and a novel indexing algorithm, create a highly scalable and efficient platform for AI workloads.

This analysis will give you insights into the architectural trade-offs involved in building vector search at scale. It also demonstrates how you can adapt a system designed for large-scale analytics so that it excels at modern AI tasks.

Table of Contents

• Prerequisites

• The Unique Challenge of Vector Search

• BigQuery's Foundational Distributed Architecture

  • Dremel: The Distributed Query Engine

  • Borg: Cluster Management and Resource Orchestration

  • Colossus: The Distributed Storage Layer

  • Jupiter: The High-Speed Network Fabric

• The Role of Columnar Storage in Vector Operations

  • Accelerating Computations with SIMD

• The TreeAH Indexing Algorithm

  • 1. Hierarchical Tree Structure

  • 2. Product Quantization (PQ)

  • 3. Asymmetric Hashing

  • Architectural Comparison: TreeAH vs. HNSW

• The End-to-End Vector Search Query Flow

• Practical Implications for Engineering Teams

  • 1. Query Latency vs. Throughput

  • 2. Cost Model Considerations

  • 3. Index Management Trade-offs

  • 4. Integration Benefits That Actually Matter

• Conclusion

• Further Reading

Prerequisites

This article assumes that you have a solid foundation in distributed systems and database internals, including familiarity with concepts like columnar storage, query execution plans, and distributed query processing.

You should understand the basics of vector embeddings and similarity search, though we'll briefly review the fundamentals. Experience with at least one vector database or search system (such as pgvector, Pinecone, or Elasticsearch) will help contextualize the architectural comparisons.

While deep knowledge of Google Cloud Platform isn't required, basic familiarity with cloud data warehouses and their typical architectures will be beneficial. The article includes discussions of SIMD operations and CPU-level optimizations, so comfort with low-level performance considerations is helpful, though not mandatory.

Code examples assume working knowledge of SQL, with some sections referencing implementation details in languages like Python or Java. Most importantly, you should have experience building or operating production data systems at scale, as many insights focus on practical engineering trade-offs rather than theoretical concepts.

The Unique Challenge of Vector Search

Vector search fundamentally differs from traditional database operations in ways that challenge our existing infrastructure assumptions. Where conventional queries leverage decades of optimization around exact matching and range scans, vector similarity search requires computing distances between high-dimensional points at massive scale.

Consider the numbers. Modern embedding models produce vectors with 768 or more dimensions. At 4 bytes per float32 value, a single embedding consumes roughly 3KB. A modest corpus of 100 million items translates to 300GB of vector data.

But the real challenge isn't storage. The killer is computation. Finding the nearest neighbors to a query vector means computing distance metrics across all those dimensions. For 100 million vectors, a brute-force search requires 76.8 billion floating-point operations per query just for the distance calculations. Even with modern SIMD instructions processing 16 floats at once, you're looking at billions of CPU cycles per search.

This computational reality forces a fundamental compromise: we abandon exact solutions for approximate ones. Approximate Nearest Neighbor (ANN) algorithms trade perfect accuracy for practical query times. They work by partitioning the vector space cleverly, building graphs of nearest neighbors, or using hashing schemes to avoid examining every vector. The engineering challenge becomes balancing query latency, recall accuracy, and resource consumption.

Most purpose-built vector databases address this through specialized in-memory indexes like HNSW or IVF. These work well for single queries but require keeping massive indexes in RAM. In case you are not familiar with these vector indexes, you can read this article.

BigQuery took a different path. Rather than optimizing for single-query latency, they asked what vector search would look like when built for analytical workloads at warehouse scale. The answer required rethinking basic assumptions about index design, storage layout, and query execution.

BigQuery's Foundational Distributed Architecture

BigQuery's vector search runs on the same infrastructure that's been processing SQL queries since 2011. No new cluster type. No specialized vector nodes. Just four core technologies that power most of Google's data processing, now handling a workload they weren't originally designed for.

This isn't the obvious choice. Most vector databases build specialized infrastructure optimized for similarity search. Graph-based indexes need fast random access. In-memory systems require careful memory management. BigQuery took its existing distributed SQL engine and asked: can we make this work for vectors, too?

The answer required leveraging four foundational systems in new ways:

• Dremel, the query engine that normally handles SQL, now orchestrates vector similarity computations.

• Borg, which allocates resources for everything from Search to YouTube, dynamically assigns thousands of workers to vector queries.

• Colossus stores embeddings in the same distributed filesystem that holds petabytes of analytics data.

• And Jupiter's datacenter network, built for bulk data processing, now shuttles vector data between computation nodes.

What's surprising isn't that it works, but how well it works. The same architecture that runs aggregate queries over trillion-row tables can search billion-scale vector collections. Understanding how requires examining each component and how they've been adapted for this new workload.

Dremel: The Distributed Query Engine

At its core, BigQuery is powered by Dremel, a distributed query execution engine developed at Google since 2006.

Dremel processes SQL queries using a hierarchical serving tree. A root server receives the query and orchestrates the execution, while mixer nodes break down the work and distribute it to hundreds or thousands of leaf nodes. These leaf nodes perform the actual computations in parallel on segments of the data.

This architecture allows BigQuery to dynamically allocate a massive number of execution threads, known as slots, to a single query, enabling it to process petabytes of data in seconds.

Borg: Cluster Management and Resource Orchestration

The serverless nature of BigQuery is made possible by Borg, Google's cluster management system that predates and inspired Kubernetes.

When a vector search query is submitted, Borg is responsible for finding available machines across Google's global data centers, allocating the precise amount of CPU and memory resources needed for the query's Dremel slots, and managing fault tolerance by automatically rescheduling work if a machine fails. This dynamic resource allocation means users do not need to provision or scale infrastructure, whether they are searching 1,000 vectors or 10 billion.

Colossus: The Distributed Storage Layer

Data in BigQuery is stored in Colossus, Google's next-generation distributed file system. Colossus is designed for exabyte-scale storage, provides high availability through automatic cross-datacenter replication, and is optimized for the high-throughput parallel reads required by Dremel's leaf nodes.

During a vector search, Colossus can deliver data to thousands of nodes simultaneously without creating a storage bottleneck.

Jupiter: The High-Speed Network Fabric

These compute and storage systems are interconnected by Jupiter, Google's internal datacenter network, which features a petabit-per-second bisection bandwidth. The network's design ensures that data can move between Colossus storage and Dremel compute nodes at extremely high speeds, making data shuffling and aggregation phases of a query efficient.

The Role of Columnar Storage in Vector Operations

Storing vectors in columns sounds wrong. Vectors are arrays. They belong together. Why split them across columnar storage?

BigQuery does it anyway, and it works brilliantly. Here's why.

When you search a million vectors, you need exactly one thing from each row: the embedding. Not the product name, price, or category. Just the vector. Row-oriented storage forces you to read entire records and throw away 90% of the data. Columnar storage reads only what you need.

The performance impact is dramatic. A table with 768-dimensional embeddings plus 20 other columns might total 3TB. Reading just the embedding column? 300GB. That's a 10x reduction in I/O before you've done any actual computation.

But the real magic happens at the CPU level. Columnar storage naturally aligns vector data for SIMD processing. Instead of jumping around memory gathering vector components, the CPU finds them laid out sequentially, ready for bulk operations. Modern processors can load 16 floating-point values into a single register and process them simultaneously.

Compression becomes almost trivial, too. BigQuery's Capacitor format applies techniques like Product Quantization directly to the column data, shrinking vectors from 3KB to under 300 bytes. Try doing that with row-oriented storage where vectors are scattered across pages.

The lesson? Sometimes the "wrong" abstraction at one level enables the right optimizations at another.

Accelerating Computations with SIMD

SIMD instructions are a form of hardware-level parallelism available in modern CPUs that provide significant speedups for vector arithmetic. This is achieved through special instruction sets built into the processor.

For example, AVX-512 (Advanced Vector Extensions 512-bit) is an instruction set found in modern high-performance CPUs, such as those from Intel, that allows a single instruction to operate on 512 bits of data at once.

Since a standard single-precision floating-point number is 32 bits, a CPU with AVX-512 can process 16 floating-point numbers in a single operation. This leads to dramatic performance gains.

The difference between scalar and SIMD processing for vector distance calculations is stark:

• Scalar approach: Loop through each dimension, multiply corresponding components, accumulate results. For 768 dimensions, that's 768 multiplications, 768 additions, and terrible cache performance as you jump between two different memory locations for each iteration.

• SIMD approach: Load 16 components from each vector into 512-bit registers. Execute a single multiply instruction that handles all 16 pairs. Execute a single horizontal add. Repeat 48 times. The CPU's pipeline stays full, the cache prefetcher knows exactly what data you need next, and you've turned 1,536 operations into 96.

The columnar storage pays off here, too. Vectors stored contiguously in memory align perfectly with SIMD register loads. No gather operations, no wasted cycles. Just pure throughput.

BigQuery's query engine is designed to leverage SIMD extensively. It automatically detects and uses the optimal instruction set available on the underlying hardware (for example, AVX-512 for Intel, NEON for ARM). The columnar storage format ensures that vector data is laid out in memory in a way that is friendly to SIMD registers, and the engine processes query vectors in large batches to maximize the utilization of these parallel instructions.

The TreeAH Indexing Algorithm

While brute-force search can be effective at smaller scales due to BigQuery's massive parallelism, efficient search over billions of vectors requires an index. BigQuery's primary vector index is TreeAH (Tree with Asymmetric Hashing), which is based on Google's open-sourced ScaNN (Scalable Nearest Neighbors) algorithm. TreeAH combines three techniques to achieve high performance and memory efficiency.

1. Hierarchical Tree Structure

The algorithm first partitions the entire vector space into thousands of smaller lists. You can think of this like organizing a massive library. Instead of having one giant room with a million books, a library has floors, sections, and shelves. This hierarchy allows you to find a book without scanning every single one.

Similarly, TreeAH groups semantically similar vectors together into partitions and arranges them in a tree. During a query, the search navigates this tree by comparing the query vector to "centroid" vectors that represent the center of each partition, effectively following a path to the most relevant partitions and pruning away large, irrelevant branches of the search space.

2. Product Quantization (PQ)

Within TreeAH, PQ serves a different purpose than just compression. The index doesn't just store smaller vectors – it fundamentally changes how distance calculations work.

TreeAH learns partition-specific codebooks that capture the local structure of vectors in each tree node. This means vectors that end up in the "shoes" partition get quantized differently than those in "electronics." The compression becomes semantic-aware.

When combined with the tree structure, this creates a powerful effect: not only are you searching fewer vectors (thanks to the tree), but you're computing distances faster on the vectors you do search (thanks to PQ).

3. Asymmetric Hashing

The "asymmetric" aspect refers to the fact that the query vector is kept in its full-precision form, while the database vectors are compared in their compressed, quantized form.

The vectors are not of different dimensions, but of different precision. The semantic matching works because the comparison is not direct. The compressed database vector is a code that points to a region in the original vector space. The distance calculation uses the full-precision query vector to look up a pre-computed distance to the center of that region. This way, the rich information in the query vector is used to accurately estimate the distance, avoiding the significant information loss that would occur if both vectors were compressed.

Architectural Comparison: TreeAH vs. HNSW

To better understand the design philosophy behind TreeAH, it’s useful to compare it with HNSW (Hierarchical Navigable Small World), a popular graph-based algorithm used in many dedicated vector databases.

HNSW constructs a multi-layered graph where vectors are nodes and edges connect them to their nearest neighbors. It’s known for excellent single-query latency.

But this performance comes with significant memory overhead, as the graph structure must be stored in addition to the full-precision vectors. HNSW index builds can also be time-consuming, and frequent data updates can lead to memory fragmentation and performance degradation.

TreeAH, in contrast, makes different architectural trade-offs that align with BigQuery's nature as a distributed analytics system.

The comparison reveals a fundamental design choice: TreeAH prioritizes batch throughput, memory efficiency, and scalability over absolute single-query latency. This makes it well-suited for analytical workloads where thousands of searches are performed simultaneously.

The End-to-End Vector Search Query Flow

The execution timeline of a BigQuery vector search demonstrates how parallel processing eliminates traditional bottlenecks. When a VECTOR_SEARCH query arrives, the system initiates multiple operations concurrently rather than executing them sequentially.

The root server begins query planning immediately upon receiving the request. In parallel, Borg starts allocating compute slots across the cluster, targeting 1,000 slots distributed across 50 or more nodes. Borg prioritizes slots that are physically close to the data in Colossus to minimize data movement costs. This allocation typically completes within 10 milliseconds.

Query planning and resource allocation overlap significantly. The mixer nodes receive partial execution plans and begin partitioning the search space before Borg completes all slot allocations. When TreeAH indexes are available, mixers use them to assign specific vector partitions to leaf nodes. This streaming approach ensures that leaf nodes receive work assignments as soon as they come online.

The parallel execution phase showcases the architecture's efficiency. Hundreds or thousands of leaf nodes simultaneously read their assigned vector partitions from Colossus. Jupiter's high-bandwidth network prevents I/O congestion even with thousands of concurrent reads. Each leaf node operates independently: loading compressed vectors, executing SIMD operations for distance calculations, and maintaining local top-k results.

Aggregation begins before all leaf nodes complete their local searches. Mixers implement a streaming merge algorithm that processes results as they arrive. This approach means that by the time the slowest leaf node reports its results, the mixers have already processed most of the data. The final global top-k emerges from this continuous merging process.

The measured 40-millisecond execution time represents the longest path through the parallel execution graph, not the sum of individual operations. Most operations complete much faster, but the overall latency is bounded by the slowest component. This design trades single-query latency for massive throughput, enabling BigQuery to process thousands of vector searches concurrently across billions of vectors.

Practical Implications for Engineering Teams

The architectural choices behind BigQuery's vector search create specific trade-offs that engineering teams need to understand before committing to this approach.

1. Query Latency vs. Throughput

BigQuery vector searches typically complete in 1-10 seconds, not the sub-100ms latency of specialized vector databases. But you can run thousands of searches concurrently without degradation. This makes BigQuery ideal for batch recommendation generation, similarity analysis across product catalogs, or embedding-based data enrichment pipelines. It's the wrong choice for autocomplete features or real-time personalization that requires immediate responses.

2. Cost Model Considerations

BigQuery charges for data scanned, not query execution time. A vector search that scans 1TB costs the same whether it completes in 2 seconds or 20 seconds. This model favors workloads where you search large datasets infrequently rather than small datasets continuously. Running vector search on a 10GB table thousands of times per day will be more expensive than a dedicated vector database with fixed infrastructure costs.

3. Index Management Trade-offs

TreeAH indexes update automatically in the background when new data arrives, typically within 5-15 minutes. You cannot force immediate index updates or control index parameters like you can with HNSW or IVF indexes. This simplicity reduces operational overhead but limits optimization options. If your use case requires fine-tuning recall/latency trade-offs or immediate consistency after updates, you'll need a different solution.

4. Integration Benefits That Actually Matter

The ability to JOIN vector search results with business data in a single query is more powerful than it initially appears. Consider this query pattern:

WITH semantic_matches AS (

  SELECT item_id, distance

  FROM VECTOR_SEARCH(

    TABLE products,

    'embedding',

    (SELECT embedding FROM queries WHERE query_id = @query_id)

  )

)

SELECT p.*, s.distance

FROM semantic_matches s

JOIN products p USING (item_id)

WHERE p.in_stock = TRUE

  AND p.price BETWEEN 50 AND 200

  AND p.category_restrictions IS NULL

ORDER BY s.distance

LIMIT 20

This combines semantic search with business logic, inventory status, and access controls in one atomic operation. Implementing this with a separate vector database requires complex synchronization between systems.

Conclusion

BigQuery's vector search implementation challenges our assumptions about what a data warehouse can do. Instead of building another specialized vector database, Google pushed their existing infrastructure to handle a fundamentally different workload.

The key insight is recognizing that vector search at scale is a data processing problem. And processing data at scale is what BigQuery was built for.

By leveraging its columnar architecture and hardware-aware algorithms like TreeAH, BigQuery makes a deliberate trade-off. It exchanges the sub-millisecond latency of in-memory systems for massive batch throughput and incredible resource efficiency. An index that uses 10x less memory than HNSW is a trade-off many teams building analytical AI systems would gladly make.

The real power emerges when vectors live alongside business data. Complex queries that would require multiple systems and synchronization nightmares become simple SQL. "Find similar products, but only from reliable suppliers, in stock locally, with no recent quality issues." One query, one system, no architectural gymnastics.

This approach validates a broader trend: vector capabilities are becoming table stakes for data platforms. The question isn't whether your data platform will support vectors, but how well it integrates them into existing workflows.

For teams building analytical AI applications, BigQuery offers a pragmatic path. It won't win latency benchmarks against dedicated vector databases. But for batch processing, integrated analytics, and operational simplicity at scale, it demonstrates that sometimes the best vector database isn't a vector database at all. It's your data warehouse, evolved.

Further Reading

• BigQuery Under the Hood: Official architecture deep dive

• ScaNN Algorithm Details: The mathematics behind TreeAH

• Dremel: Interactive Analysis of Web-Scale Datasets: The foundational paper

• Large-scale cluster management at Google with Borg: Understanding resource orchestration

• Jupiter Rising: A Decade of Clos Topologies: Google's datacenter networking

• BigQuery Vector Search: A Practitioner's Guide: Optimization strategies

Naturally, I had to prepare my source code as per instruction. In this article, I will share how I packaged and submitted my Add-on's source code in Ubuntu OS.

But before that, let's talk a little about Tier 2 CASA assessment.

What is Tier 2 CASA Security Assessment?

The Tier 2 CASA (Cloud Application Security Assessment) is a self-service security assessment process for applicants seeking access to Google Workspace data or to comply with specific Google Workspace policies.

It allows developers to scan their applications and submit the results for verification without an external assessor accessing the code or infrastructure.

Importance Tier 2 CASA security assessment

Tier 2 CASA is important for several reasons:

• Security Assurance: It provides independent verification of your application's security posture, reducing the risk of data breaches and protecting user privacy.
• Compliance: It helps meet security requirements for accessing Google Workspace data or adhering to Google policies, like the Workspace Marketplace Terms of Service.
• Efficiency: It's a faster and more cost-effective alternative to Tier 1 assessments, which involves external assessors directly examining your application.
• Trust: If your addon is published without verification it'll display an "unverified" message to the clients while installing the addon, which creates distrust and can lead to the installation process of your addon to be abandoned.

In the context of my Google Workspace Addon Scan Me, the use of restrictive OAuth scope auth/drive of Google Drive API likely triggered the need for a Tier 2 assessment. This scope grants your addon access to see, edit, create, and delete all of your Google Drive files, which falls under Google's security and privacy requirements.

Additional Resources

• CASA Tier 2 Overview
• CASA Documentation
• Google Workspace Marketplace Terms of Service

Disclaimer: While I'll explain the Tier 2 CASA process, it's crucial to consult the official documentation and Google's security guidelines for specific requirements and guidance.

The assessment certification is free, by the way. To prepare your addon for the CASA assessment process follow the following steps.

Step 1 – Sign up for the new Assessment Procedure

If you're using restrictive scopes, you'll receive an email from Google's Verification team at some point requesting to verify the scopes after you've submitted your add-on for verification.

This email is the notification document. So, you need to download this email as a PDF, which must be submitted in the application form later on.

In that email, you'll find the following instructions for Tier 2 evaluation. You'll find a link to register or log-in to the CASA portal. Click the link and register to the site. Then click on Start New Assessment> Create New Assessment.

Fill in the information asked carefully. Upload the previously downloaded email where you're asked for a Tier 2 notification pdf.

Starting New CASA Assesment of the Addon

Note: For Google Workspace Addon, the type of application is Local App.

Caution: As shown in the image above, even though "Project ID" is asked in the input field, they are asking for the Project Number included in the email, not the Project ID of your Google Cloud Console project.

After you carefully fill in the details and submit the form, you'll arrive at a new screen – Application Screening – where there are two things that you should download:

Download Scan Cenral Package and Setup Insruction

1. Fortify Scan Central Package.
2. Instruction on compressing your application's source code for initial assessment.

Step 2 – Download and Setup Java JDK

To use the Scan Central package as mentioned in the instructions, a minimum of JDK 11 is required.

For setting up the path for the Java environment in Linux, I followed this instruction on StackOverflow.

Step 3 – Setup Path for Scan Central

Now let's add the path to the Scan Central in our system.

In your CLI, open .bashrc file with sudo nano ~/.bashrc. Append the following path at the end of the file:

# SCAN Central 
# Path looks like following
#/home/<username>/Fortify_ScanCentral_Client_22.2.1_x64/bin

 export PATH=$PATH:<Path To bin folder in Scan Central>

Save (CTRL+S) and exit (CTRL + X) the file.

Open .profile with sudo nano ~/.profile and add the same path as above. You can check the version of Scan Central in your CLI with the command scancentral -version, to make sure the setup was successful.

Step 4 – Packaging Source Code for Assessment

To package the source code for your Google Workspace Addon, go to the root directory of your project. If you're following the instruction manual, go to the section for JavaScript code packaging.

In the root directory run any of the following commands:

#cmd 1 
scancentral package -bt none -o myPackage.zip
# or cmd 2
scancentral package -bt none --scan-node-modules -o myPackage.zip

Note: The command scancentral.bat is for Windows users.

As mentioned in the instruction, command 2 increases the size of the package and is not necessary for Node.js or Angular. I created Workspace Addon so I don't have node-modules in my source code.

After that, you'll see a compressed package named myPackage in the directory where you ran the packaging operation.

Step 5 – Initiate the Scan Process

After packaging, go back to the CASA portal and click on your assessment ID link in the list, and open up the Application Screening window. Here:

1. Click the Begin Scan Process button.
2. Upload the package you just compressed.
3. Click the Upload File & Initiate Scan button.

Upload Source Code To Fortify Scan

This will initiate auto scanning of your application which is the beginning of assessment for your Addon.

Reminder: As I've personally experienced, if your source code uses the Math.random() method, then the auto-scanner will not pass your code.

If you pass this phase, the manual verification process will begin where you'll have to fill in forms for the survey. Go to this link for the questions that'll be asked in the CASA survey. Here, choose the Local App option for App Type for a Google Workspace Add-on. I want to remind you that they will change based on the answer provided.

Conclusion

Alright, I'm hoping this blog helped you reduce the time and confusion that I had to encounter when I was trying to assess my addon. And please don't give up midway during the evaluation otherwise your months of hard work will be in vain.

My addon Scan Me, scans the Google Drive and prepares an audit report in a spreadsheet file of your choosing in your Google Drive. It makes it extremely easy for you to analyze your Google Drive from one place, and it also offers a free quota. If you're looking for a similar addon I hope you'll try this addon.

This is Nibesh khadka, have a good day.

What is the Google Cloud Associate Cloud Engineer?

The Associate Cloud Engineer also commonly referred to as the ACE is the associate level certification by Google Cloud.

They key difference between the Associate Cloud Engineer and the Google Cloud Digital Cloud Leader is that the Associate Cloud Engineer is a hands-on certification requiring you to have hands-on experience with the Google Cloud's core services.

Google Cloud only has a single Associate level certification. This is different compared to AWS which has three, and Azure which has more than six associate level certifications.

Having only a single associate level certification means that you are expected to know more information the compared to say the AWS Solutions Architect Associate.

However the Associate Cloud Engineer better reflects both in role-based title and knowledge closer to entry level jobs into the cloud industry.

What is a Cloud Engineer?

A Cloud Engineer is the most popular a IT/DevOps role within the cloud industry. A Cloud Engineer is an entry level role where you tasked with designing, implement and maintain cloud infrastructure.

This cloud role lets you perform a bit of everything can and from there you can future your cloud career by specializing in an advanced role such as Solutions Architect, DevOps Engineer, Data Engineer and many more.

To obtain a cloud role its recommended to hold an associate level certification from a Cloud Servicer Provider (CSPs) like Google Cloud as well as building your cloud project to showcase within your portfolio.

Overview of the Google Cloud Associate Cloud Engineer

The exam guide divides the exam questions into the following five sections.

• Section 1. Setting up a cloud solution environment
• Section 2. Planning and configuring a cloud solution
• Section 3. Deploying and implementing a cloud solution
• Section 4. Ensuring successful operation of a cloud solution
• Section 5. Configuring access and security

How do you get Certified?

Google uses Kryterion as its test center. You can take the exam in-person or online.

There are 50 multiple-choice and multiple-select questions and you have to score 70% to pass

You get 2 hours to complete this course.

The Google Cloud Associate Cloud Engineer is $125 USD.

Can I simply watch the videos and pass the exam?

Is it recommended that you complete the labs within your Google Cloud account to ensure you completely understand each concept. Cloud Service Providers often change their UI and the experience can vary based on the region you wish to launch services.

Students that perform labs within their own accounts are more likely to be able to recall information.

Practice exams are strongly recommended after your study course. ExamPro has a full free practice to help you study for the final exam.

Head on over to freeCodeCamp's YouTube channel to start working through the full 6.0 hour course.

Gone are the days of storing your data in a CSV file or an Excel spreadsheet. If you want to quickly analyze millions of data rows in seconds, BigQuery is the way to go.

In this getting started guide, we'll learn about BigQuery and how we can use it to query and analyze data.

What is BigQuery?

BigQuery is an enterprise data warehouse that many companies use who need a fully-managed cloud based solution for their massive datasets.

BigQuery's serverless architecture allows you to quickly execute standard SQL queries and analyze millions of data rows in seconds. You can then store your data both in Google Cloud Storage in files and buckets or in BigQuery storage.

BigQuery also has excellent integrations with other GCP products, like Data Flow and Data Studio that makes it a great choice for data analytics tasks.

Before You Begin:

We are going to query tables in a public dataset that Google has provided to try out BigQuery using the Google Cloud Platform. Therefore, this guide assumes that:

• You have an access on Google Cloud Platform.
• You have already created a Google Cloud project.
• Google sandbox environment is up and running.

How to Access a Public Dataset

A public dataset is available to the general public through the Google Cloud Public Dataset Program. We'll use a Hacker News dataset that contains all stories and comments from Hacker News from its launch in 2006 to present. Let's get started.

Navigate to Hacker News dataset and click the VIEW DATASET button. It will take you to the Google Cloud Platform login screen. Login to the account and it will open the BigQuery Editor window with the dataset.

How the BigQuery Interface Is Organized

BigQuery is structured as a hierarchy with 4 levels:

• Projects: Top-level containers that store the data
• Datasets: Within projects, datasets allow you to organize your data and hold one or more tables of data
• Tables: Within datasets, tables hold actual data.
• Jobs: task performed on data such as running queries, loading data, and exporting data.

Note: Please note that while working with tables, you'll also notice that:

• Tables are broken out by day meaning that you will need to use a wildcard, or * to pull a larger date range.
• There is also an “intraday” table that will give you data for the last 24 hours.

How to Check the Table Schema

Click on the table name. This will allow you to see what columns are in the table, as well as some buttons to perform various operations on the table.

How to Preview the Data

Use the preview button to get a sample of some rows in the table. Don’t do a SELECT * in BigQuery:

How to Query Big Data

SQL statements are used to perform various database tasks, such as querying data, creating tables, and updating databases.

Basic Queries

Basic queries contain the following components:

• SELECT (required): identifies the columns to be included in the query
• FROM (required): the table that contains the columns in the SELECT statement
• WHERE: a condition for filtering records
• ORDER BY: Used to sort the result-set in ascending or descending order.
• GROUP BY: how to aggregate data in the result set

How to Compose a Query in BigQuery

For our first query, let’s find out what are the top 5 domains shared in Hacker News in year 2021 so far (query executed on July 9th 2021).

Click the Compose New query button. It will open the editor tab.

Write your first query as below:

SELECT REGEXP_EXTRACT(url, '//([^/]*)/?') domain, COUNT(*) total
FROM `bigquery-public-data.hacker_news.full`
WHERE url!='' AND EXTRACT(YEAR FROM timestamp)=2021
GROUP BY domain ORDER BY total DESC LIMIT 5

You'll notice that BigQuery debugs your code as you construct it. If the query is valid, then a check mark appears along with the amount of data that the query will process. This helps you determine the cost of running the query.

If the query is invalid, then an exclamation point appears along with an error message.

To run this query, click on the Run button. In a few seconds, you should see results returned from the query:

You can click on the JSON tab if you want the results in JSON format. You'll also find interesting details under the 'Execution details' column.

How to Query Multiple Tables Using a Wildcard Table

Wildcard tables enable you to query multiple tables using concise SQL statements. A wildcard table represents a union of all the tables that match the wildcard expression:

FROMtablename.stories_*``

_TABLE_SUFFIX Pseudo Column

Queries with wildcard tables support the _TABLE_SUFFIX pseudo column in the WHERE clause. To restrict a query so that it scans only a specified set of tables, use the _TABLE_SUFFIX pseudo column in a WHERE clause with a condition that is a constant expression.

Using _TABLE_SUFFIX can greatly reduce the number of bytes scanned, which helps reduce the cost of running your queries.

How to Get Data by Providing a Date Range

WHERE _TABLE_SUFFIX BETWEEN
    FORMAT_DATE(‘%Y%m%d’,DATE_SUB(CURRENT_DATE(), INTERVAL 36 MONTH))
    AND
    FORMAT_DATE(‘%Y%m%d’,DATE_SUB(CURRENT_DATE(), INTERVAL 1 DAY))

How to Use UNNEST to Flatten the Date

To convert an ARRAY into a set of rows, also known as "flattening," use the UNNEST operator. UNNEST takes an ARRAY and returns a table with a single row for each element in the ARRAY:

SELECT * FROM UNNEST (['Ambreen', 'Abdul', 'Adam', 'David']) AS names;

How to Save and Share Queries

You can save your queries for later use. There are 3 types of saved queries:

• Private: Private saved queries are visible only to the user who creates them.
• Project-level: Project-level saved queries are visible to members of the predefined BigQuery IAM roles with the required permissions.
• Public: Public saved queries are visible to anyone with a link to the query.

Summary

BigQuery is much more sophisticated than what we explored in this simple tutorial. You can also export Firebase Analytics data to BigQuery, which will let you run sophisticated ad hoc queries against your analytics data.

And with BigQuery ML, you can create and execute machine learning models using standard SQL queries.

If you’re feeling excited and want to learn more about BigQuery, check out the links below.

Resources:

• BigQuery cookbook
• Filtering selected tables using _TABLE_SUFFIX
• BigQuery Tip: The UNNEST Function
• BigQuery UNNEST: How to work with nested data in BigQuery

Do you have the knowledge and skills to design a mobile gaming analytics platform that collects, stores, and analyzes large amounts of bulk and real-time data?

Well, after reading this article, you will.

I aim to take you from zero to hero in Google Cloud Platform (GCP) in just one article. I will show you how to:

• Get started with a GCP account for free
• Reduce costs in your GCP infrastructure
• Organize your resources
• Automate the creation and configuration of your resources
• Manage operations: logging, monitoring, tracing, and so on.
• Store your data
• Deploy your applications and services
• Create networks in GCP and connect them with your on-premise networks
• Work with Big Data, AI, and Machine Learning
• Secure your resources

Once I have explained all the topics in this list, I will share with you a solution to the system I described.

If you do not understand some parts of it, you can go back to the relevant sections. And if that is not enough, visit the links to the documentation that I have provided.

Are you up for a challenge? I have selected a few questions from old GCP Professional Certification exams. They will test your understanding of the concepts explained in this article.

I recommend trying to solve both the design and the questions on your own, going back to the guide if necessary. Once you have an answer, compare it to the proposed solution.

Try to go beyond what you are reading and ask yourself what would happen if requirement X changed:

• Batch vs streaming data
• Regional vs global solution
• A small number of users vs huge volume of users
• Latency is not a problem vs real-time applications

And any other scenarios you can think of.

At the end of the day, you are not paid just for what you know but for your thought process and the decisions you make. That is why it is vitally important that you exercise this skill.

At the end of the article, I'll provide more resources and next steps if you want to continue learning about GCP.

How to get started with Google Cloud Platform for free

GCP currently offers a 3 month free trial with $300 US dollars of free credit. You can use it to get started, play around with GCP, and run experiments to decide if it is the right option for you.

You will NOT be charged at the end of your trial. You will be notified and your services will stop running unless you decide to upgrade your plan.

I strongly recommend using this trial to practice. To learn, you have to try things on your own, face problems, break things, and fix them. It doesn't matter how good this guide is (or the official documentation for that matter) if you do not try things out.

Why would you migrate your services to Google Cloud Platform?

Consuming resources from GCP, like storage or computing power, provides the following benefits:

• No need to spend a lot of money upfront for hardware
• No need to upgrade your hardware and migrate your data and services every few years
• Ability to scale to adjust to the demand, paying only for the resources you consume
• Create proof of concepts quickly since provisioning resources can be done very fast
• Secure and manage your APIs
• Not just infrastructure: data analytics and machine learning services are available in GCP

GCP makes it easy to experiment and use the resources you need in an economical way.

How to optimize your VMs to reduce costs in GCP

In general, you will only be charged for the time your instances are running. Google will not charge you for stopped instances. However, if they consume resources, like disks or reserved IPs, you might incur charges.

Here are some ways you can optimize the cost of running your applications in GCP.

Custom Machine Types

GCP provides different machine families with predefined amounts of RAM and CPUs:

• General-purpose. Offers the best price-performance ratio for a variety of workloads.
• Memory-optimized. Ideal for memory-intensive workloads. They offer more memory per core than other machine types.
• Compute-optimized. They offer the highest performance per core and and are optimized for compute-intensive workloads
• Shared-core. These machine types timeshare a physical core. This can be a cost-effective method for running small applications.

Besides, you can create your custom machine with the amount of RAM and CPUs you need.

Preemptible VM's

You can use preemptible virtual machines to save up to 80% of your costs. They are ideal for fault-tolerant, non-critical applications. You can save the progress of your job in a persistent disk using a shut-down script to continue where you left off.

Google may stop your instances at any time (with a 30-second warning) and will always stop them after 24 hours.

To reduce the chances of getting your VMs shut down, Google recommends:

• Using many small instances and
• Running your jobs during off-peak times.

Note: Start-up and shut-down scripts apply to non-preemptible VMS as well. You can use them the control the behavior of your machine when it starts or stops. For instance, to install software, download data, or backup logs.

There are two options to define these scripts:

• When you are creating your instance in the Google Console, there is a field to paste your code.
• Using the metadata server URL to point your instance to a script stored in Google Cloud Storage.

This latter is preferred because it is easier to create many instances and to manage the script.

Sustained Use Discounts

The longer you use your virtual machines (and Cloud SQL instances), the higher the discount - up to 30%. Google does this automatically for you.

Committed Use Discounts

You can get up to 57% discount if you commit to a certain amount of CPU and RAM resources for a period of 1 to 3 years.

To estimate your costs, use the Price Calculator. This helps prevent any surprises with your bills and create budget alerts.

How to manage resources in GCP

In this section, I will explain how you can manage and administer your Google Cloud resources.

Resource Hierarchy

There are four types of resources that can be managed through Resource Manager:

• The organization resource. It is the root node in the resource hierarchy. It represents an organization, for example, a company.
• The projects resource. For example, to separate projects for production and development environments. They are required to create resources.
• The folder resource. They provide an extra level of project isolation. For example, creating a folder for each department in a company.
• Resources. Virtual machines, database instances, load balancers, and so on.

There are quotas that limit the maximum number of resources you can create to prevent unexpected spikes in billing. However, most quotas can be increased by opening a support ticket.

Resources in GCP follow a hierarchy via a parent/child relationship, similar to a traditional file system, where:

• Permissions are inherited as we descend the hierarchy. For example, permissions granted and the organization level will be propagated to all the folders and projects.
• More permissive parent policies always overrule more restrictive child policies.

This hierarchical organization helps you manage common aspects of your resources, such as access control and configuration settings.

You can create super admin accounts that have access to every resource in your organization. Since they are very powerful, make sure you follow Google's best practices.

Labels

Labels are key-value pairs you can use to organize your resources in GCP. Once you attach a label to a resource (for instance, to a virtual machine), you can filter based on that label. This is useful also to break down your bills by labels.

Some common use cases:

• Environments: prod, test, and so on.
• Team or product owners
• Components: backend, frontend, and so on.
• Resource state: active, archive, and so on.

Labels vs Network tags

These two similar concepts seem to generate some confusion. I have summarized the differences in this table:

Cloud IAM

Simply put, Cloud IAM controls who can do what on which resource. A resource can be a virtual machine, a database instance, a user, and so on.

It is important to notice that permissions are not directly assigned to users. Instead, they are bundled into roles, which are assigned to members. A policy is a collection of one or more bindings of a set of members to a role.

Identities

In a GCP project, identities are represented by Google accounts, created outside of GCP, and defined by an email address (not necessarily @gmail.com). There are different types:

• Google accounts*. To represent people: engineers, administrators, and so on.
• Service accounts. Used to identify non-human users: applications, services, virtual machines, and others. The authentication process is defined by account keys, which can be managed by Google or by users (only for user-created service accounts).
• Google Groups are a collection of Google and service accounts.
• G Suite Domain* is the type of account you can use to identify organizations. If your organization is already using Active Directory, it can be synchronized with Cloud IAM using Cloud Identity.
• allAuthenticatedUsers. To represent any authenticated user in GCP.
• allUsers. To represent anyone, authenticated or not.

Regarding service accounts, some of Google's best practices include:

• Not using the default service account
• Applying the Principle of Least Privilege. For instance:
• Restrict who can act as a service account
• Grant only the minimum set of permissions that the account needs
• Create service accounts for each service, only with the permissions the account needs

Roles

A role is a collection of permissions. There are three types of roles:

• Primitive. Original GCP roles that apply to the entire project. There are three concentric roles: Viewer, Editor, and Owner. Editor contains Viewer and Owner contains Editor.
• Predefined. Provides access to specific services, for example, storage.admin.
• Custom. lets you create your own roles, combining the specific permissions you need.

When assigning roles, follow the principle of least privilege, too. In general, prefer predefined over primitive roles.

Cloud Deployment Manager

Cloud Deployment Manager automates repeatable tasks like provisioning, configuration, and deployments for any number of machines.

It is Google's Infrastructure as Code service, similar to Terraform - although you can deploy only GCP resources. It is used by GCP Marketplace to create pre-configured deployments.

You define your configuration in YAML files, listing the resources (created through API calls) you want to create and their properties. Resources are defined by their name (VM-1, disk-1), type (compute.v1.disk, compute.v1.instance) and properties (zone:europe-west4, boot:false).

To increase performance, resources are deployed in parallel. Therefore you need to specify any dependencies using references. For instance, do not create virtual machine VM-1 until the persistent disk disk-1 has been created. In contrast, Terraform would figure out the dependencies on its own.

You can modularize your configuration files using templates so that they can be independently updated and shared. Templates can be defined in Python or Jinja2. The contents of your templates will be inlined in the configuration file that references them.

Deployment Manager will create a manifest containing your original configuration, any templates you have imported, and the expanded list of all the resources you want to create.

Cloud Operations (formerly Stackdriver)

Operations provide a set of tools for monitoring, logging, debugging, error reporting, profiling, and tracing of resources in GCP (AWS and even on-premise).

Cloud Logging

Cloud Logging is GCP's centralized solution for real-time log management. For each of your projects, it allows you to store, search, analyze, monitor, and alert on logging data:

• By default, data will be stored for a certain period of time. The retention period varies depending on the type of log. You cannot retrieve logs after they have passed this retention period.
• Logs can be exported for different purposes. To do this, you create a sink, which is composed of a filter (to select what you want to log) and a destination: Google Cloud Storage (GCS) for long term retention, BigQuery for analytics, or Pub/Sub to stream it into other applications.
• You can create log-based metrics in Cloud Monitoring and even get alerted when something goes wrong.

Logs are a named collection of log entries. Log entries record status or events and includes the name of its log, for example, compute.googleapis.com/activity. There are two main types of logs:

First, User Logs:

• These are generated by your applications and services.
• They are written to Cloud Logging using the Cloud Logging API, client libraries, or logging agents installed on your virtual machines.
• They stream logs from common third-party applications like MySQL, MongoDB, or Tomcat.

Second, Security logs, divided into:

• Audit logs, for administrative changes, system events, and data access to your resources. For example, who created a particular database instance or to log a live migration. Data access logs must be explicitly enabled and may incur additional charges. The rest are enabled by default, cannot be disabled, and are free of charges.
• Access Transparency logs, for actions taken by Google staff when they access your resources for example to investigate an issue you reported to the support team.

VPC Flow Logs

They are specific to VPC networks (which I will introduce later). VPC flow logs record a sample of network flows sent from and received by VM instances, which can be later access in Cloud Logging.

They can be used to monitor network performance, usage, forensics, real-time security analysis, and expense optimization.

Note: you may want to log your billing data for analysis. In this case, you do not create a sink. You can directly export your reports to BigQuery.

Cloud Monitoring

Cloud Monitoring lets you monitor the performance of your applications and infrastructure, visualize it in dashboards, create uptime checks to detect resources that are down and alert you based on these checks so that you can fix problems in your environment. You can monitor resources in GCP, AWS, and even on-premise.

It is recommended to create a separate project for Cloud Monitoring since it can keep track of resources across multiple projects.

Also, it is recommended to install a monitoring agent in your virtual machines to send application metrics (including many third-party applications) to Cloud Monitoring. Otherwise, Cloud Monitoring will only display CPU, disk traffic, network traffic, and uptime metrics.

Alerts

To receive alerts, you must declare an alerting policy. An alerting policy defines the conditions under which a service is considered unhealthy. When the conditions are met, a new incident will be created and notifications will be sent (via email, Slack, SMS, PagerDuty, etc).

A policy belongs to an individual workspace, which can contain a maximum of 500 policies.

Trace

Trace helps find bottlenecks in your services. You can use this service to figure out how long it takes to handle a request, which microservice takes the longest to respond, where to focus to reduce the overall latency, and so on.

It is enabled by default for applications running on Google App Engine (GAE) - Standard environment - but can be used for applications running on GCE, GKE, and Google App Engine Flexible.

Error Reporting

Error Reporting will aggregate and display errors produced in services written in Go, Java, Node.js, PHP, Python, Ruby, or .NET. running on GCE, GKE, GAP, Cloud Functions, or Cloud Run.

Debug

Debug lets you inspect the application's state without stopping your service. Currently supported for Java, Go, Node.js and Python. It is automatically integrated with GAE but can be used on GCE, GKE, and Cloud Run.

Profile

Profiler that continuously gathers CPU usage and memory-allocation information from your applications. To use it, you need to install a profiling agent.

How to store data in GCP

In this section, I will cover both Google Cloud Storage (for any type of data, including files, images, video, and so on), the different database services available in GCP, and how to decide which storage option works best for you.

Google Cloud Storage (GCS)

GCS is Google's storage service for unstructured data: pictures, videos, files, scripts, database backups, and so on.

Objects are placed in buckets, from which they inherit permissions and storage classes.

Storage classes provide different SLAs for storing your data to minimize costs for your use case. A bucket's storage class can be changed (under some restrictions), but it will affect new objects added to the bucket only.

In addition to Google's console, you can interact with GCS from your command line, using gsutil. You can use specify:

• Multithreaded updates when you need to upload a large number of small files. The command looks like gsutil -m cp files gs://my-bucket)
• Parallel updates when you need to upload large files. For more details and restrictions, visit this link.

Another option to upload files to GCS is Storage Transfer Service (STS), a service that imports data to a GCS bucket from:

• An AWS S3 bucket
• A resource that can be accessed through HTTP(S)
• Another Google Cloud Storage bucket

If you need to upload huge amounts of data (from hundreds of terabytes up to one petabyte) consider Data Transfer Appliance: ship your data to a Google facility. Once they have uploaded the data to GCS, the process of data rehydration reconstitutes the files so that they can be accessed again.

Object lifecycle management

You can define rules that determine what will happen to an object (will it be archived or deleted) when a certain condition is met.

For example, you could define a policy to automatically change the storage class of an object from Standard to Nearline after 30 days and to delete it after 180 days.

This is the way a rule can be defined:

{
   "lifecycle":{
      "rule":[
         {
            "action":{
               "type":"Delete"
            },
            "condition":{
               "age":30,
               "isLive":true
            }
         },
         {
            "action":{
               "type":"Delete"
            },
            "condition":{
               "numNewerVersions":2
            }
         },
         {
            "action":{
               "type":"Delete"
            },
            "condition":{
               "age":180,
               "isLive":false
            }
         }
      ]
   }
}

It will be applied through gsutils or a REST API call. Rules can be created also through the Google Console.

Permissions in GCS

In addition to IAM roles, you can use Access Control Lists (ACLs) to manage access to the resources in a bucket.

Use IAM roles when possible, but remember that ACLs grant access to buckets and individual objects, while IAM roles are project or bucket wide permissions. Both methods work in tandem.

To grant temporary access to users outside of GCP, use Signed URLs.

Bucket lock

Bucket locks allow you to enforce a minimum retention period for objects in a bucket. You may need this for auditing or legal reasons.

Once a bucket is locked, it cannot be unlocked. To remove, you need to first remove all objects in the bucket, which you can only do after they all have reached the retention period specified by the retention policy. Only then, you can delete the bucket.

You can include the retention policy when you are creating the bucket or add a retention policy to an existing bucket (it retroactively applies to existing objects in the bucket too).

Fun fact: the maximum retention period is 100 years.

Relational Managed Databases in GCP

Cloud SQL and Cloud Spanner are two managed database services available in GCP. If you do not want to deal with all the work necessary to maintain a database online, they are a great option. You can always spin a virtual machine and manage your own database.

Cloud SQL

Cloud SQL provides access to a managed MySQL or PostgreSQL database instance in GCP. Each instance is limited to a single region and has a maximum capacity of 30 TB.

Google will take care of the installation, backups, scaling, monitoring, failover, and read replicas. For availability reasons, replicas must be defined in the same region but a different zone from the primary instances.

Data can be easily imported (first uploading the data to Google Cloud Storage and then to the instance) and exported using SQL dumps or CSV files format. Data can be compressed to reduce costs (you can directly import .gz files). For "lift and shift" migrations, this is a great option.

If you need global availability or more capacity, consider using Cloud Spanner.

Cloud Spanner

Cloud Spanner is globally available and can scale (horizontally) very well.

These two features make it capable of supporting different use cases than Cloud SQL and more expensive too. Cloud Spanner is not an option for lift and shift migrations.

NoSQL Managed Databases in GCP

Similarly, GCP provides two managed NoSQL databases, Bigtable and Datastore, as well as an in-memory database service, Memorystore.

Datastore

Datastore is a completely no-ops, highly-scalable document database ideal for web and mobile applications: game states, product catalogs, real-time inventory, and so on. It's great for:

• User profiles - mobile apps
• Game save states

By default, Datastore has a built-in index that improves performance on simple queries. You can create your own indices, called composite indexes, defined in YAML format.

If you need extreme throughput (huge number of reads/writes per second), use Bigtable instead.

Bigtable

Bigtable is a NoSQL database ideal for analytical workloads where you can expect a very high volume of writes, reads in the milliseconds, and the ability to store terabytes to petabytes of information. It's great for:

• Financial analysis
• IoT data
• Marketing data

Bigtable requires the creation and configuration of your nodes (as opposed to the fully-managed Datastore or BigQuery). You can add or remove nodes to your cluster with zero downtime. The simplest way to interact with Bigtable is the command-line tool cbt.

Bigtable's performance will depend on the design of your database schema.

• You can only define one key per row and must keep all the information associated with an entity in the same row. Think of it as a hash table.
• Tables are sparse: if there is no information associated with a column, no space is required.
• To make reads more efficient, try to store related entities in adjacent rows.

Since this topic is worth an article on its own, I recommend you read the documentation.

Memorystore

It provides a managed version of Redis and Memcache (in-memory databases), resulting in very fast performance. Instances are regional, like Cloud SQL, and have a capacity of up to 300 GB.

How to choose your database

Google loves decision trees. This one will help you choose the right database your your projects. For unstructured data consider GCS or process it using Dataflow (discussed later).

How does networking work in GCP?

Virtual Private Cloud (VPC) - see the docs here

You can use the same network infrastructure that Google uses to run its services: YouTube, Search, Maps, Gmail, Drive, and so on.

Google infrastructure is divided into:

• Regions: Independent geographical areas, at least 100 miles apart from each other, where Google hosts datacenters. It consists of 3 or more zones. For example, us-central1.
• Zones: Multiple individual datacenters within a region. For example, us-central1-a.
• Edge Points of Presence: points of connection between Google's network and the rest of the internet.

GCP infrastructure is designed in a way that all traffic between regions travels through a global private network, resulting in better security and performance.

On top of this infrastructure, you can build networks for your resources, Virtual Private Clouds. They are software-defined networks, where all the traditional network concepts apply:

• Subnets. Logical partitions of a network defined using CIDR notation. They belong to one region only but can span multiple zones. If you have multiple subnets (including your on-premise networks if they are connected to GCP), make sure the CIDR ranges do not overlap.
• IP addresses. Can be internal (for private communication within GCP) or external (to communicate with the rest of the internet). For external IP addresses, you can use an ephemeral IP or pay for a static IP. In general, you need an external IP address to connect to GCP services. However, in some cases, you can configure private access for instances that only have an internal IP.
• Firewalls rules, to allow or deny traffic to your virtual machines, both incoming (ingress) and outgoing (egress). By default, all ingress traffic is denied and all egress traffic is allowed. Firewall rules are defined at the VPC level but they apply to individual instances or groups of instances using network tags or IP ranges.
  Common issue: If you know your VMs are working correctly but you cannot access them through HTTP(s) or cannot SSH into them, have a look at your firewall rules.

You can create hybrid networks connecting your on-premise infrastructure to your VPC.

When you create a project, a default network will be created with subnets in each region (auto mode). You can delete this network, but you need to create at least one network to be able to create virtual machines.

You can also create your custom networks, where no subnets are created by default and you have full control over subnet creation (custom mode).

The main goal of a VPC is the separation of network resources. A GCP project is a way to organize resources and manage permissions.

Users of project A need permissions to access resources in project B. All users can access any VPC defined in any project to which they belong. Within the same VPC, resources in subnet 1 need to be granted access to resources in subnet 2.

In terms of IAM roles, there is a distinction between who can create network resources (Network admin, to create subnets, virtual machines, and so on) and who is responsible for the security of the resources (Security Admin, to create firewall rules, SSL certificates, and so on).

The Compute Instance Admin role combines both roles.

As usual, there are quotas and limits to what you can do in a VPC, amongst them:

• The maximum number of VPCs in a project.
• The maximum number of virtual machines per VPC.
• No broadcast or multicast.
• VPCs cannot use IPv6 to communicate internally, although global load balancers support IPv6 traffic.

How to share resources between multiple VPCs

Shared VPC

Shared VPCs are a way to share resources between different projects within the same organization. This allows you to control billing and manage access to the resources in different projects, following the principle of least privilege. Otherwise you'd have to put all the resources in a single project.

To design a shared VPC, projects fall under three categories:

• Host project. It is the project that hosts the common resources. There can only be one host project.
• Service project: Projects that can access the resources in the host project. A project cannot be both host and service.
• Standalone project. Any project that does not make use of the shared VPC.

You will only be able to communicate between resources created after you define your host and service projects. Any existing resources before this will not be part of the shared VPC.

VPC Network Peering

Shared VPCs can be used when all the projects belong to the same organization. However, if:

• You need private communication across VPCs.
• The VPCs are in projects that may belong to different organizations.
• Want decentralized control, that is, no need to define host projects, server projects, and so on.
• Want to reuse existing resources.

VPC Network peering is the right solution.

In the next section, I will discuss how to connect your VPC(s) with networks outside of GCP.

How to connect on-premise and GCP infrastructures

There are three options to connect your on-premise infrastructure to GCP:

• Cloud VPN
• Cloud Interconnect
• Cloud Peering

Each of them with different capabilities, use cases, and prices that I will describe in the following sections.

Cloud VPN

With Cloud VPN, your traffic travels through the public internet over an encrypted tunnel. Each tunnel has a maximum capacity of 3 Gb per second and you can use a maximum of 8 for better performance. These two characteristics make VPN the cheapest option.

You can define two types of routes between your VPC and your on-premise networks:

• Static routes. You have to manually define and update them, for example when you add a new subnet. This is not the preferred option.
• Dynamic routes. Routes are automatically handled (defined and updated) for you using Cloud Router. This is the preferred option when BGP is available.

Your traffic gets encrypted and decrypted by VPN Gateways (in GCP, they are regional resources).

To have a more robust connection, consider using multiple VPN gateways and tunnels. In case of failure, this redundancy guarantees that traffic will still flow.

Cloud Interconnect

With Cloud VPN, traffic travels through the public internet. With Cloud Interconnect, there is a direct physical connection between your on-premises network and your VPC. This option will be more expensive but will provide the best performance.

There are two types of interconnect available, depending on how you want your connection to GCP to materialize:

• Dedicated interconnect. There is "a direct cable" connecting your infrastructure and GCP. This is the fastest option, with a capacity of 10 to 200 Gb per second. However, it is not available everywhere: at the time of this writing, only in 62 locations in the world.
• Partner interconnect. You connect through a service provider. This option is more geographically available, but the not as fast as a dedicated interconnects: from 50 Mb per second to 10 Gb per second.

Cloud Peering

Cloud peering is not a GCP service, but you can use it to connect your network to Google's network and access services like Youtube, Drive, or GCP services.

A common use case is when you need to connect to Google but don't want to do it over the public internet.

Other networking services

Load Balancers (LB)

In GCP, load balancers are pieces of software that distribute user requests among a group of instances.

A load balancer may have multiple backends associated with it, having rules to decide the appropriate backend for a given request.

There are different types of load balancers. They differ in the type of traffic (HTTP vs TCP/UDP - Layer 7 or Layer 4), whether they handle external or internal traffic, and whether their scope is regional or global:

• HTTP(s). Global LB that handles HTTP(s) requests, distributing traffic to multiple regions based on user location (to the closest region with available instances) or URL maps (the LB can be configured to forward requests to URL/news to a backend service and URL/videos to a different one). It can receive both IPv4 and IPv6 traffic (but this one is terminated at the LB level and proxied as IPv4 to the backends) and has native support for WebSockets.
• SSL Proxy LB. Global LB that handles encrypted TCP traffic, managing SSL certificates for you.
• TCP Proxy LB. Global LB that handles unencrypted TCP traffic. Like SSL Proxy LB, by default, it will not preserve the client's IP, but this can be changed.
• Network Load Balancer. Regional LB that handles TCP/UDP external traffic, based on IP address and port.
• Internal Load Balancer. Like a Network LB, but for internal traffic.

For the visual learners:

Cloud DNS

Cloud DNS is Google's managed Domain Name System (DNS) host, both for internal and external (public) traffic. It will map URLs like https://www.freecodecamp.org/ to an IP address. It is the only service in GCP with 100% SLA - it is available 100% of the time.

Google Cloud CDN

Cloud DNS is Google's Content Delivery Network. If you have data that does not change often (images, videos, CSS, etc.) it makes sense to cache it close to your users. Cloud CDN provides 90 Edges Point of Presence (POP) to cache the data close to your end-users.

After the first request, static data can be stored in a POP, usually much closer to your user than your main servers. Thus, in subsequent requests, you can retrieve the data faster from the POP and reduce the load on your backend servers.

Where can you run your applications in GCP?

I will present 4 places where your code can run in GCP:

• Google Compute Engine
• Google Kubernetes Engine
• App Engine
• Cloud Functions

Note: there is a 5th option: Firebase is Google's mobile platform that helps you quickly develop apps.

Compute Engine (GCE)

Compute engine allows you to spin up virtual machines in GCP. This section will be longer since GCE provides the infrastructure where GKE and GAE run.

In the introduction, I talked about the different types of VMs you can create in GCE. Now, I will cover where to store the data, how to back it up, and how to create instances with all the data and configuration you need.

Where to store your VM's data: disks

Your data can be stored in Persistent disks, Local SSDs, or in Cloud Storage.

Persistent Disk

Persistent disks provide durable and reliable block storage. They are not local to the machine. Rather, they are networked attached, which has its pros and cons:

• Disks can be resized, attached, or detached from a VM even if the instance is in use.
• They have high reliability.
• Disks can survive the instance after its deletion.
• If you need more space, simply attach more disks.
• Larger disks will provide higher performance.
• Being networked attached, they are less performant than local options. SSD persistent disks are also available for more demanding workloads.

Every instance will need one boot disk and it must be of this type.

Local SSD

Local SSDs are attached to a VM to which they provide high-performance ephemeral storage. As of now, you can attach up to eight 375GB local SSDs to the same instance. However, this data will be lost if the VM is killed.

Local SSDs can only be attached to a machine when it is created, but you can attach both local SSDs and persistent disks to the same machine.

Both types of disks are zonal resources.

Cloud Storage

We have extensively covered GCS in a previous section. GCS is not a filesystem, but you can use GCS-Fuse to mount GCS buckets as filesystems in Linux or macOS systems. You can also let apps download and upload data to GCS using standard filesystem semantics.

How to back up your VM's data: Snapshots

Snapshots are backups of your disks. To reduce space, they are created incrementally:

• Back up 1 contains all your disk content
• Back up 2 only contains the data that has changed since back up 1
• Back up 3 only contains the data that has changed since back up 2, and so on

This is enough to restore the state of your disk.

Even though snapshots can be taken without stopping the instance, it is best practice to at least reduce its activity, stop writing data to disk, and flush buffers. This helps you make sure you get an accurate representation of the content of the disk.

Images

Images refer to the operating system images needed to create boot disks for your instances. There are two types of images:

• Public images. They are provided and maintained by Google, open-source communities, and third-party vendors. Ready for you to use as soon as you create your project. Available to anyone
• Custom images. Images that you have created.
• They are linked to the project in which you created them but you can share them with other projects.
• You can create images from persistent disks and other images, both from the same project or shared from another project.
• Related images can be grouped in image families to simplify the management of the different image versions.
• For Linux-based images, you can share them also by exporting them to Cloud Storage as a tar.gz file.

You might be asking yourself what is the difference between an image and a snapshot. Mainly, their purpose. Snapshots are taken as incremental backups of a disk while images are created to spin up new virtual machines and configure instance templates.

Note on images vs startup scripts:

For simple setups, startup scripts are also an option. They can be used to test changes quickly, but the VMs will take longer to be ready compared to using an image where all the needed software is installed, configured, and so on.

Instance groups

Instance groups let you treat a group of instances as a single unit and they come in two flavors:

• Unmanaged instance group. Formed by a heterogeneous group of instances that required individual configuration settings.
• Managed instance group (MIG). This is the preferred option when possible. All the machines look the same, making it easy to configure them, create them in multiple zones (high availability), replace them if they become unhealthy (auto-healing), balance the traffic among them, and create new instances if they traffic increases (horizontal scaling).

To create a MIGs, you need to define an instance template, specifying your machine type, zone, OS image, startup and shutdown scripts, and so on. Instance templates are immutable.

To update a MIG, you need to create a new template and use the Managed Instance Group Updated to deploy the new version to every machine in the group.

This functionality can be used to create canary tests, deploying your changes to a small fraction of your machines first.

Visit this link to know more about Google's recommendations to ensure an application deployed via a managed instance group can handle the load even if an entire zone fails.

Security best practices for GCE

To increase the security of your infrastructure in GCE, have a look at:

• Shielded VMs
• Prevent instances from being reached from the public internet
• [Trusted images](https://cloud.google.com/compute/docs/images/restricting-image-access#:~:text=Use the Trusted image feature,images%2C disks%2C and snapshots.) to make sure your users can only create disks from images in specific projects

App Engine

App Engine is a great choice when you want to focus on the code and let Google handle your infrastructure. You just need to choose the region where your app will be deployed (this cannot be changed once it is set). Amongst its main use cases are websites, mobile apps, and game backends.

You can easily update the version of your app that is running via the command line or the Google Console.

Also, if you need to deploy a risky update to your application, you can split the traffic between the old and the risky versions for a canary deployment. Once you are happy with the results, you can route all the traffic to the new version.

There are two App Engine environments:

• Standard. This version can quickly scale up or down (even to zero instances) to adjust to the demand. Currently, only a few programming languages are supported (Go, Java, PHP, and Python) and you do not have access to a VPC (including VPN connections). It can be scaled down to zero instances.
• Flexible. Your code runs in Docker containers in GCE, hence more flexible than the Standard environment. However, creating new instances is slower and it cannot be scaled down to zero instances. It is suited for more consistent traffic.

Regardless of the environment, there are no up-front costs and you only pay for what you use (billed per second).

Memcache is a built-in App Engine, giving you the possibility to choose between a shared cache (default, free option) or a dedicated cache for better performance.

Visit this link to know more about the best practices you should follow to maximize the performance of your app.

Google Kubernetes Engine (GKE)

Kubernetes is an open-source container orchestration system, developed by Google.

Kubernetes is a very extensive topic in itself and I will not cover here. You just need to know that GKE makes it easy to run and manage your Kubernetes clusters on GCP.

Google also provides Container Registry to store your container images - think of it as your private Docker Hub.

Note: You can use Cloud Build to run your builds in GCP and, among other things, produce Docker images and store them in Container Registry. Cloud Build can import your code from Google Cloud Storage, Cloud Source Repository, GitHub, or Bitbucket.

Cloud Functions

Cloud Functions are the equivalent of Lambda functions in AWS. Cloud functions are serverless. They let you focus on the code and not worry about the infrastructure where it is going to run.

With Cloud Functions it is easy to respond to events such as uploads to a GCS bucket or messages in a Pub/Sub topic. You are only charged for the time your function is running in response to an event.

How to work with Big Data in GCP

BigQuery

BigQuery is Google's serverless data warehousing and provides analytics capabilities for petabyte-scale databases.

BigQuery automatically backs up your tables, but you can always export them to GCS to be on the safe side - incurring extra costs.

Data can be ingested in batches (for instance, from a GCS bucket) or from a stream in multiple formats: CSV, JSON, Parquet, or Avro (most performant). Also, you can query data that resides in external sources, called federated sources, for example, GCS buckets.

You can interact with your data in BigQuery using SQL via the

• Google Console.
• Command-line, running commands like bq query 'SELECT field FROM ....
• REST API.
• Code using client libraries.

User-Defined Functions allow you to combine SQL queries with JavaScript functions to create complex operations.

BigQuery is a columnar data store: records are stored in columns. Tables are collections of columns and datasets are collections of tables.

Jobs are actions to load, export, query, or copy data that BigQuery runs on your behalf.

Views are virtual tables defined by a SQL query and are useful sharing data with others when you want to control exactly what they have access to.

Two important concepts related to tables are:

• Partitioned tables. To limit the amount of data that needs to be queried, tables can be divided into partitions. This can be done based on ingest time or including a timestamp or date column or an integer range. This way it is easy to query for certain periods without querying the full table. To reduce costs, you can define an expiration period after which the partition will be deleted.
• Clustered tables. Data are clustered by column (for instance, order_id). When you query your table, only the rows associated with this column will be read. BigQuery will perform this clustering automatically based on one or more columns.

Using IAM roles, you can control access at a project, dataset, or view level, but not at the table level. Roles are complex for BigQuery, so I recommend checking the documentation.

For instance, the jobUser role only lets you run jobs while the user role lets you run jobs and create datasets (but not tables).

Your costs depend on how much data you store and stream into BigQuery and how much data you query. To reduce costs, BigQuery automatically caches previous queries (per user). This behavior can be disabled.

When you don't edit data for 90 days, it automatically moves to a cheaper storage class. You pay for what you use, but it is possible to opt for a flat rate (only if you need more than the 2000 slots that are allocated by default).

Check these links to see how to optimize your performance and costs.

Cloud Pub/Sub

Pub/Sub is Google's fully-managed message queue, allowing you to decouple publishers (adding messages to the queue) and subscribers (consuming messages from the queue).

Although it is similar to Kafka, Pub/Sub is not a direct substitute. They can be combined in the same pipeline (Kafka deployed on-premise or even in GKE). There are open-source plugins to connect Kafka to GCP, like Kafka Connect.

Pub/Sub guarantees that every message will be delivered at least once but it does not guarantee that messages will be processed in order. It is usually connected to Dataflow to process the data, ensure that the messages are processed in order, and so on.

Pub/Sub support both push and pull modes:

• Push. Messages are sent to subscribers, resulting in lower latency.
• Pull. Subscribers pull messages from topics, better suited for a large volume of messages.

Cloud Pub/Sub vs Cloud Task

Cloud Tasks is another fully-managed service to execute tasks asynchronously and manage messages between services. However, there are differences between Cloud Tasks and Pub/Sub:

• In Pub/Sub, publishers and subscribers are decoupled. Publishers know nothing about their subscribers. When they publish a message, they implicitly cause one or multiple subscribers to react to a publishing event.
• In Cloud Tasks, the publisher stays in control of the execution. Besides, Cloud Tasks provide other features unavailable for Pub/Sub like scheduling specific delivery times, delivery rate controls, configurable retries, access and management of individual tasks in a queue, task/message creation deduplication.

For more details, check out this link.

Cloud Dataflow

Cloud Dataflow is Google's managed service for stream and batch data processing, based on Apache Beam.

You can define pipelines that will transform your data, for example before it is ingested in another service like BigQuery, BigTable, or Cloud ML. The same pipeline can process both stream and batch data.

A common pattern is to stream data into Pub/Sub, let's say from IoT devices, process it in Dataflow, and store it for analysis in BigQuery.

But Pub/Sub does not guarantee that the order in which messages are pushed to the topics will be the order in which the messages are consumed. However, this can be done with Dataflow.

Cloud Dataproc

Cloud Dataproc is Google's managed the Hadoop and Spark ecosystem. It lets you create and manage your clusters easily and turn them off when you are not using them, to reduce costs.

Dataproc can only be used to process batch data, while Dataflow can handle also streaming data.

Google recommends using Dataproc for a lift and leverage migration of your on-premise Hadoop clusters to the cloud:

• Reduce costs turning your cluster off when you are not using it.
• Leverage Google's infrastructure
• Use some preemptible virtual machines to reduce costs
• Add larger (SSD) persistent disks to improve performance
• BigQuery can replace Hive and BigTable can replace HBase
• Cloud Storage replaces HDFS. Just upload your data to GCS and change the prefixes hdfs:// to gs://

Otherwise, you should choose Cloud Dataflow.

Dataprep

Cloud Dataprep provides you with a web-based interface to clean and prepare your data before processing. The input and output formats include, among others, CSV, JSON, and Avro.

After defining the transformations, a Dataflow job will run. The transformed data can be exported to GCS, BigQuery, etc.

Cloud Composer

Cloud Composer is Google's fully-managed Apache Airflow service to create, schedule, monitor, and manage workflows. It handles all the infrastructure for you so that you can concentrate on combining the services I have described above to create your own workflows.

Under the hood, a GKE cluster will be created with Airflow in it and GCS will be used to store files.

AI and Machine Learning in GCP

Covering the basics of machine learning would take another article. So here, I assume you are familiar with it and will show you how to train and deploy your models in GCP.

We'll also look at what APIs are available to leverage Google's machine learning capabilities in your services, even if you are not an expert in this area.

AI Platform

AI Platform provides you with a fully-managed platform to use machine learning libraries like Tensorflow. You just need to focus on your model and Google will handle all the infrastructure needed to train it.

After your model is trained, you can use it to get online and batch predictions.

Cloud AutoML

Google lets you use your data to train their models. You can leverage models to build applications that are based on natural language processing (for example, document classification or sentiment analysis applications), speech processing, machine translation, or video processing (video classification or object detection).

How to explore and visualize your data in GCP

Cloud Data Studio

Data Studio lets you create visualizations and dashboards based on data that resides in Google services (YouTube Analytics, Sheets, AdWords, local upload), Google Cloud Platform (BigQuery, Cloud SQL, GCS, Spanner), and many third-party services, storing your reports in Google Drive.

Data Studio is not part of GCP, but G-Suite, thus its permissions are not managed using IAM.

There are no additional costs for using Data Studio, other than the storage of the data, queries in BigQuery, and so on. Caching can be used to improve performance and reduce costs.

Cloud Datalab