This article explores various approaches to handling database migrations in a Kubernetes environment, with a focus on Go tooling. You'll get the most out of this article if you already have some experience with Go, Kubernetes, and relational databases.

Table of Contents

• The challenge of migrations in Kubernetes

• Popular migration tools for Golang

• Run migrations inside the application

• Run migrations in initContainers

• Run migrations as a Kubernetes Job

• Helm hooks

• Best practices for Kubernetes migrations

• Conclusion

• Resources

The Challenge of Migrations in Kubernetes

Kubernetes introduces new challenges for database migrations:

• Multiple replicas starting simultaneously. These can span the same migration twice which may introduce some database locks.

• Separation of concerns between application and migration logic. This means it’s good to be able to run or rollback migrations without redeploying your application.

Popular Migration Tools for Golang

As I mentioned in another post, there are a few different tools you can use to manage your migrations. They are quite similar, so I personally don’t have a strong preference between once or another. I just wanted to provide a few options so you know what the popular tools are.

1. golang-migrate

• Widely used and supports numerous databases.

• Simple CLI and API.

• Supports various migration sources (local files, S3, Google Storage).

2. goose

• Supports main SQL databases.

• Allows migrations written in Go for complex scenarios.

• Flexible versioning schemas.

3. atlas

• Powerful database schema management tool.

• Supports declarative and versioned migrations.

• Offers integrity checks and migration linting.

• Provides GitHub Actions and Terraform provider.

Run Migrations Inside the Application

A naïve implementation would be to run the code of the migration directly inside your main function before you start your server.

Example using golang-migrate:

package main

import (
    "database/sql"
    "fmt"
    "log"
    "net/http"

    "github.com/golang-migrate/migrate/v4"
    "github.com/golang-migrate/migrate/v4/database/postgres"
    _ "github.com/golang-migrate/migrate/v4/source/file"
    _ "github.com/lib/pq"
)

func main() {
    // Database connection parameters
    url := "postgres://user:pass@localhost:5432/dbname"

    // Connect to the database
    db, err := sql.Open("postgres", url)
    if err != nil {
        log.Fatalf("could not connect to database: %v", err)
    }
    defer db.Close()

    // Run migrations
    if err := runMigrations(db); err != nil {
        log.Fatalf("could not run migrations: %v", err)
    }

    // Run the application, for example start the server
    if err := http.ListenAndServe(":8080", nil); err != nil {
        log.Fatalf("server failed to start: %v", err)
    }
}

func runMigrations(db *sql.DB) error {
    driver, err := postgres.WithInstance(db, &postgres.Config{})
    if err != nil {
        return fmt.Errorf("could not create database driver: %w", err)
    }

    m, err := migrate.NewWithDatabaseInstance(
        "file://migrations", // Path to your migration files
        "postgres",          // Database type
        driver,
    )
    if err != nil {
        return fmt.Errorf("could not create migrate instance: %w", err)
    }

    if err := m.Up(); err != nil && err != migrate.ErrNoChange {
        return fmt.Errorf("could not run migrations: %w", err)
    }

    log.Println("migrations completed successfully")
    return nil
}

However, these could cause different issues like your migrations being slow and Kubernetes considering that the pod didn’t start successfully and therefore killing it. You could run those migrations in a Go routine, but how do you handle failures then?

In cases when multiple pods are created at the same time, you would have a potential concurrency problem.

It also means your migrations need to be inside your Docker image.

Even with its downsides, this approach might work well for quick and stable database changes and small projects.

Run Migrations in initContainers

By using initContainers in your Kubernetes Deployment, it will run the migration before the main application container starts. This is a good first solution for when scaling is not a problem yet.

If the initContainer fails, the blue/green deployment from Kubernetes won’t go further and your previous pods stay where they are. This prevents having a newer version of the code without the planned migration.

Example:

initContainers:
  - name: migrations
    image: migrate/migrate:latest
    command: ['/migrate']
    args: ['-source', 'file:///migrations', '-database','postgres://user:pass@db:5432/dbname', 'up']

This approach might work well for quick and stable database changes for deployments with a single Pod. And it already separates the application and migration layers.

Run Migrations as a Kubernetes Job

You could create a Kubernetes Job that runs your migrations, and trigger that job during the deployment process before rolling out the application.

Example:

apiVersion: batch/v1
kind: Job
metadata:
  name: db-migrate
spec:
  template:
    spec:
      containers:
      - name: migrate
        image: your-migration-image:latest
        command: ['/app/migrate']

You can also combine it with initContainers, making sure that the pod starts only when the job is successful.

initContainers:
  - name: migrations-wait
    image: ghcr.io/groundnuty/k8s-wait-for:v2.0
    args:
      - "job"
      - "my-migration-job"

This approach can solve the problems related to multiple replicas mentioned above.

Helm Hooks

If you use Helm, it has hooks that you can use for running migrations during chart installation/upgrade. You just define a pre-install or pre-upgrade hook in your Helm chart.

pre-install-hook.yaml:

apiVersion: batch/v1
kind: Job
metadata:
  name: {{ include "mychart.fullname" . }}-migrations
  annotations:
    "helm.sh/hook": pre-install,pre-upgrade
    "helm.sh/hook-weight": "-5"
    "helm.sh/hook-delete-policy": hook-succeeded
spec:
  template:
    spec:
      containers:
        - name: migrations
          image: your-migrations-image:tag
          command: ["./run-migrations.sh"]

In this example, the pre-install hook executes after templates are rendered, but before any resources are created in Kubernetes.

This of course works only when you use Helm, meaning you need to find something else if you decide not to use Helm.

Best Practices for Kubernetes Migrations

Decouple migrations from application code:

1. Create a separate Docker image for migrations. This ensures that migration logic is encapsulated and doesn't interfere with the application codebase.

2. Use tools like Atlas to manage migrations independently. Tools like Atlas provide features for automating migration processes, scheduling, and rollback.

Use version control for migrations:

1. Store migration files in your Git repository. This ensures a complete history of migration changes, making it easier to track and revert changes.

2. Use sequential or timestamp-based versioning. Sequential versioning guarantees the correct order of migrations which is very important for relational databases.

Ensure idempotent migrations:

1. Ensure migrations can be run multiple times without side effects. Idempotent migrations prevent accidental data corruption or inconsistencies if a migration is run multiple times.

Have a rollback strategy

1. Implement and test rollback procedures for each migration. Having a rollback strategy ensures that you can revert changes if a migration fails or causes unexpected issues.

Perform monitoring and logging

1. Use tools like Atlas Cloud for visibility into migration history. Atlas Cloud provides detailed logs and history of migrations, making it easy to track changes and troubleshoot issues.

Conclusion

Managing database migrations in a Kubernetes environment requires careful planning and execution.

By leveraging tools like golang-migrate, goose, or atlas, and following best practices, you can create robust, scalable, and maintainable migration strategies.

Remember to decouple migrations from application code, use version control, and implement proper monitoring to ensure smooth database evolution in your Kubernetes-based architecture.

Resources

• Discover more articles from packagemain.tech

• Helm Hooks

Kubernetes is a very helpful tool for cloud-native developers. But it doesn't cover all the bases on its own – there are some things that Kubernetes cannot solve or that are outside its scope.

This is one of the reasons why open source projects are so great. They help amazing tools become even more amazing when we combine them with other awesome open-source tools. And often these tools were developed for the sole purpose of filling the gaps. One of these tools is Helm.

What is Helm?

Helm is widely known as "the package manager for Kubernetes". Although it presents itself like this, its scope goes way beyond that of a simple package manager. However, let's start at the beginning.

Helm is an open-source project which was originally created by DeisLabs and donated to CNCF, which now maintains it. The original goal of Helm was to provide users with a better way to manage all the Kubernetes YAML files we create on Kubernetes projects.

The path Helm took to solve this issue was to create Helm Charts. Each chart is a bundle with one or more Kubernetes manifests – a chart can have child charts and dependent charts as well.

This means that Helm installs the whole dependency tree of a project if you run the install command for the top-level chart. You just a single command to install your entire application, instead of listing the files to install via kubectl.

Charts allow you to version your manifest files too, just like we do with Node.js or any other package. This lets you install specific chart versions, which means keeping specific configurations for your infrastructure in the form of code.

Helm also keeps a release history of all deployed charts, so you can go back to a previous release if something went wrong.

Helm supports Kubernetes natively, which means you don't have to write any complex syntax files or anything to start using Helm. Just drop your template files into a new chart and you're good to go.

But why should we use it? Managing application manifests can be easily done with a few combinations of commands.

Why Should You Use Helm?

Helm really shines where Kubernetes didn't go. For instance, templating. The scope of the Kubernetes project is to deal with your containers for you, not your template files.

This makes it overly difficult to create truly generic files to be used across a large team or a large organization with many different parameters that need to be set for each file.

And also, how do you version sensitive information using Git when template files are plain text?

The answer: Go templates. Helm allows you to add variables and use functions inside your template files. This makes it perfect for scalable applications that'll eventually need to have their parameters changed. Let's look at an example.

I have an open-source project called Zaqar, a simple email microservice for Node.js which communicates with SendGrid. The project is basically composed of a service, a deployment and an autoscaler.

Let's take the deployment file as the example. I'd have something like this:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: zaqar
  namespace: default
  labels:
    app: zaqar
    version: v1.0.0
    env: production
spec:
  replicas: 1
  selector:
    matchLabels:
      app: zaqar
      env: production
  template:
    metadata:
      labels:
        app: zaqar
        version: v1.0.0
        env: production
    spec:
      containers:
        - name: zaqar
          image: "khaosdoctor/zaqar:v1.0.0"
          imagePullPolicy: IfNotPresent
          env:
            - name: SENDGRID_APIKEY
              value: "MY_SECRET_KEY"
            - name: DEFAULT_FROM_ADDRESS
              value: "my@email.com"
            - name: DEFAULT_FROM_NAME
              value: "Lucas Santos"
          ports:
            - name: http
              containerPort: 3000
              protocol: TCP
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              cpu: 250m
              memory: 256Mi

If I wanted to use this template on a CI pipeline, or publish it on my GitHub, I'd need to replace the variable parts with placeholders. So we can replace these texts with the required information.

In this case, both the version tag, the env label, and the environment variables would be replaced by placeholders, like this:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: zaqar
  namespace: default
  labels:
    app: zaqar
    version: #!VERSION!#
    env: #!ENV!#
spec:
  replicas: 1
  selector:
    matchLabels:
      app: zaqar
      env: #!ENV!#
  template:
    metadata:
      labels:
        app: zaqar
        version: #!VERSION!#
        env: #!ENV!#
    spec:
      containers:
        - name: zaqar
          image: "khaosdoctor/zaqar:#!VERSION!#"
          imagePullPolicy: IfNotPresent
          env:
            - name: SENDGRID_APIKEY
              value: "#!SENDGRID_KEY!#"
            - name: DEFAULT_FROM_ADDRESS
              value: "#!FROM_ADDR!#"
            - name: DEFAULT_FROM_NAME
              value: "#!FROM_NAME!#"
          ports:
            - name: http
              containerPort: 3000
              protocol: TCP
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              cpu: 250m
              memory: 256Mi

We can now run our CI pipeline. But before we do that, we need to replace our placeholders with the actual values.

For this we can use sed and its "super easy" syntax sed 's/#!PLACEHOLDER!#/replacement/g', and pipe this down until we finish all the placeholders. The final command would be something like this:

cat deploy.yaml | \
    sed 's/#!ENV!#/production/g' | \
    sed 's/#!VERSION!#/v1.0.0/g' | \
    sed 's/#!SENDGRID_KEY!#/MyKey/g' | \
    sed 's/#!FROM_ADDR!#/my@email.com/g' | \
    sed 's/#!FROM_NAME!#/Lucas Santos/g'

By default, sed outputs everything to the stdout, so we can add another pipe to kubectl -f, like <all the command from before> | kubectl -f -. Then we'll have our deployment in place. The only problem is that we need to do the same for all the other files.

Now imagine a larger project, with lots of other variables and placeholders. You'd probably write a script to do it for you, wouldn't you? That script is Helm.

When you create a Chart (more on this later on), we have a specific directory tree that we must follow so Helm understands what we want to do. Inside the templates directory, we can add our manifest files, with native go templating, like this:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Values.name }}
  namespace: {{ default .Release.Namespace .Values.namespace }}
  labels:
    app: {{ .Values.name }}
    version: {{ .Values.image.tag }}
    env: {{ .Values.env }}
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      app: {{ .Values.name }}
      env: {{ .Values.env }}
  template:
    metadata:
      labels:
        app: {{ .Values.name }}
        version: {{ .Values.image.tag }}
        env: {{ .Values.env }}
    spec:
      containers:
        - name: {{ .Chart.Name }}
          image: "khaosdoctor/zaqar:{{ .Values.image.tag }}"
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          env:
            - name: SENDGRID_APIKEY
              value: {{ required "You must set a valid Sendgrid API key" .Values.environment.SENDGRID_APIKEY | quote }}
            - name: DEFAULT_FROM_ADDRESS
              value: {{ required "You must set a default from address" .Values.environment.DEFAULT_FROM_ADDRESS | quote }}
            - name: DEFAULT_FROM_NAME
              value: {{ required "You must set a default from name" .Values.environment.DEFAULT_FROM_NAME | quote }}
          ports:
            - name: http
              containerPort: 3000
              protocol: TCP
          resources:
            {{- toYaml .Values.resources | nindent 12 }}

All those values can be obtained from a Values.yaml file (for default values), or you can set them in the CLI using the --set <path> value flag.

If we want to install our chart we can issue the following command:

helm upgrade --install --create-namespace myChart ./path/to/my/chart \
  --set image.tag=v1.0.0 \
  --set env=production \
  --set environment.SENDGRID_APIKEY=myKey \
  --set environment.DEFAULT_FROM_ADDRESS="my@email.com" \
  --set environment.DEFAULT_FROM_NAME="Lucas Santos"

Helm also allows us to use functions inside our deployments. So we can have default functions to fallback to default values if they're not filled, like the namespace. Or we can have required which displays a message and fails to install the chart if the value is not provided, which is the case of our environment variables.

There are a lot of other useful functions in their docs - check 'em out.

Now we are able to not only more efficiently manage our application's resources, but also to publish these resources in an open-source version system without any hassle or security issue.

How to Create a Helm Chart

It's pretty easy to create a chart in Helm. First, you need to have Helm installed. Then, just type in helm create <chart name> and it will create a directory filled with files and other directories. Those files are required for Helm to create a chart.

Let's take a closer look at what this file tree looks like and what the files are within it:

• chart.yaml: This is where you'll put the information related to your chart. That includes the chart version, name, and description so you can find it if you publish it on an open repository. Also in this file you'll be able to set external dependencies using the dependencies key.
• values.yaml: Like we saw before, this is the file that contains defaults for variables.
• templates (dir): This is the place where you'll put all your manifest files. Everything in here will be passed on and created in Kubernetes.
• charts: If your chart depends on another chart you own, or if you don't want to rely on Helm's default library (the default registry where Helm pull charts from), you can bring this same structure inside this directory. Chart dependencies are installed from the bottom to the top, which means if chart A depends on chart B, and B depends on C, the installation order will be C ->B ->A.

There are other fields, but these are the most common, and they're the required ones. You can take a quick look at Zaqar's repository to check on how we can publish open source charts.

A quick note: When installing Helm, make sure you're installing version 3. Version 2 still works, but it needs a server-side component called Tiller, which ties your helm installation to a single cluster. Helm 3 removed this need with the addition of several CRDs, but it's not supported in all Kubernetes versions.

How to Host a Helm Chart

Ok, you created your chart, now what? Do we have to download the entire repository to install those charts? No! Helm has a public library for the most used charts, which kind of works like Docker Hub.

You can also create your own chart repository and host it online. Helm drinks from the same fountain as HomeBrew, or Linux. You can tap these repositories to download charts contained in them.

Since a chart repository is basically an index.yaml file served from a static web server, you can pretty much create a chart repository out of anywhere.

Take Zaqar, for example – it's hosted on GitHub Pages and is accessible through my domain. When Helm looks for an index.yaml file it's actually looking for the list of available versions of that chart, their SHA256 digests, and the location of the packaged .tgz file to download the chart itself. This is pretty much what NPM does under the hood (overly simplified).

This means you don't need to have your repository cloned forever, and your charts can be private as well. You only need to create a chart repository.

You can even use hosted services like Azure CR to do the job, or you can have a full solution called Chart Museum, which allows you to store your charts and provides you with a neat UI.

Conclusion

Helm is here to stay. It has helped and will help a lot of Kubernetes developers out there for a long time.

If you want to know how to use Helm, you can refer to their docs, or you can take this free learn module on how to deploy your applications on Kubernetes the easy way with Helm.

There are different ways of running production services at a high scale. One popular solution for running containers in production is Kubernetes. But interacting with Kubernetes directly comes with some caveats.

Helm tries to solve some of the challenges with useful features that increase productivity and reduce maintenance efforts of complex deployments.

In this post you will learn:

• What Helm is
• The most common use-cases of Helm
• How to configure and deploy a publicly available Helm package
• How to deploy a custom application using Helm

Every code example in this post requires a Kubernetes cluster. The easiest way to get a cluster to play with is to install Docker and activate its Kubernetes cluster feature. Also, you need to install kubectl and Helm to interact with your cluster.

Please note: When you try the examples, be patient. If you are too fast then the containers are not ready. It might take a few minutes until the containers can receive requests.

What is Helm?

Helm calls itself ”The Kubernetes package manager”. It is a command-line tool that enables you to create and use so-called Helm Charts.

A Helm Chart is a collection of templates and settings that describe a set of Kubernetes resources. Its power spans from managing a single node definition to a highly scalable multi-node cluster.

The architecture of Helm has changed over the last years. The current version of Helm communicates directly to your Kubernetes cluster via Rest. If you read something about Tiller in the context of Helm, then you're reading an old article. Tiller was removed in Helm 3.

Helm itself is stateful. When a Helm Chart gets installed, the defined resources are getting deployed and meta-information is stored in Kubernetes secrets.

How to Deploy a Simple Helm Application

Let’s get our hands dirty and make sure Helm is ready to use.

First, we need to be connected to a Kubernetes cluster. In this example, I will concentrate on a Kubernetes cluster that comes with your Docker setup. So if you use some other Kubernetes cluster, configurations and outputs might differ.

$ kubectl config use-context docker-desktop

  Switched to context "docker-desktop".

$ kubectl get node

   NAME             STATUS   ROLES    AGE   VERSION
   docker-desktop   Ready    master   20d   v1.19.3

Let’s deploy an Apache webserver using Helm. As a first step, we need to tell Helm what location to search by adding a Helm repository:

$ helm repo add bitnami https://charts.bitnami.com/bitnami

Let’s install the actual container:

$ helm install my-apache bitnami/apache --version 8.0.2

After a few minutes your deployment is ready. We can check the state of the containers using kubectl:

$ kubectl get pods

  NAME                               READY   STATUS    RESTARTS   AGE
  my-apahe-apache-589b8df6bd-q6m2n   1/1     Running   0          2m27s

Now, open http://localhost to see the default Apache exposed website locally. Also, Helm can show us information about current deployments:

$ helm list

 NAME         REVISION    STATUS      CHART           VERSION
 my-apache    1        deployed      apache-8.0.2    2.4.46

How to Upgrade a Helm Application

We can upgrade our deployed application to a new version like this:

$ helm upgrade my-apache bitnami/apache --version 8.0.3

$ helm list

NAME         REVISION    STATUS      CHART          VERSION
my-apache    2           deployed    apache-8.0.3    2.4.46

The column Revision indicates that this is the 2nd version we've deployed.

How to Rollback a Helm Application

So let’s try to rollback to the first deployed version:

$ helm rollback my-apache 1                    

Rollback was a success! Happy Helming!

$ helm list

NAME         REVISION    STATUS      CHART        VERSION
my-apache    3           deployed    apache-8.0.2    2.4.46

This is a very powerful feature that allows you to roll back changes in production quickly.

I mentioned that Helm stores deployment information in secrets – here they are:

$ kubectl get secret

NAME                    TYPE                         DATA   AGE
default-token-nc4hn               kubernetes.io/sat        3      20d
sh.helm.release.v1.my-apache.v1   helm.sh/release.v1        1      1m
sh.helm.release.v1.my-apache.v2   helm.sh/release.v1        1      1m
sh.helm.release.v1.my-apache.v3   helm.sh/release.v1        1      1m

How to Remove a Deployed Helm Application

Let’s clean up our Kubernetes by removing the my-apache release:

$ helm delete my-apache

  release "my-apache" uninstalled

Helm gives you a very convenient way of managing a set of applications that enables you to deploy, upgrade, rollback and delete.

Now, we are ready to use more advanced Helm features that will boost your productivity!

How to Access Production-Ready Helm Charts

You can search public hubs for Charts that enable you to quickly deploy your desired application with a customizable configuration.

A Helm Chart doesn't just contain a static set of definitions. Helm comes with capabilities to hook into any lifecycle state of a Kubernetes deployment. This means during the installation or upgrade of an application, various actions can be executed like creating a database update before updating the actual database.

This powerful definition of Helm Charts lets you share and improve an executable description of a deployment setup that spans from initial installation and version upgrades to rollback capabilities.

Helm might be heavy for a simple container like a single node web server, but it’s very useful for more complex applications. For example it works great for a distributed system like Kafka or Cassandra that usually runs on multiple distributed nodes on different datacenters.

We've already leveraged Helm to deploy a single Apache container. Now, we will deploy a production-ready WordPress application that contains:

• Containers that serve WordPress,
• Instances of MariaDB for persistence and
• Prometheus sidecar containers for each WordPress container to expose health metrics.

Before we deploy, it’s recommended to increase your Docker limits to at least 4GB of memory.

Setting everything up sounds like a job that would take weeks. To make it resilient and scale, probably a job that would take months. In these areas, Helm Charts can really shine. Due to the growing community, there might already be a Helm Chart that we can use.

How to Deploy WordPress and MariaDB

There are different public hubs for Helm Charts. One of them is artifacthub.io. We can search for “WordPress” and find an interesting WordPress Chart.

On the right side, there is an install button. If you click it, you get clear instructions about what to do:

$ helm repo add bitnami https://charts.bitnami.com/bitnami

$ helm install my-wordpress bitnami/wordpress --version 10.1.4

You will also see some instructions that tell you how to access the admin interface and the admin password after installation.

Here is how you can get and decode the password for the admin user on Mac OS:

$ echo Username: user
$ echo Password: $(kubectl get secret --namespace default my-wordpress-3 -o jsonpath="{.data.wordpress-password}" | base64 --decode)

Username: user
Password: sZCa14VNXe

On windows, you can get the password for the user user in the powershell:

$pw=kubectl get secret --namespace default my-wordpress -o jsonpath="{.data.wordpress-password}"
[System.Text.Encoding]::UTF8.GetString([System.Convert]::FromBase64String($pw))

Our local development will be available at: http://localhost.
Our admin interface will be available at: https://localhost/admin.

So we have everything to run it locally. But in production, we want to scale some parts of it to serve more and more visitors. We can scale the number of WordPress services. We also want to expose some health metrics like the usage of our CPU and memory.

We can download the example configuration for production from the maintainer of the WordPress Chart. The most important changes are:

### Start 3 WordPress instances that will all receive 
### requests from our visitors. A load-balancer will distribute calls 
### to all containers evenly.
replicaCount: 3

### start a sidecar container that will expose metrics for your wordpress container
metrics:
  enabled: true
  image:
    registry: docker.io
    repository: bitnami/apache-exporter
    tag: 0.8.0-debian-10-r243

Let’s stop the default application:

$ helm delete my-wordpress

  release "my-wordpress" uninstalled

How to Start a Multi-instance WordPress and MariaDB Deployment

Deploy a new release using the production values:

$ helm install my-wordpress-prod bitnami/wordpress --version 10.1.4 -f values-production.yaml

This time, we have more containers running:

$ kubectl get pods
NAME                                 READY   STATUS    RESTARTS   AGE
my-wordpress-prod-5c9776c976-4bs6f   2/2     Running   0          103s
my-wordpress-prod-5c9776c976-9ssmr   2/2     Running   0          103s
my-wordpress-prod-5c9776c976-sfq84   2/2     Running   0          103s
my-wordpress-prod-mariadb-0          1/1     Running   0          103s

We see 4 lines: 1 line for MariaDB, and 3 lines for our actual WordPress pods.

A pod in Kubernetes is a group of containers. Each group contains 2 containers, one for WordPress and one for an exporter for Prometheus that exposes valuable metrics in a special format.

As in the default setup, we can open localhost and play with our WordPress application.

How to Access Exposed Health Metrics

We can check the exposed health metrics by proxying to one of the running pods:

kubectl port-forward my-wordpress-prod-5c9776c976-sfq84 9117:9117

Make sure to replace the pod-id with your own pod ID when you execute the port-forward command.

Now, we are connected to port 9117 of the WordPress Prometheus exporter and map the port to our local port 9117. Open http://localhost:9117 to check the output.

If you are not used to the Prometheus format, it might be a little bit confusing in the beginning. But it’s actually pretty easy to read. Each line without ‘#’ contains a metric key and a value behind it:

apache_cpuload 1.2766
process_resident_memory_bytes 1.6441344e+07

If you are not used to such metrics, don't worry – you will get used to them quickly. You can Google each of the keys and find out what it means. After some time, you will identify what metrics are the most valuable for you and how they behave as soon as your containers receive more and more production traffic.

Let’s tidy up our setup by:

$ helm delete my-wordpress-prod

  release "my-wordpress-prod" uninstalled

We touched on a lot of deployment areas and features. We deployed multiple WordPress instances and scaled it up to more containers for production. You could even go one step further and activate auto-scaling. Check out the documentation of the Helm Chart and play around with it!

MariaDB Helm Chart

The persistence of the helm Chart for WordPress depends on MariaDB. It builds on another Helm Chart for MariaDB that you can configure and scale to your needs by, for example, starting multiple replicas.

The possibilities that you have when running containers in production using Kubernetes are enormous. The definition of the WordPress Chart is publicly available.

In the next section, we will create our own Helm Chart with a basic application to understand the fundamentals of creating a Helm Chart and to make a static container deployment more dynamic.

How to Create a Template for Custom Applications

Helm adds a lot more flexibility to your Kubernetes deployment files. Kubernetes deployment files are static by their nature. This means, adjustments like

• desired container count,
• environment variables or
• CPU and memory limit

are not adjustable by using plain Kubernetes deployment files. Either you solve this by duplicating configuration files or you put placeholders in your Kubernetes deployment files that are replaced at deploy-time.

Both of these solutions require some additional work and will not scale well if you deploy a lot of applications with different variations.

But for sure, there is a smarter solution that is based on Helm that contains a lot of handy features from the Helm community. Let’s create a custom Chart for a blogging engine, this time for a NodeJS based blog called ghost blog.

How to Start a Ghost Blog Using Docker

A simple instance can be started using pure Docker:

docker run --rm -p 2368:2368 --name my-ghost ghost

Our blog is available at: http://localhost:2368.

Let's stop the instance to be able to launch another one using Kubernetes:

$ docker rm -f my-ghost

  my-ghost

Now, we want to deploy the ghost blog with 2 instances in our Kubernetes cluster. Let’s set up a plain deployment first:

# file 'application/deployment.yaml'

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ghost-app
spec:
  selector:
    matchLabels:
      app: ghost-app
  replicas: 2
  template:
    metadata:
      labels:
        app: ghost-app
    spec:
      containers:
        - name: ghost-app
          image: ghost

          ports:
            - containerPort: 2368

and put a load balancer before it to be able to access our container and to distribute the traffic to both containers:

# file 'application/service.yaml'

apiVersion: v1
kind: Service
metadata:
  name: my-service-for-ghost-app
spec:
  type: LoadBalancer
  selector:
    app: ghost-app
  ports:
    - protocol: TCP
      port: 80
      targetPort: 2368

We can now deploy both resources using kubectl:

$ kubectl apply -f ./appplication/deployment.yaml -f ./appplication/service.yaml

  deployment.apps/ghost-app created
  service/my-service-for-ghost-app created

The ghost application is now available via http://localhost. Let's again stop the application:

$ kubectl delete -f ./appplication/deployment.yaml -f ./appplication/service.yaml

  deployment.apps/ghost-app delete
  service/my-service-for-ghost-app delete

So far so good, it works with plain Kubernetes. But what if we need different settings for different environments?

Imagine that we want to deploy it to multiple data centers in different stages (non-prod, prod). You will end up duplicating your Kubernetes files over and over again. It will be hell for maintenance. Instead of scripting a lot, we can leverage Helm.

Let’s create a new Helm Chart from scratch:

$ helm create my-ghost-app

  Creating my-ghost-app

Helm created a bunch of files for you that are usually important for a production-ready service in Kubernetes. To concentrate on the most important parts, we can remove a lot of the created files. Let’s go through the only required files for this example.

We need a project file that is called Chart.yaml:

# Chart.yaml

apiVersion: v2
name: my-ghost-app
description: A Helm chart for Kubernetes
type: application
version: 0.1.0
appVersion: 1.16.0

The deployment template file:

# templates/deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ghost-app
spec:
  selector:
    matchLabels:
      app: ghost-app
  replicas: {{ .Values.replicaCount }}
  template:
    metadata:
      labels:
        app: ghost-app
    spec:
      containers:
        - name: ghost-app
          image: ghost
          ports:
            - containerPort: 2368
          env:
            - name: url
              {{- if .Values.prodUrlSchema }}
              value: http://{{ .Values.baseUrl }}
              {{- else }}
              value: http://{{ .Values.datacenter }}.non-prod.{{ .Values.baseUrl }}
              {{- end }}

It looks very similar to our plain Kubernetes file. Here, you can see different placeholders for the replica count, and an if-else condition for the environment variable called url. In the following files, we will see all the values defined.

The service template file:

# templates/service.yaml

apiVersion: v1
kind: Service
metadata:
  name: my-service-for-my-webapp
spec:
  type: LoadBalancer
  selector:
    app: ghost-app
  ports:
    - protocol: TCP
      port: 80
      targetPort: 2368

Our Service configuration is completely static.

The values for the templates are the last missing parts of our Helm Chart. Most importantly, there is a default values file required called values.yaml:

# values.yaml

replicaCount: 1
prodUrlSchema: false
datacenter: us-east
baseUrl: myapp.org

A Helm Chart should be able to run just by using the default values. Before you proceed, make sure that you have deleted:

• my-ghost-app/templates/tests/test-connection.yaml
• my-ghost-app/templates/serviceaccount.yaml
• my-ghost-app/templates/ingress.yaml
• my-ghost-app/templates/hpa.yaml
• my-ghost-app/templates/NOTES.txt.

We can get the final output that would be sent to Kubernetes by executing a “dry-run”:

$ helm template --debug my-ghost-app

install.go:159: [debug] Original chart version: ""
install.go:176: [debug] CHART PATH: /helm/my-ghost-app

---
# Source: my-ghost-app/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: my-service-for-my-webapp
spec:
  type: LoadBalancer
  selector:
    app: my-example-app
  ports:
    - protocol: TCP
      port: 80
      targetPort: 2368
---
# Source: my-ghost-app/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ghost-app
spec:
  selector:
    matchLabels:
      app: ghost-app
  replicas: 1
  template:
    metadata:
      labels:
        app: ghost-app
    spec:
      containers:
        - name: ghost-app
          image: ghost
          ports:
            - containerPort: 2368
          env:
            - name: url
              value: us-east.non-prod.myapp.org

Helm inserted all the values and also set url to us-east.non-prod.myapp.org because in the values.yaml, prodUrlSchema is set to false and the datacenter is set to us-east.

To get some more flexibility, we can define some override value files. Let’s define one for each datacenter:

# values.us-east.yaml
datacenter: us-east

# values.us-west.yaml
datacenter: us-west

and one for each stage:

# values.nonprod.yaml
replicaCount: 1
prodUrlSchema: false

# values.prod.yaml
replicaCount: 3
prodUrlSchema: true

We can now use Helm to combine them as we want and check the result again:

$ helm template --debug my-ghost-app -f my-ghost-app/values.nonprod.yaml  -f my-ghost-app/values.us-east.yaml 

install.go:159: [debug] Original chart version: ""
install.go:176: [debug] CHART PATH: /helm/my-ghost-app

---
# Source: my-ghost-app/templates/service.yaml
# templates/service.yaml

apiVersion: v1
kind: Service
metadata:
  name: my-service-for-my-webapp
spec:
  type: LoadBalancer
  selector:
    app: my-example-app
  ports:
    - protocol: TCP
      port: 80
      targetPort: 2368
---
# Source: my-ghost-app/templates/deployment.yaml
# templates/deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ghost-app
spec:
  selector:
    matchLabels:
      app: ghost-app
  replicas: 1
  template:
    metadata:
      labels:
        app: ghost-app
    spec:
      containers:
        - name: ghost-app
          image: ghost
          ports:
            - containerPort: 2368
          env:
            - name: url
              value: http://us-east.non-prod.myapp.org

And for sure, we can do a final deployment:

$ helm install -f my-ghost-app/values.prod.yaml my-ghost-prod ./my-ghost-app/

NAME: my-ghost-prod
LAST DEPLOYED: Mon Dec 21 00:09:17 2020
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None

As before, our ghost blog is available via http://localhost.

We can delete this deployment and deploy the application with us-east and non prod settings like this:

$ helm delete my-ghost-prod                                                 
  release "my-ghost-prod" uninstalled

$ helm install -f my-ghost-app/values.nonprod.yaml -f my-ghost-app/values.us-east.yaml my-ghost-nonprod ./my-ghost-app

We finally clean up our Kubernetes deployment via Helm:

$ helm delete my-ghost-nonprod

So we can combine multiple override value files as we want. We can automate deployments in a flexible way that we need for many use-cases of deployment pipelines.

Especially for companies, this means defining Chart Skeletons once to ensure the required criteria are fulfilled. Later, you can copy them and adjust them to the needs of your application.

Conclusion

The power of a great templating engine and the possibility of executing releases, upgrades, and rollbacks makes Helm great. On top of that comes the publicly available Helm Chart Hub that contains thousands of production-ready templates. This makes Helm a must-have tool in your toolbox if you work with Kubernetes on a bigger scale!

I hope you enjoyed this hands-on tutorial. Motivate yourself to Google around, check out other examples, deploy containers, connect them, and use them.

You will learn many cool features in the future that enable you to ship your application to production in an effortless, reusable and scalable way.

As always, I appreciate any feedback and comments. I hope you enjoyed the article. If you like it and feel the need for a round of applause, follow me on Twitter. I work at eBay Kleinanzeigen, one of the biggest classified companies globally. By the way, we are hiring!

References:

• https://helm.sh/docs/chart_template_guide/getting_started/

Before we dive into the Helm package manager, I'm going to explain some key concepts to deploying any application anywhere. I'll also give you a brief introduction to Kubernetes terminology.

What is Kubernetes?

Kubernetes (K8s) is an open-source system for automating deployment, scaling, and management of containerized applications
kubernetes.io

Now you might be asking yourself, "Well, what does that mean?"

Kubernetes is essentially a very nice set of APIs for deploying, managing, and scaling applications.

The applications are packaged with Docker, and then the logic surrounding the deployment of the application is expressed using Helm templates. The templates themselves are instructions that are then run using the Kubernetes API.

There are a ton of Helm packages already created to take care of your application deployment needs!

I like to think of Kubernetes + Helm as a one stop shop for my application DevOps needs.

Fun Fact Time!

The entire container ecosystem, including Docker, has a very fun nautical theme. Docker has whales, Kubernetes has pods (of whales), and its logo looks like the steering portion of a ship, and Helm is the helm of a ship.

Aren't they cute?

[https://hub.docker.com//swarm](https://hub.docker.com//swarm)

Deploying an Application on Kubernetes

First of all, no matter where you deploy an application there are going to be some things that remain the same anywhere, and I do mean anywhere! ;-) Whether you are deploying to your laptop, a remote server, an AWS EC2 instance, High Performance Computing systems, or Kubernetes, the underlying concepts do not change.

I think of pretty much everything, tech concepts in particular, as a series of layers. Once you understand what those fundamental layers are you can get cooking.

Application Layers

These are:

Let's take these on one at a time.

Data Layer / Persistent Volume Claims (PVCs)

This is nice and straight forward. When you need to persist data you persist it to a filesystem. This can be local storage or some sort of networked file system (NFS). If you are using a database the database also eventually persists to a filesystem.

Application Layer / Pods

The application layer is what we typically think of in a deployment. Its the part we apt-get install, npm run or docker run. An application could be an NGINX web server, a Python or Node.js app, or a Spark application to name a few.

Applications are either Kubernetes Deployments or Stateful Set, depending on whether or not they persist data (or have a state).

A MySQL database would be an example of a Stateful application. It needs to keep track of information about itself.

An NGINX server would be a Kubernetes Deployment, because it does not need to keep track of any information about itself — it is stateless.

Services Layer / SVC

The services layer is where we expose our Application to the outside world. This is generally accomplished by saying "Hey, I have an app running on this port". You might have run these directly, or done something like a proxy pass in NGINX or Apache.

Site Reliability Layers

Site reliability is our ability to confidently say our application is up, running, and will probably stay up and running!

To be real, we want an API to essentially do this. ?

XKCD - Hard Reboot

Monitoring Layer / Metrics Server

The monitoring layer answers the question "How is our app doing"? Ideally it would answer questions like "How much CPU is left on that machine?" and "Are we out of memory yet"?

Scaling Layer / HPA

Have you ever had an application that worked great until too many people started using it at once? You take care of this by scaling the instances of your application up or down.

With web applications, you'll often see the term load balancing, too. This functionality is built into many process managers and https servers such as PM2 and Gunicorn.

In Kubernetes you accomplish this with a Horizontal Pod Autoscaler, or HPA, which you give specific rules for how to scale up or down.

Service Rules Layer

Have you ever wanted to automate when / how your application should restart? Maybe you want it to restart 3 times and then give up. Or maybe you want it to restart, but not right away.

Give it some time! You may also want some objective measure to test whether or not you application is up and running.

Deploying Applications on Kubernetes

Kubernetes applications can be deployed either through the CLI, or by writing YAML templates that describe the various PVCs, Pods (whether they are Deployments or Stateful sets), and Service (SVC) layers.

The Helm Package Manager

Helm is the best way to find, share, and use software built for Kubernetes.
https://helm.sh/

The Helm package manager allows us to wire up complex Kubernetes deployments into a single package, that can be installed with a single command.

Helm uses a templating language on top of Kubernetes YAML definitions to allow more versatility to our deployments.

Probably the most important point to note with Helm is that it has been widely accepted by the community. This means that there are lots of resources for using Helm, getting started, and also oodles of preconfigured Helm Charts!

It's very rare for me to ever have to create a Helm package completely from scratch. I can nearly always find a good starting point from one or more of the Helm charts that are already available.

Deploy NGINX on Kubernetes

First, let's talk about a base NGINX deployment without Helm.

As you can see, there is a lot of stuff to keep track of and we probably wouldn't be typing this by hand. That is where the Helm package manager comes in, but it's good to take a look to know what is happening first! ;-)

# Source: nginx/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  labels:
    app.kubernetes.io/name: nginx
    helm.sh/chart: nginx-6.0.1
    app.kubernetes.io/instance: nginx
    app.kubernetes.io/managed-by: Helm
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: nginx
      app.kubernetes.io/instance: nginx
  replicas: 1
  template:
    metadata:
      labels:
        app.kubernetes.io/name: nginx
        helm.sh/chart: nginx-6.0.1
        app.kubernetes.io/instance: nginx
        app.kubernetes.io/managed-by: Helm
    spec:      
      containers:
        - name: nginx
          image: docker.io/bitnami/nginx:1.19.0-debian-10-r2
          imagePullPolicy: "IfNotPresent"
          ports:
            - name: http
              containerPort: 8080

          livenessProbe:
            failureThreshold: 6
            initialDelaySeconds: 30
            tcpSocket:
              port: http
            timeoutSeconds: 5
          readinessProbe:
            initialDelaySeconds: 5
            periodSeconds: 5
            tcpSocket:
              port: http
            timeoutSeconds: 3

Now let's break down the different parts of the Kubernetes Deployment definition.

Metadata

I want to very briefly touch on the labels. Only briefly, because chances are you will be fine with the defaults and not need to touch them.

One of the goals of Kubernetes is that it should abstract the actual physical server away. You shouldn't usually have to care if your app is running on node1 or node2. Of course at some point you do care, and then you will start to get into labels.

Until then, don't worry about them and just stick with the defaults.

Containers

This is the part of the application that will be most relevant to you when you are deploying applications. You need to define your containers.

A single Deployment Pod can have many containers. That container has, at the very least a name, a repo, and a tag:

      containers:
        - name: nginx
          #image:  "repo:tag"
          image: docker.io/bitnami/nginx:1.19.0-debian-10-r2

Once you have the base you need to define the ports that will get picked up by the service. See that separation of concerns?:

          ports:
            - name: http
              containerPort: 8080

App Rules

Then, at some point, we want to know if our app is running. We can even determine exactly where it is in it's lifecycle with the various hooks:

         livenessProbe:
            failureThreshold: 6
            initialDelaySeconds: 30
            tcpSocket:
              # This corresponds to the ports[0].name
              port: http
            timeoutSeconds: 5
          readinessProbe:
            initialDelaySeconds: 5
            periodSeconds: 5
            # This corresponds to the ports[0].name
            tcpSocket:
              port: http
            timeoutSeconds: 3

Names

This is more of a general concept, but I do want to point out that giving things names is very important in the Kubernetes ecosystem. Notice that we gave our container and port a name. Later on when we need to refer to them we use that name.

Deploy NGINX on Kubernetes with a Helm Chart

##

The Helm package manager creates a series of templates that can be modified through the Helm CLI. Each of these templates corresponds to one of our Kubernetes types that we discussed earlier.

Here is an example of the bitnami/nginx helm chart:

Bitnami/NGINX Helm Chart Templates

Here is that same block with the Helm templating language. For the sake of brevity I have omitted some parts of the template. If you would like to see the whole thing you can take a look at it in the GitHub repo.

(This is for demonstration purposes and is not a fully functional Helm chart. Please don't use this. Grab the actual Helm chart instead.)

# Source: nginx/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ template "nginx.fullname" . }}
  labels: {{- include "nginx.labels" . | nindent 4 }}
spec:
  selector:
    matchLabels: {{- include "nginx.matchLabels" . | nindent 6 }}
  replicas: {{ .Values.replicaCount }}
  template:
    metadata:
      labels: {{- include "nginx.labels" . | nindent 8 }}
      # Omitted the annotation labels!
    spec:
      containers:
        - name: nginx
          image: {{ template "nginx.image" . }}
          imagePullPolicy: {{ .Values.image.pullPolicy | quote }}
          ports:
            - name: http
              containerPort: {{ .Values.containerPort }}
            {{ if .Values.containerTlsPort }}
            - name: https
              containerPort: {{ .Values.containerTlsPort }}
            {{ end }}
          {{- if .Values.livenessProbe }}
          livenessProbe: {{- toYaml .Values.livenessProbe | nindent 12 }}
          {{- end }}
          {{- if .Values.readinessProbe }}
          readinessProbe: {{- toYaml .Values.readinessProbe | nindent 12 }}
          {{- end }}
          {{- if .Values.resources }}
          resources: {{- toYaml .Values.resources | nindent 12 }}
          {{- end }}

Where do the Helm Template values come from?

Now, this is what I really like about Helm. The values that are exposed in the template come from one of two places.

Templated Functions

They come from the template itself, as shown here.

{{ template "nginx.fullname" . }}

We can find that this is defined in our templates/_helpers.tpl, which is a way of getting more complex functions that we could get with just a yaml file.

# templates/_helpers.tpl
{{/*
Create a default fully qualified app name.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
*/}}
# Here is the nginx.fullname
{{- define "nginx.fullname" -}}
{{- if .Values.fullnameOverride -}}
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" -}}
{{- else -}}
{{- $name := default .Chart.Name .Values.nameOverride -}}
{{- if contains $name .Release.Name -}}
{{- .Release.Name | trunc 63 | trimSuffix "-" -}}
{{- else -}}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{- end -}}

Values exposed in the Values.yaml

This is actually a very neat feature and what makes Helm so powerful and configurable.

Each Helm chart comes along with a values.yaml. You can put whatever you want in the values.yaml, and then use it throughout your Helm chart, and it's exposed through the CLI!

## Bitnami NGINX image version
## ref: https://hub.docker.com/r/bitnami/nginx/tags/
##
image:
  registry: docker.io
  repository: bitnami/nginx
  tag: 1.19.1-debian-10-r0
  ## Specify a imagePullPolicy
  ## Defaults to 'Always' if image tag is 'latest', else set to 'IfNotPresent'
  ## ref: http://kubernetes.io/docs/user-guide/images/#pre-pulling-images
  ##
  pullPolicy: IfNotPresent

Then we see this referred to in our templates as:

# templates/deployment.yaml 

# ...
      containers:
        - name: nginx
          image: {{ template "nginx.image" . }}
          imagePullPolicy: {{ .Values.image.pullPolicy | quote }}

Everything in the values.yaml can also be modified through the Helm CLI:

helm upgrade --install nginx bitnami/nginx \
    --set image.tag="my-new-tag"

It would then render the containers[0].image as image: docker.io/bitnami/nginx:my-new-tag

Wrap Up

That's it! I hope that you've learned a little about Kubernetes and its package manager Helm. Hopefully it isn't quite as scary as it once was.

I asked myself this question when I started to learn Kubernetes: how can we simplify the deployment of containerized apps? At first look, deploying containerized apps seems pretty simple. All you need is a bunch of YAML files, and by using [kubectl](https://kubernetes.io/docs/reference/generated/kubectl/kubectl/) (the Kubernetes command line utility), you’ll have your service up and running in your Kubernetes cluster.

But, Although deploying one app is an easy task, how do you deploy hundreds of apps? At Soluto, we have more than 100 live microservices and this number keeps growing. So as we started thinking about shifting workload to our Kubernetes cluster, we faced a few challenges:

First, Kubernetes deployment is actually more complex. There are many moving parts that you need to set up correctly: pod autoscaler, pod resources, ingress, and so on. Those parts require some experience with how Kubernetes works, and not setting it up correctly might cause issues in production. Ideally we’d have a way to simplify this, so developers can focus on writing their code and worry less about deployment.

Second, security is also a challenge. All services in production should have certain things, like Transport Layer Security (TLS). These are not necessarily complex things, but need to be taken care of nonetheless. We would like to pre-configure them so that any new deployment will be secured by default.

Finding a solution

To solve these challenges and speed up and ease the adoption process, we looked for a way to create a template for Kubernetes. Something that any developer would be able to use, and would require just a few parameters (for example, the Docker image of the service) to get the service up and running in production.

On the other hand, we needed to be careful not to hide too much — developers must be able to understand what’s going on so they can handle production issues. We had to find the right level of abstraction that makes it easier to deploy to Kubernetes, without hiding too much detail.

With that in mind, we started to look for a solution. After trying a few things, we found Helm. Helm is a package manager for Kubernetes.You can use it to install any app on your cluster, and Helm will take care of getting all the required configuration files and installing them on your cluster. Helm also support updating deployments, rollbacks, and many other cool features. Each Helm package is called a “Chart”, and the charts are stored in a repository. With helm, installing Mongo, for instance, is as easy as helm install stable/mongodb.

Sound like an excellent solution! We can define a chart for each type of service — like one for all our web APIs, which will handle things like load balancer and TLS — and the developer simply needs to specify the required parameters using Helm’s configuration files.

Me and my teammate, when we found Helm

Helm: Let’s see how it’s done

In order to use Helm, we first need to install it. Helm has two components: Helm client running on your computer, and Tiller, a server-side component running on your cluster. Then, we need to create the chart by using this simple Helm CLI command: helm create web-api

After running this command, you’ll notice the creation of a new folder named “web-api”. Within this folder, you’ll find all the familiar Kubernetes configuration files: deployment, service, ingress, and so on. Now it’s time to customize a bit: we can add a horizontal pod autoscaler, define the default resources the pod requires, and of course, enable TLS by default.

Everything is highly customizable based on the Go templating mechanism. So anything we add can be overridden later by the developer, in case the default configuration doesn’t work as needed.

So now we have a chart — but how can we consume it? The chart has to exist in a Helm repository, which is basically a server with a few Zip archives (which are all the charts in the repository) and one index file that is consumed by the CLI. You can manually set up your repo using any storage service like Azure Blob or AWS S3, but the simplest option is Chart Museum.

Chart museum is a Helm repository with a CRUD API to manage your charts. It supports basic authentication so you can restrict who can push new charts to your Helm repository. Helm doesn’t supply any museum-as-a-service solution, so you’re gonna have to roll your own. But it’s dead simple — simply use its docker image.

Now we can build a CI/CD pipeline for our web-api chart, to ease the process of modifying it:

• Run some kind of tests, to make sure that the new version is not broken. I’ll discuss how in the next paragraph.
• Pack the new chart, using Helm CLI.
• Push the new package to our Chart Museum instance, using Chart Museum’s API.

Testing it out

Now our chart is ready to be used by developers! But wait… how can we know that the chart actually works? And how can we make sure it will continue to work? This is why we need to write tests for our chart (like we write for anything else).

There are basically two things we want to test.

1. We want to test our template — for example, if an ingress is supposed to exist with a TLS and specific rules (defined by the developer), we should test the generated template and make sure the ingress was created correctly.
2. We want to test that the files are valid Kubernetes configurations and that they work as expected.

Testing the first thing is relatively simple — check out this sample repo to see just how simple it is.

This allows us to test the generated Kubernetes files, by using [kubetest](https://github.com/garethr/kubetest). This is great, but can be complex and messy, especially when having a lot of branching in your template files.

A better solution is required — one that will allow us to do unit testing for the templates, without generating Kubernetes files. This wasn’t a problem until recently when we started to have a lot of branching in our templates, and we’re now looking for options.

The second thing — testing that Kubernetes files are valid — is a bit more tricky. For now, at Soluto we’re using Helm’s version mechanism: each chart has a version, and all of our services will use the latest stable version. When a new chart version is pushed, we can test this version on a specific service. If it works correctly, update the rest of the services. Another option is to test that using minikube, but it was too complex for our needs.

Finally: Deploying!

So now we have a CI/CD pipeline for our Helm charts, and we have prepared a Helm chart that the developers can use. Now, when a new developer wants to deploy a new service to production, all they have to do is:

1. Add our repository to their local Helm using helm repo add chartmuseum http://<chart-museum-url>
2. Create a new Helm config file and specify the required parameters (for example, docker image of the service)
   3. Run helm upgrade —install <service-name> chartmusuem/web-api -f<path_to_config_file> and that’s it. The service is alive.

And to make it even easier to understand, I’ve created a sample repository. The repository contains all the things that I’ve discussed in this blog post: a generic chart for a web app that can be deployed with this chart and chart museum. Check it out to better understand what’s going on — and there is even a walkthrough to help you get started.

Thank you for reading along. If you have any questions, or you need help getting started with Helm, feel free to reach out either via the comments here or via Twitter.

Happy Helming!

Originally posted on Soluto’s Blog