CI/CD is a set of practices and tools designed to automate and streamline the process of integrating code changes, testing them, and deploying them to production. By adopting CI/CD, your team can reduce manual errors, speed up release cycles, and ensure that your code is always in a deployable state.

In this tutorial, we’ll focus on a beginner-friendly approach to setting up a basic CI/CD pipeline using Bitbucket, a Linux server, and Python with Flask. Specifically, we’ll create an automated process that pulls the latest changes from a Bitbucket repository to your Linux server whenever there’s a push or merge to a specific branch.

This process will be powered by Bitbucket webhooks and a simple Flask-based Python server that listens for incoming webhook events and triggers the deployment.

It’s important to note that CI/CD is a vast and complex field, and this tutorial is designed to provide a foundational understanding rather than to be an exhaustive guide.

We’ll cover the basics of setting up a CI/CD pipeline using tools that are accessible to beginners. Just keep in mind that real-world CI/CD systems often involve more advanced tools and configurations, such as containerization, orchestration, and multi-stage testing environments.

By the end of this tutorial, you’ll have a working example of how to automate deployments using Bitbucket, Linux, and Python, which you can build upon as you grow more comfortable with CI/CD concepts.

Table of Contents:

1. Why is CI/CD Important?

2. Step 1: Set Up a Webhook in Bitbucket

3. Step 2: Set Up the Flask Listener on Your Linux Server

4. Step 3: Expose the Flask App (Optional)

5. Step 4: Test the Setup

6. Step 5: Security Considerations

7. Wrapping Up

Why is CI/CD Important?

CI/CD has become a cornerstone of modern software development for several reasons. First and foremost, it accelerates the development process. By automating repetitive tasks like testing and deployment, developers can focus more on writing code and less on manual processes. This leads to faster delivery of new features and bug fixes, which is especially important in competitive markets where speed can be a differentiator.

Another key benefit of CI/CD is reduced errors and improved reliability. Automated testing ensures that every code change is rigorously checked for issues before it’s integrated into the main codebase. This minimizes the risk of introducing bugs that could disrupt the application or require costly fixes later. Automated deployment pipelines also reduce the likelihood of human error during the release process, ensuring that deployments are consistent and predictable.

CI/CD also fosters better collaboration among team members. In traditional development workflows, integrating code changes from multiple developers can be a time-consuming and error-prone process. With CI/CD, code is integrated and tested frequently, often multiple times a day. This means that conflicts are detected and resolved early, and the codebase remains in a stable state. As a result, teams can work more efficiently and with greater confidence, even when multiple contributors are working on different parts of the project simultaneously.

Finally, CI/CD supports continuous improvement and innovation. By automating the deployment process, teams can release updates to production more frequently and with less risk. This enables them to gather feedback from users faster and iterate on their products more effectively.

What We’ll Cover in This Tutorial

In this tutorial, we’ll walk through the process of setting up a simple CI/CD pipeline that automates the deployment of code changes from a Bitbucket repository to a Linux server. Here’s what you’ll learn:

1. How to configure a Bitbucket repository to send webhook notifications whenever there’s a push or merge to a specific branch.

2. How to set up a Flask-based Python server on your Linux server to listen for incoming webhook events.

3. How to write a script that pulls the latest changes from the repository and deploys them to the server.

4. How to test and troubleshoot your automated deployment process.

By the end of this tutorial, you’ll have a working example of a basic CI/CD pipeline that you can customize and expand as needed. Let’s get started!

Step 1: Set Up a Webhook in Bitbucket

Before starting with the setup, let’s briefly explain what a webhook is and how it fits into our CI/CD process.

A webhook is a mechanism that allows one system to notify another system about an event in real-time. In the context of Bitbucket, a webhook can be configured to send an HTTP request (often a POST request with payload data) to a specified URL whenever a specific event occurs in your repository, such as a push to a branch or a pull request merge.

In our case, the webhook will notify our Flask-based Python server (running on your Linux server) whenever there’s a push or merge to a specific branch. This notification will trigger a script on the server to pull the latest changes from the repository and deploy them automatically. Essentially, the webhook acts as the bridge between Bitbucket and your server, enabling seamless automation of the deployment process.

Now that you understand the role of a webhook, let’s set one up in Bitbucket:

1. Log in to Bitbucket and navigate to your repository.

2. On the left-hand sidebar, click on Settings.

3. Under the Workflow section, find and click on Webhooks.

4. Click the Add webhook button.

5. Enter a name for your webhook (for example, "Automatic Pull").

6. In the URL field, provide the URL to your server where the webhook will send the request. If you’re running a Flask app locally, this would be something like http://your-server-ip/pull-repo. (For production environments, it’s highly recommended to use HTTPS to secure the communication between Bitbucket and your server.)

7. In the Triggers section, choose the events you want to listen to. For this example, we will select Push (and optionally, Pull Request Merged if you want to deploy after merges, too).

8. Save the webhook with a self-explanatory name so it’s easy to identify later.

Once the webhook is set up, Bitbucket will send a POST request to the specified URL every time the selected event occurs. In the next steps, we’ll set up a Flask server to handle these incoming requests and trigger the deployment process.

Here is what you should see when you setup up the Bitbucket webhook

Step 2: Set Up the Flask Listener on Your Linux Server

In the next step, you’ll set up a simple web server on your Linux machine that will listen for the webhook from Bitbucket. When it receives the notification, it will execute a git pull or a force pull (in case of local changes) to update the repository.

Install Flask:

To create the Flask application, first install Flask by running:

pip install flask

Create the Flask App:

Create a new Python script (for example, app_repo_pull.py) on your server and add the following code:

from flask import Flask
import subprocess

app = Flask(__name__)

@app.route('/pull-repo', methods=['POST'])
def pull_repo():
    try:
        # Fetch the latest changes from the remote repository
        subprocess.run(["git", "-C", "/path/to/your/repository", "fetch"], check=True)
        # Force reset the local branch to match the remote 'test' branch
        subprocess.run(["git", "-C", "/path/to/your/repository", "reset", "--hard", "origin/test"], check=True)  # Replace 'test' with your branch name
        return "Force pull successful", 200
    except subprocess.CalledProcessError:
        return "Failed to force pull the repository", 500

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=5000)

Here’s what this code does:

• subprocess.run(["git", "-C", "/path/to/your/repository", "fetch"]): This command fetches the latest changes from the remote repository without affecting the local working directory.

• subprocess.run(["git", "-C", "/path/to/your/repository", "reset", "--hard", "origin/test"]): This command performs a hard reset, forcing the local repository to match the remote test branch. Replace test with the name of your branch.

Make sure to replace /path/to/your/repository with the actual path to your local Git repository.

Step 3: Expose the Flask App (Optional)

If you want the Flask app to be accessible from outside your server, you need to expose it publicly. For this, you can set up a reverse proxy with NGINX. Here's how to do that:

First, install NGINX if you don't have it already by running this command:

sudo apt-get install nginx

Next, you’ll need to configure NGINX to proxy requests to your Flask app. Open the NGINX configuration file:

sudo nano /etc/nginx/sites-available/default

Modify the configuration to include this block:

server {
    listen 80;
    server_name your-server-ip;

    location /pull-repo {
        proxy_pass http://localhost:5000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Now just reload NGINX to apply the changes:

sudo systemctl reload nginx

Step 4: Test the Setup

Now that everything is set up, go ahead and start the Flask app by executing this Python script:

python3 app_repo_pull.py

Now to test if everything is working:

1. Make a commit: Push a commit to the test branch in your Bitbucket repository. This action will trigger the webhook.

1. Webhook trigger: The webhook will send a POST request to your server. The Flask app will receive this request, perform a force pull from the test branch, and update the local repository.

2. Verify the pull: Check the log output of your Flask app or inspect the local repository to verify that the changes have been pulled and applied successfully.

Step 5: Security Considerations

When exposing a Flask app to the internet, securing your server and application is crucial to protect it from unauthorized access, data breaches, and attacks. Here are the key areas to focus on:

1. Use a Secure Server with Proper Firewall Rules

A secure server is one that is configured to minimize exposure to external threats. This involves using firewall rules, minimizing unnecessary services, and ensuring that only required ports are open for communication.

Example of a secure server setup:

• Minimal software: Only install the software you need (for example, Python, Flask, NGINX) and remove unnecessary services.

• Operating system updates: Ensure your server's operating system is up-to-date with the latest security patches.

• Firewall configuration: Use a firewall to control incoming and outgoing traffic and limit access to your server.

For example, a basic UFW (Uncomplicated Firewall) configuration on Ubuntu might look like this:

# Allow SSH (port 22) for remote access
sudo ufw allow ssh

# Allow HTTP (port 80) and HTTPS (port 443) for web traffic
sudo ufw allow http
sudo ufw allow https

# Enable the firewall
sudo ufw enable

# Check the status of the firewall
sudo ufw status

In this case:

• The firewall allows incoming SSH connections on port 22, HTTP on port 80, and HTTPS on port 443.

• Any unnecessary ports or services should be blocked by default to limit exposure to attacks.

Additional Firewall Rules:

• Limit access to webhook endpoint: Ideally, only allow traffic to the webhook endpoint from Bitbucket's IP addresses to prevent external access. You can set this up in your firewall or using your web server (for example, NGINX) by only accepting requests from Bitbucket's IP range.

• Deny all other incoming traffic: For any service that does not need to be exposed to the internet (for example, database ports), ensure those ports are blocked.

2. Add Authentication to the Flask App

Since your Flask app will be publicly accessible via the webhook URL, you should consider adding authentication to ensure only authorized users (such as Bitbucket's servers) can trigger the pull.

Basic Authentication Example:

You can use a simple token-based authentication to secure your webhook endpoint. Here’s an example of how to modify your Flask app to require an authentication token:

from flask import Flask, request, abort
import subprocess

app = Flask(__name__)

# Define a secret token for webhook verification
SECRET_TOKEN = 'your-secret-token'

@app.route('/pull-repo', methods=['POST'])
def pull_repo():
    # Check if the request contains the correct token
    token = request.headers.get('X-Hub-Signature')
    if token != SECRET_TOKEN:
        abort(403)  # Forbidden if the token is incorrect

    try:
        subprocess.run(["git", "-C", "/path/to/your/repository", "fetch"], check=True)
        subprocess.run(["git", "-C", "/path/to/your/repository", "reset", "--hard", "origin/test"], check=True)
        return "Force pull successful", 200
    except subprocess.CalledProcessError:
        return "Failed to force pull the repository", 500

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=5000)

How it works:

• The X-Hub-Signature is a custom header that you add to the request when setting up the webhook in Bitbucket.

• Only requests with the correct token will be allowed to trigger the pull. If the token is missing or incorrect, the request is rejected with a 403 Forbidden response.

You can also use more complex forms of authentication, such as OAuth or HMAC (Hash-based Message Authentication Code), but this simple token approach works for many cases.

3. Use HTTPS for Secure Communication

It’s crucial to encrypt the data transmitted between your Flask app and the Bitbucket webhook, as well as any sensitive data (such as tokens or passwords) being transmitted over the network. This ensures that attackers cannot intercept or modify the data.

Why HTTPS?

• Data encryption: HTTPS encrypts the communication, ensuring that sensitive data like your authentication token is not exposed to man-in-the-middle attacks.

• Trust and integrity: HTTPS helps ensure that the data received by your server hasn’t been tampered with.

Using Let’s Encrypt to Secure Your Flask App with SSL:

1. Install Certbot (the tool for obtaining Let’s Encrypt certificates):

sudo apt-get update
sudo apt-get install certbot python3-certbot-nginx

Obtain a free SSL certificate for your domain:

sudo certbot --nginx -d your-domain.com

• This command will automatically configure Nginx to use HTTPS with a free SSL certificate from Let’s Encrypt.

• Ensure HTTPS is used: Make sure that your Flask app or Nginx configuration forces all traffic to use HTTPS. You can do this by setting up a redirection rule in Nginx:

server {
    listen 80;
    server_name your-domain.com;

    # Redirect HTTP to HTTPS
    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl;
    server_name your-domain.com;

    ssl_certificate /etc/letsencrypt/live/your-domain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/your-domain.com/privkey.pem;

    # Other Nginx configuration...
}

Automatic Renewal: Let’s Encrypt certificates are valid for 90 days, so it’s important to set up automatic renewal:

sudo certbot renew --dry-run

This command tests the renewal process to make sure everything is working.

4. Logging and Monitoring

Implement logging and monitoring for your Flask app to track any unauthorized attempts, errors, or unusual activity:

• Log requests: Log all incoming requests, including the IP address, request headers, and response status, so you can monitor for any suspicious activity.

• Use monitoring tools: Set up tools like Prometheus, Grafana, or New Relic to monitor server performance and app health.

Wrapping Up

In this tutorial, we explored how to set up a simple, beginner-friendly CI/CD pipeline that automates deployments using Bitbucket, a Linux server, and Python with Flask. Here’s a recap of what you’ve learned:

1. CI/CD Fundamentals: We discussed the basics of Continuous Integration (CI) and Continuous Delivery/Deployment (CD), which are essential practices for automating the integration, testing, and deployment of code. You learned how CI/CD helps speed up development, reduce errors, and improve collaboration among developers.

2. Setting Up Bitbucket Webhooks: You learned how to configure a Bitbucket webhook to notify your server whenever there’s a push or merge to a specific branch. This webhook serves as a trigger to initiate the deployment process automatically.

3. Creating a Flask-based Webhook Listener: We showed you how to set up a Flask app on your Linux server to listen for incoming webhook requests from Bitbucket. This Flask app receives the notifications and runs the necessary Git commands to pull and deploy the latest changes.

4. Automating the Deployment Process: Using Python and Flask, we automated the process of pulling changes from the Bitbucket repository and performing a force pull to ensure the latest code is deployed. You also learned how to configure the server to expose the Flask app and accept requests securely.

5. Security Considerations: We covered critical security steps to protect your deployment process:

   • Firewall Rules: We discussed configuring firewall rules to limit exposure and ensure only authorized traffic (from Bitbucket) can access your server.

   • Authentication: We added token-based authentication to ensure only authorized requests can trigger deployments.

   • HTTPS: We explained how to secure the communication between your server and Bitbucket using SSL certificates from Let's Encrypt.

   • Logging and Monitoring: Lastly, we recommended setting up logging and monitoring to keep track of any unusual activity or errors.

Next Steps

By the end of this tutorial, you now have a working example of an automated deployment pipeline. While this is a basic implementation, it serves as a foundation you can build on. As you grow more comfortable with CI/CD, you can explore advanced topics like:

• Multi-stage deployment pipelines

• Integration with containerization tools like Docker

• More complex testing and deployment strategies

• Use of orchestration tools like Kubernetes for scaling

CI/CD practices are continually evolving, and by mastering the basics, you’ve set yourself up for success as you expand your skills in this area. Happy automating and thank you for reading!

You can fork the code from here.

In this article, I want to show another alternative in case you use GitHub Actions as your CI platform (the most popular CI/CD solution at the moment). This alternative is called Service Containers, and I’ve realized that not many developers seem to know about it.

In this hands-on tutorial, I’ll demonstrate how to create a GitHub Actions workflow for integration tests with external dependencies (MongoDB and Redis) using the demo Go application we created in that previous tutorial. We’ll also review the pros and cons of GitHub Service Containers.

Prerequisites

• A basic understanding of GitHub Actions workflows.

• Familiarity with Docker containers.

• Basic knowledge of Go toolchain.

Table of Contents

• What are Service Containers?

• Why not Docker Compose?

• Job Runtime

• Readiness Healthcheck

• Private Container Registries

• Sharing Data Between Services

• Golang Integration Tests

• Personal Experience & Limitations

• Conclusion

• Resources

What are Service Containers?

Service Containers are Docker containers that offer a simple and portable way to host dependencies like databases (MongoDB in our example), web services, or caching systems (Redis in our example) that your application needs within a workflow.

This article focuses on integration tests, but there are many other possible applications for service containers. For example, you can also use them to run supporting tools required by your workflow, such as code analysis tools, linters, or security scanners.

Why Not Docker Compose?

Sounds similar to services in Docker Compose, right? Well, that’s because it is.

But while you could technically use Docker Compose within a GitHub Actions workflow by installing Docker Compose and running docker-compose up, service containers provide a more integrated and streamlined approach that’s specifically designed for the GitHub Actions environment.

Also, while they are similar, they solve different problems and have different general purposes:

• Docker Compose is good when you need to manage a multi-container application on your local machine or a single server. It’s best suited for long-living environments.

• Service Containers are ephemeral and exist only for the duration of a workflow run, and they’re defined directly within your GitHub Actions workflow file.

Just keep in mind that the feature set of service containers (at least as of now) is more limited compared to Docker Compose, so be ready to discover some potential bottlenecks. We will cover some of them at the end of this article.

Job Runtime

You can run GitHub jobs directly on a runner machine or in a Docker container (by specifying the container property). The second option simplifies the access to your services by using labels you define in the services section.

To run directly on a runner machine:

.github/workflows/test.yaml

jobs:
  integration-tests:
    runs-on: ubuntu-24.04

    services:
      mongo:
        image: mongodb/mongodb-community-server:7.0-ubi8
        ports:
          - 27017:27017

    steps:
      - run: |
          echo "addr 127.0.0.1:27017"

Or you can run it in a container (Chainguard Go Image in our case):

jobs:
  integration-tests:
    runs-on: ubuntu-24.04
    container: cgr.dev/chainguard/go:latest

    services:
      mongo:
        image: mongodb/mongodb-community-server:7.0-ubi8
        ports:
          - 27017:27017
    steps:
      - run: |
          echo "addr mongo:27017"

You can also omit the host port, so the container port will be randomly assigned to a free port on the host. You can then access the port using the variable.

Benefits of omitting the host port:

• Avoids port conflicts – for example when you run many services on the same host.

• Enhances Portability – your configurations become less dependent on the specific host environment.

jobs:
  integration-tests:
    runs-on: ubuntu-24.04
    container: cgr.dev/chainguard/go:1.23

    services:
      mongo:
        image: mongodb/mongodb-community-server:7.0-ubi8
        ports:
          - 27017/tcp
    steps:
      - run: |
          echo "addr mongo:${{ job.services.mongo.ports['27017'] }}"

Of course, there are pros and cons to each approach.

Running in a container:

• Pros: Simplified network access (use labels as hostnames), and automatic port exposure within the container network. You also get better isolation/security as the job runs in an isolated environment.

• Cons: Implied overhead of containerization.

Running on the runner machine:

• Pros: Potentially less overhead than running the job inside a container.

• Cons: Requires manual port mapping for service container access (using localhost:). There’s also less isolation/security, as the job runs directly on the runner machine. This potentially affects other jobs or the runner itself if something goes wrong.

Readiness Healthcheck

Prior to running the integration tests that connect to your provisioned containers, you’ll often need to make sure that the services are ready. You can do this by specifying docker create options such as health-cmd.

This is very important – otherwise the services may not be ready when you start accessing them.

In the case of MongoDB and Redis, these will be the following:

    services:
      mongo:
        image: mongodb/mongodb-community-server:7.0-ubi8
        ports:
          - 27017/27017
        options: >-
          --health-cmd "echo 'db.runCommand("ping").ok' | mongosh mongodb://localhost:27017/test --quiet"
          --health-interval 5s
          --health-timeout 10s
          --health-retries 10

      redis:
        image: redis:7
        ports:
          - 6379:6379
        options: >-
          --health-cmd "redis-cli ping"
          --health-interval 5s
          --health-timeout 10s
          --health-retries 10

In the Action logs, you can see the readiness status:

Private Container Registries

In our example, we’re using public images from Dockerhub, but it’s possible to use private images from you private registries as well, such as Amazon Elastic Container Registry (ECR), Google Artifact Registry, and so on.

Make sure to store the credentials in Secrets and then reference them in the credentials section.

services:
  private_service:
    image: ghcr.io/org/service_repo
    credentials:
      username: ${{ secrets.registry_username }}
      password: ${{ secrets.registry_token }}

Sharing Data Between Services

You can use volumes to share data between services or other steps in a job. You can specify named Docker volumes, anonymous Docker volumes, or bind mounts on the host. But it’s not directly possible to mount the source code as a container volume. You can refer to this open discussion for more context.

To specify a volume, you specify the source and destination path: <source>:<destinationPath>

The <source> is a volume name or an absolute path on the host machine, and <destinationPath> is an absolute path in the container.

volumes:
  - /src/dir:/dst/dir

Volumes in Docker (and GitHub Actions using Docker) provide persistent data storage and sharing between containers or job steps, decoupling data from container images.

Project Setup

Before diving into the full source code, let's set up our project for running integration tests with GitHub Service Containers.

1. Create a new GitHub repository.

2. Initialize a Go module using go mod init

3. Create a simple Go application.

4. Add integration tests in integration_test.go

5. Create a .github/workflows directory.

6. Create a file named integration-tests.yaml inside the .github/workflows directory.

Golang Integration Tests

Now as we can provision our external dependencies, let’s have a look at how to run our integration tests in Go. We will do it in the steps section of our workflow file.

We will run our tests in a container which uses Chainguard Go image. This means we don’t have to install/setup Go. If you want to run your tests directly on a runner machine, you need to use the setup-go Action.

You can find the full source code with tests and this workflow here.

.github/workflows/integration-tests.yaml

name: "integration-tests"

on:
  workflow_dispatch:
  push:
    branches:
      - main

jobs:
  integration-tests:
    runs-on: ubuntu-24.04
    container: cgr.dev/chainguard/go:latest

    env:
      MONGO_URI: mongodb://mongo:27017
      REDIS_URI: redis://redis:6379

    services:
      mongo:
        image: mongodb/mongodb-community-server:7.0-ubi8
        ports:
          - 27017:27017
        options: >-
          --health-cmd "echo 'db.runCommand("ping").ok' | mongosh mongodb://localhost:27017/test --quiet"
          --health-interval 5s
          --health-timeout 10s
          --health-retries 10

      redis:
        image: redis:7
        ports:
          - 6379:6379
        options: >-
          --health-cmd "redis-cli ping"
          --health-interval 5s
          --health-timeout 10s
          --health-retries 10

    steps:
      - name: Check out repository code
        uses: actions/checkout@v4

      - name: Download dependencies
        run: go mod download

      - name: Run Integration Tests
        run: go test -tags=integration -timeout=120s -v ./...

To summarize what’s going on here:

1. We run our job in a container with Go (container)

2. We spin up two services: MongoDB and Redis (services)

3. We configure healthchecks to make sure our services are “Healthy” when we run the tests (options)

4. We perform a standard code checkout

5. Then we run the Go tests

Once the Action is completed (it took ~1 min for this example), all the services will be stopped and orphaned so we don’t need to worry about that.

Personal Experience & Limitations

We’ve been using service containers for running backend integration tests at BINARLY for some time, and they work great. But the initial workflow creation took some time and we encountered the following bottlenecks:

• It’s not possible to override or run custom commands in an action service container (as you would do in Docker Compose using the command property). Open pull request

  • Workaround: we had to find a solution that doesn’t require that. In our case, we were lucky and could do the same with environment variables.

• It’s not directly possible to mount the source code as a container volume. Open discussion

  • While this is indeed a big limitation, you can copy the code from your repository into your mounted directory after the service container has started.

Conclusion

GitHub service containers are a great option to scaffold an ephemeral testing environment by configuring it directly in your GitHub workflow. With configuration being somewhat similar to Docker Compose, it’s easy to run any containerised application and communication with it in your pipeline. This ensures that GitHub runners take care of shutting everything down upon completion.

If you use Github Actions, this approach works extremely well as it is specifically designed for the GitHub Actions environment.

Resources

• Source Code

• GitHub Documentation

• Discover more articles on packagemain.tech

These terms might seem complex or interchangeable at first glance, leaving you wondering: What do they actually mean? How do they differ from one another? 🤔

In this handbook, I’ll break down these concepts in a clear and approachable way, drawing on relatable analogies to make each term easier to understand. 🧠💡 Beyond just theory, we’ll dive into a hands-on tutorial where you’ll learn how to set up a CI/CD workflow step by step.

Together, we’ll:

• Set up a Node.js project. ✨

• Implement automated tests using Jest and Supertest. 🛠️

• Set up a CI/CD workflow using GitHub Actions, triggered on push, and pull requests, or after a new release. ⚙️

• Build and publish a Docker image of your application to Docker Hub. 📦

• Deploy your application to a staging environment for testing. 🚀

• Finally, roll it out to a production environment, making it live! 🌐

By the end of this guide, not only will you understand the difference between CI/CD concepts, but you’ll also have practical experience in building your own automated pipeline. 😃

Table of Contents

1. What is Continuous Integration, Deployment, and Delivery?

2. Differences Between Continuous Integration, Continuous Delivery, and Continuous Deployment

3. How to Set Up a Node.js Project with a Web Server and Automated Tests

4. How to Create a GitHub Repository to Host Your Codebase

5. How to Set Up the CI and CD Workflows Within Your Project

6. Set Up a Docker Hub Repository for the Project's Image and Generate an Access Token for Publishing the Image

7. Create a Google Cloud Account, Project, and Billing Account

8. Create a Google Cloud Service Account to Enable Deployment of the Node.js Application to Google Cloud Run via the CD Pipeline

9. Create the Staging Branch and Merge the Feature Branch into It (Continuous Integration and Continuous Delivery)

10. Merge the Staging Branch into the Main Branch (Continuous Integration and Continuous Deployment)

11. Conclusion

What is Continuous Integration, Deployment, and Delivery? 🤔

Continuous Integration (CI)

Imagine you’re part of a team of six developers, all working on the same project. Without a proper system, chaos would ensue.

Let’s say Mr. A is building a new login feature, Mrs. B is fixing a bug in the search bar, and Mr. C is tweaking the dashboard UI—all at the same time. If everyone is editing the same "folder" or codebase directly, things could go horribly wrong: "Hey! Who just broke the app?!" 😱

To keep everything in order, teams use Version Control Systems (VCS) like GitHub, GitLab, or BitBucket. Think of it as a digital workspace where everyone can safely collaborate without stepping on each other’s toes. 🗂️✨

Here’s how Continuous Integration fits into this process step-by-step:

1. The Main Branch: The General Folder ✨

At the heart of every project is the main branch—the ultimate source of truth. It contains the stable codebase that powers your live app. It’s where every team member contributes their work, but with one important rule: only tested and approved code gets merged here. 🚀

2. Feature Branches: Personal Workspaces 🔨

When someone like Mr. A wants to work on a new feature, they create a feature branch. This branch is essentially a personal copy of the main branch where they can tinker, write code, and test without affecting others. Mrs. B and Mr. C are also working on their own branches. Everyone’s experiments stay neatly organized. 🧪💡

3. Merging Changes: The CI Workflow 🎉

When Mr. A is satisfied with his feature, he doesn’t just shove it into the main branch—CI ensures it’s done safely:

• Automated Tests: Before merging, CI tools automatically run tests on Mr. A’s code to check for bugs or errors. Think of it as a bouncer guarding the main branch, ensuring no bad code gets in. 🕵️‍♂️

• Build Verification: The feature branch code is also "built" (converted into a deployable version of the app) to confirm it works as intended.

Once these checks are passed, Mr. A’s feature branch is merged into the main branch. This frequent merging of changes is what we call Continuous Integration.

Continuous Delivery (CD)

Continuous Delivery (CD) often gets mixed up with Continuous Deployment, and while they share similarities, they serve distinct purposes in the development lifecycle. Let’s break it down! 🧐

The Need for a Staging Area 🌉

In the Continuous Integration (CI) process we discussed above, we primarily dealt with feature branches and the main branch. But directly merging changes from feature branches into the main branch (which powers the live product) can be risky. Why? 🛑

While automated tests and builds catch many errors, they’re not foolproof. Some edge cases or bugs might slip through unnoticed. This is where the staging branch and staging environment come into play! 🎭

Think of the staging branch as a “trial run.” Before unleashing changes to real customers, the codebase from feature branches is merged into the staging branch and deployed to a staging environment. This environment is an exact replica of the production environment, but it’s used exclusively by the Quality Assurance (QA) team for testing.

The QA team takes the role of a “test driver,” running the platform through its paces just as a real user would. They check for usability issues, edge cases, or bugs that automated tests might miss, and provide feedback to developers for fixes. 🚦 If everything passes, the codebase is cleared for deployment to production.

Continuous Delivery in Action 📦

The process of merging changes into the staging branch and deploying them to the staging environment is what we call Continuous Delivery. 🛠️ It ensures that the application is always in a deployable state, ready for the next step in the pipeline.

Unlike Continuous Deployment (which we’ll discuss later), Continuous Delivery doesn’t automatically push changes to production (live platform). Instead, it pauses to let humans—namely the QA team or stakeholders—decide when to proceed. This adds an extra layer of quality assurance, reducing the chances of errors making it to the live product. 🕵️‍♂️

Continuous Deployment (CD)

Continuous Deployment (CD) takes automation to its peak. While it shares similarities with Continuous Delivery, the key difference lies in the final step: there’s no manual approval required. The final process—merging the codebase and deploying it live for end users (the QA testers or the team lead could do this).

Let’s explore what makes Continuous Deployment so powerful (and a little scary)! 😅

The Last Mile of the CI/CD Pipeline 🛣️

Imagine you’ve gone through the rigorous process of Continuous Integration: teammates have merged their feature branches, automated tests were run, and the codebase was successfully deployed to the staging environment during Continuous Delivery.

Now, you’re confident that the application is free of bugs and ready to shine in the production environment—the live version of your platform used by real customers.

In Continuous Deployment, this final step of deploying changes to the live environment happens automatically. The pipeline triggers whenever specific events occur, such as:

• A Pull Request (PR) is merged into the main branch.

• A new release version is created.

• A commit is pushed directly to the production branch (though this is rare for most teams).

Once triggered, the pipeline springs into action, building, testing, and finally deploying the updated codebase to the production environment. 📡

Differences Between Continuous Integration, Continuous Delivery, and Continuous Deployment 🔍

Now that we’ve untangled the mysteries of Continuous Integration, Continuous Delivery, and Continuous Deployment, it’s time to roll up our sleeves and put theory into practice 😁.

How to Set Up a Node.js Project with a Web Server and Automated Tests ✨

In this hands-on section, we’ll build a Node.js web server with automated tests using Jest. From there, we’ll create a CI/CD pipeline with GitHub Actions that automates testing for every pull request to the staging and main branches. Finally, we’ll publish an Image of our application to DockerHub and deploy the image to Google Cloud Run, first to a staging environment for testing and later to the production environment for live use.

Ready to bring your project to life? Let’s get started! 🚀✨

Step 1: Install Node.js 📥

To get started, you’ll need to have Node.js installed on your machine. Node.js provides the JavaScript runtime we’ll use to create our web server.

1. Visit https://nodejs.org/en/download/package-manager

2. Choose your operating system (Windows, macOS, or Linux) and download the installer.

3. Follow the installation instructions to complete the setup.

To verify that Node.js was installed successfully, open your terminal and run node -v. This should display the installed version of Node.js

Step 2: Clone the Starter Repository 📂

The next step is to grab the starter code from GitHub. If you don’t have Git installed, you can download it at https://git-scm.com/downloads. Choose your OS and follow the instructions to install Git. Once you’re set, it’s time to clone the repository.

Run the following command in your terminal to clone the boilerplate code:

git clone --single-branch --branch initial https://github.com/onukwilip/ci-cd-tutorial

This will download the project files from the initial branch, which contains the starter template for our Node.js web server.

Navigate into the project directory:

cd ci-cd-tutorial

Step 3: Install Dependencies 📦

Once you’re in the project directory, install the required dependencies for the Node.js project. These are the packages that power the application:

npm install --force

This will download and set up all the libraries specified in the project. Alright, dependencies installed? You’re one step closer!

Step 4: Run Automated Tests ✅

Before diving into the code, let’s confirm that the automated tests are functioning correctly. Run:

npm test

You should see two successful test results in your terminal. This indicates that the starter project is correctly configured with working automated tests.

Step 5: Start the Web Server 🌐

Finally, let’s start the web server and see it in action. Run the following command:

npm start

Wait for the application to start running. Open your browser and visit http://localhost:5000. 🎉 You should see the starter web server up and running, ready for your CI/CD magic:

How to Create a GitHub Repository to Host Your Codebase 📂

Step 1: Sign In to GitHub

1. Go to GitHub: Open your browser and visit GitHub - https://github.com.

2. Sign In: Click on the Sign In button in the top-right corner and enter your username and password to log in, OR create an account if you don’t have one by clicking the Sign up button.

Step 2: Create a New Repository

Once you're signed in, on the main GitHub page, you’ll see a "+" sign in the top-right corner next to your profile picture. Click on it, and select “New repository” from the dropdown.

Now it’s time to set the repository details. You’ll include:

• Repository Name: Choose a name for your repository. For example, you can call it ci-cd-tutorial.

• Description (Optional): You can add a short description, like “A tutorial project for CI/CD with Docker and GitHub Actions.”

• Visibility: Choose whether you want your repository to be public (accessible by anyone) or private (only accessible by you and those you invite). For the sake of this tutorial, make it public.

• Do Not Check the Add a README File Box: Important: Make sure you do not check the option to Add a README file. This will automatically create a README.md file in your repository, which could cause conflicts later when you push your local files. We'll add the README file manually if needed later.

After filling out the details, click on “Create repository”.

Step 3: Change the Remote Destination and Push to Your New Repository

Update the Remote Repository URL:

Since you've already cloned the codebase from my repository, you need to update the remote destination to point to your newly created GitHub repository.

Copy your repository URL (the URL of the page you were redirected to after creating the repository). It should look similar to this: https://github.com/<username>/<repo-name>.

Open your terminal in the project directory and run the following commands:

git remote set-url origin <your-repo-url>

Replace <your-repo-url> with your GitHub repository URL which you copied earlier.

Rename the Current Branch to main:

If your branch is named something other than main, you can rename it to main using:

git branch -M main

Push to Your New Repository:

Finally, commit any changes you’ve made and push your local repository to the new remote GitHub repository by running:

git add .
git commit -m 'Created boilerplate'
git push -u origin main

Now your local codebase is linked to your new GitHub repository, and the files are successfully pushed there. You can verify by visiting your repository on GitHub.

How to Set Up the CI and CD Workflows Within Your Project ⚙️

Now it’s time to create the CI and CD workflows for our project! These workflows won’t run on your local PC but will be automatically triggered and executed in the cloud once you push your changes to the remote repository. GitHub Actions will detect these workflows and run them based on the triggers you define.

Step 1: Prepare the Workflow Directory 📂

Before adding the CI/CD pipelines, it's a good practice to first create a feature branch. This step mirrors the workflow commonly used in teams, where new features or changes are made in separate branches before they are merged into the main codebase.

To create and switch to a new branch, run the following command:

git checkout -b feature/ci-cd-pipeline

This will create a new branch called feature/ci-cd-pipeline and switch to it. Now, you can safely add and test the CI/CD workflows without affecting the main branch.

Once you finish, you’ll be able to merge this feature branch back into main or staging as part of the pull request process.

In the project’s root directory, create a folder named .github. Inside .github, create another folder called workflows.

Any YAML file placed in the .github/workflows directory is automatically recognized as a GitHub Actions workflow. These workflows will execute based on specific triggers, such as pull requests, pushes, or releases.

Step 2: Create the Continuous Integration Workflow 🚀

We’ll now create a CI workflow that automatically tests the application whenever a pull request is made to the main or staging branches.

First, inside the workflows directory, create a file named ci-pipeline.yml.

Paste the following code into the file:

name: CI Pipeline to staging/production environment
on:
  pull_request:
    branches:
      - staging
      - main
jobs:
  test:
    runs-on: ubuntu-latest
    name: Setup, test, and build project
    env:
      PORT: 5001
    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Install dependencies
        run: npm ci

      - name: Test application
        run: npm test

      - name: Build application
        run: |
          echo "Run command to build the application if present"
          npm run build --if-present

Explanation of the CI Workflow

Here’s a breakdown of each section in the workflow:

1. name: CI Pipeline to staging/production environment: This is the title of your workflow. It helps you identify this pipeline in GitHub Actions.

2. on: The on parameter is what determines the events that trigger your workflow. When the workflow YAML file is pushed to the remote GitHub repository, GitHub Actions automatically registers the workflow using the configured triggers in the on field. These triggers act as event listeners that tell GitHub when to execute the workflow

   For example:

   If we set pull_request as the value for the on parameter and specify the branches we want to monitor using the branches key, GitHub sets up event listeners for pull requests to those branches.

    on:
      pull_request:
        branches:
          - main
          - staging
   

   This configuration means that GitHub will trigger the workflow whenever a pull request is made to the main or staging branches.

   Multiple Triggers:
   You can define multiple event listeners in the on parameter. For instance, in addition to pull requests, you can add a listener for push events.

    on:
      pull_request:
        branches:
          - main
          - staging
      push:
        branches:
          - main
   

   This configuration ensures that the workflow is triggered when:

   • A pull request is made to either the main or staging branch.

   • A push is made directly to the main branch.

📘 Learn more about triggers: Check out the official GitHub documentation here.

3. jobs: The jobs section outlines the specific tasks (or jobs) that the workflow will execute. Each job is an independent unit of work that runs on a separate virtual machine (VM). This isolation ensures a clean, unique environment for every job, avoiding potential conflicts between tasks.

   Key Points About Jobs:

   1. Clean VM for Each Job: When GitHub Actions runs a workflow, it assigns a dedicated VM instance to each job. This means the environment is reset for every job, ensuring there’s no overlap or interference between tasks.

   2. Multiple Jobs: Workflows can have multiple jobs, each responsible for a specific task. For example:

      • A Test job to install dependencies and run automated tests.

      • A Build job to compile the application.

   3. Job Organization: Jobs can be organized to run:

      • Sequentially: Ensures one job is completed before the next starts, for example the Test job must finish before the Build job. This sequential flow mimics the "pipeline" structure.

      • Simultaneously: Multiple jobs can run in parallel to save time, especially if the jobs are independent of one another.

   4. Single Job in This Workflow: In our current workflow, there is only one job, test, which:

      • Installs dependencies.

      • Runs automated tests.

      • Builds the application.

📘 Learn more about jobs: Dive into the GitHub Actions jobs documentation here.

4. runs-on: ubuntu-latest: Specifies the operating system the job will run on. GitHub provides pre-configured virtual environments, and we’re using the latest Ubuntu image.

5. env: Sets environment variables for the job. Here, we define the PORT variable used by our application.

6. Steps: Steps define the individual actions to execute within a job:

   • Checkout: Uses the actions/checkout action to clone the repository containing the codebase in the feature branch into the virtual machine instance environment. This step ensures the pipeline has access to the project files.

   • Install dependencies: Runs npm ci to install the required Node.js packages.

   • Test application: Runs the automated tests using the npm test command. This validates the codebase for errors or failing test cases.

   • Build application: Builds the application if a build script is defined in the package.json. The --if-present flag ensures this step doesn’t fail if no build script is present.

Now that we’ve completed the CI pipeline, which runs on pull requests to the main or staging branches, let’s move on to setting up the Continuous Delivery (CD) and Continuous Deployment pipelines. 🚀

Step 3: The Continuous Delivery and Deployment Workflow

First, create the Pipeline File:
In the .github/workflows folder, create a new file called cd-pipeline.yml. This file will define the workflows for automating delivery and deployment.

Next, paste the configuration:
Copy and paste the following configuration into the cd-pipeline.yml file:

name: CD Pipeline to Google Cloud Run (staging and production)
on:
  push:
    branches:
      - staging
  workflow_dispatch: {}
  release:
    types: published

env:
  PORT: 5001
  IMAGE: ${{vars.IMAGE}}:${{github.sha}}
jobs:
  test:
    runs-on: ubuntu-latest
    name: Setup, test, and build project
    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Install dependencies
        run: npm ci

      - name: Test application
        run: npm test
  build:
    needs: test
    runs-on: ubuntu-latest
    name: Setup project, Authorize GitHub Actions to GCP and Docker Hub, and deploy
    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Authenticate for GCP
        id: gcp-auth
        uses: google-github-actions/auth@v0
        with:
          credentials_json: ${{ secrets.GCP_SERVICE_ACCOUNT }}

      - name: Set up Cloud SDK
        uses: google-github-actions/setup-gcloud@v0

      - name: Authenticate for Docker Hub
        id: docker-auth
        env:
          D_USER: ${{secrets.DOCKER_USER}}
          D_PASS: ${{secrets.DOCKER_PASSWORD}}
        run: |
          docker login -u $D_USER -p $D_PASS
      - name: Build and tag Image
        run: |
          docker build -t ${{env.IMAGE}} .
      - name: Push the image to Docker hub
        run: |
          docker push ${{env.IMAGE}}
      - name: Enable the Billing API
        run: |
          gcloud services enable cloudbilling.googleapis.com --project=${{secrets.GCP_PROJECT_ID}}
      - name: Deploy to GCP Run - Production environment (If a new release was published from the master branch)
        if: github.event_name == 'release' && github.event.action == 'published' && github.event.release.target_commitish == 'main'
        run: |
          gcloud run deploy ${{vars.GCR_PROJECT_NAME}} \
          --region ${{vars.GCR_REGION}} \
          --image ${{env.IMAGE}} \
          --platform "managed" \
          --allow-unauthenticated \
          --tag production \
      - name: Deploy to GCP Run - Staging environment
        if: github.ref != 'refs/heads/main'
        run: |
          echo "Deploying to staging environment"
          # Deploy service with to staging environment
          gcloud run deploy ${{vars.GCR_STAGING_PROJECT_NAME}} \
          --region ${{vars.GCR_REGION}} \
          --image ${{env.IMAGE}} \
          --platform "managed" \
          --allow-unauthenticated \
          --tag staging \

The CD pipeline configuration combines Continuous Delivery and Continuous Deployment workflows into a single file for simplicity. It builds on the concepts of CI/CD we discussed earlier, automating testing, building, and deploying the application to Google Cloud Run.

Explanation of the CD pipeline:

1. Workflow Triggers (on)

• push: Workflow triggers on pushes to the staging branch.

• workflow_dispatch: Enables manual execution of the workflow via the GitHub Actions interface.

• release: Triggers when a new release is published.
  Example: When a release is published from the main branch, the app deploys to the production environment.

2. Job 1 – Testing the Codebase: The first job in the pipeline, Test, ensures the codebase is functional and error-free before proceeding with delivery or deployment

3. Job 2 – Building and Deploying the Application: Aha! Moment ✨: These jobs run sequentially. 😃 The Build job begins only after the Test job is completed successfully. It prepares the application for deployment and manages the actual deployment process.

   Here's what happens:

   • Authorization for GCP and Docker Hub: The workflow authenticates with both Google Cloud Platform (GCP) and Docker Hub. For GCP, it uses the google-github-actions/auth@v0 action to handle service account credentials stored as secrets. Similarly, it logs into Docker Hub with stored credentials to enable image uploads.

   • Build and Push Docker Image: The application is built into a Docker image and tagged with a unique identifier (${{env.IMAGE}}). This image is then pushed to Docker Hub, making it accessible for deployment.

   • Deploy to Google Cloud Run: Based on the event that triggered the workflow, the application is deployed to either the staging or production environment in Google Cloud Run. A push to the staging branch deploys to the staging environment (Continuous Delivery), while a release from the main branch deploys to production (Continuous Deployment).

To ensure the security and flexibility of our pipeline, we rely on external variables and secrets rather than hardcoding sensitive information directly into the workflow file.

Why? Workflow configuration files are part of your repository and accessible to anyone with access to the codebase. If sensitive data, like API keys or passwords, is exposed here, it can be easily compromised. 😨

Instead, we use GitHub’s Secrets to securely store and access this information. Secrets allow us to define variables that are encrypted and only accessible by our workflows. For example:

• DockerHub Credentials: We’ll add a Docker username and access token to the repository’s secrets. These are essential for authenticating with DockerHub to upload the built Docker images.

• Google Cloud Service Account Key: This key will grant the pipeline the necessary permissions to deploy the application on Google Cloud Run securely.

We'll set up these variables and secrets incrementally as we proceed, ensuring each step is fully secure and functional. 🎯

Set Up a Docker Hub Repository for the Project's Image and Generate an Access Token for Publishing the Image 📦

Before we dive into the steps, let’s quickly go over what we’re about to do. In this section, you’ll learn how to create a Docker Hub repository, which acts like an online storage space for your application’s container image.

Think of a container image as a snapshot of your application, ready to be deployed anywhere. To ensure smooth and secure access, we’ll also generate a special access token, kind of like a revokable password that our CI/CD pipeline can use to upload your app’s image to Docker Hub. Let’s get started! 🚀

Step 1: Sign Up for Docker Hub

Here are the steps to follow to sign up for Docker Hub:

1. Go to the Docker Hub website: Open your web browser and visit Docker Hub - https://hub.docker.com/.

2. Create an account: On the Docker Hub homepage, you’ll see a button labelled "Sign Up" in the top-right corner. Click on it.

3. Fill in your details: You'll be asked to provide a few details like your username, email address, and password. Choose a strong password that you can remember.

4. Agree to the terms: You’ll need to check a box to agree to Docker’s terms of service. After that, click “Sign Up” to create your account.

5. Verify your email: Docker Hub will send you an email to verify your account. Open that email and click on the verification link to complete your account creation.

Step 2: Sign In to Docker Hub

After verifying your email, go back to Docker Hub, and click on "Sign In" at the top right. Then you can use the credentials you just created to log in.

Step 3: Generate an Access Token (for the CI/CD pipeline)

Now that you have an account, you can create an access token. This token will allow your GitHub Actions workflow to securely sign into Docker Hub and upload Docker images.

Once you’re logged into Docker Hub, click on your profile picture (or avatar) in the top right corner. This will open a menu. From the menu, click “Account Settings”.

Then in the left-hand menu of your account settings, scroll to the "Security" tab. This section is where you manage your tokens and passwords.

Now you’ll need to create a new access token. In the Security tab, you’ll see a link labelled “Personal access tokens” – click on it. Click the button labelled “Generate new token”.

You’ll be asked to give your token a description. You can name it something like "GitHub Actions CI/CD" so that you know what it's for.

After giving it a description, click on the “Access permissions dropdown“ and select “Read & Write“, or “Read, Write, Delete“. Click “Generate“

Now, you need to copy the credentials. After clicking the generate button, Docker Hub will create an access token. Immediately copy this token along with your username and save it somewhere safe, like in a file (don’t worry, we’ll add it to our GitHub secrets). You won’t be able to see this token again, so make sure you save it!

Step 4: Add the Token to GitHub as a Secret

To do this, open your GitHub repository where the codebase is hosted. In the GitHub repo, click on the Settings tab (located near the top of your repo page).

Then on the left sidebar, scroll down and click on “Secrets and Variables”, then choose “Actions”.

Here are the steps to create and manage your new secret:

1. Add a new secret: Click on the “New repository secret” button.

2. Set up the secret:

   • In the Name field, type DOCKER_PASSWORD.

   • In the Value field, paste the access token you copied earlier.

3. Save the secret: Finally, click Add secret to save your Docker access token securely in GitHub.

Then you’ll repeat the process for your Docker username. Create a new secret called DOCKER_USER and add your Docker username that you copied earlier.

And that’s it! Now your CI/CD pipeline can use this token to securely log in to Docker Hub and upload images automatically when triggered. 🎉

Step 5: Creating the Dockerfile for the Project

Before you can build and publish the Docker image to Docker Hub, you need to create a Dockerfile that contains the necessary instructions to build your application.

Follow the steps below to create the Dockerfile in the root folder of your project:

1. Navigate to your project’s root folder.

2. Create a new file named Dockerfile.

3. Open the Dockerfile in a text editor and paste the following content into it:

FROM node:18-slim

WORKDIR /app

COPY package.json .

RUN npm install -f

COPY . .

# EXPOSE 5001
EXPOSE 5001

CMD ["npm", "start"]

Explanation of the Dockerfile:

• FROM node:18-slim: This sets the base image for the Docker container, which is a slim version of the official Node.js image based on version 18.

• WORKDIR /app: Sets the working directory for the application inside the container to /app.

• COPY package.json .: Copies the package.json file into the working directory.

• RUN npm install -f: Installs the project dependencies using npm.

• COPY . .: Copies the rest of the project files into the container.

• EXPOSE 5001: This tells Docker to expose port 5001, which is the port our app will run on inside the container.

• CMD ["npm", "start"]: This sets the default command to start the application when the container is run, using npm start.

Create a Google Cloud Account, Project, and Billing Account ☁️

In this section, we’re laying the foundation for deploying our application to Google Cloud. First, we’ll set up a Google Cloud account (don’t worry, it’s free to get started!). Then, we’ll create a new project where all the resources for your app will live.

Finally, we’ll enable billing so you can unlock the cloud services needed for deployment. Think of this as setting up your workspace in the cloud—organized, ready, and secure! Let’s dive in! ☁️

Step 1: Create or Sign in to a Google Cloud Account 🌐

First, go to Google Cloud Console. If you don’t have a Google Cloud account, you’ll need to create one.

To do this, click on Get Started for Free and follow the steps to set up your account (you’ll need to provide payment information, but Google offers $300 in free credits to get started). If you already have a Google account, simply sign in using your credentials.

Once you’ve signed in, you’ll be taken to your Google Cloud dashboard. This is where you can manage all your cloud projects and resources.

Step 2: Create a New Google Cloud Project 🏗️

At the top left of the Google Cloud Console, you’ll see a drop-down menu beside the Google Cloud logo. Click on this drop-down to display your current projects.

Now it’s time to create a new project. In the top-left corner of the pop-up modal, click on the New Project button.

You’ll be redirected to a page where you’ll need to provide some basic details for your new project. So now enter the following information:

• Project Name: Enter a name of your choice for the project (for example, gcr-ci-cd-project).

• Location: Select a location for your project. You can leave it as the default "No organization" if you're just getting started.

Once you've entered the project name, click the Create button. Google Cloud will now start creating your new project. It may take a few seconds.

Step 3: Access Your New Project 🛠️

After a few seconds, you’ll be redirected to your Google Cloud dashboard.

Click on the drop-down menu beside the Google Cloud logo again, and you should now see your newly created project listed in the modal where you can select it.

Then click on the project name (for example, gcr-ci-cd-project) to enter your project’s dashboard.

Step 4: Link A Billing Account To Your Project 💳

To access the billing page, in the Google Cloud Console, find the Navigation Menu (the three horizontal lines) at the top left of the screen. Click on it to open a list of options. Scroll down and click on Billing. This will take you to the billing section of your Google Cloud account.

If you haven't set up a billing account yet, you'll be prompted to do so. Click on the "Link a billing account" button to start the process.

Now you can create a new billing account (if you don’t have one). You’ll be redirected to a page where you can either select an existing billing account or create a new one. If you don't already have a billing account, click on "Create a billing account".

Provide the necessary details, including:

• Account name (for example, "Personal Billing Account" or your business name).

• Country: Choose the country where your business or account is based.

• Currency: Choose the currency in which you want to be billed.

  

Next, enter your payment information (credit card or bank account details). Google Cloud will verify your payment method, so make sure the information is correct.

Read and agree to the Google Cloud Terms of Service and Billing Account Terms. Once you’ve done this, click "Start billing" to finish setting up your billing account

After setting up your billing account, you’ll be taken to a page that asks you to link it to your project. Select the billing account you just created or an existing billing account you want to use. Click Set Account to link the billing account to your project.

After you’ve linked your billing account to your project, you should see a confirmation message indicating that billing has been successfully enabled for your project.

You can always verify this by returning to the Billing section in the Google Cloud Console, where you’ll see your billing account listed.

Create a Google Cloud Service Account to Enable Deployment of the Node.js Application to Google Cloud Run via the CD Pipeline 🚀

Why Do We Need a Service Account and Key? 🤔

A service account allows our CI/CD pipeline to authenticate and interact with Google Cloud services programmatically. By assigning specific roles (permissions), we ensure the service account can only perform tasks related to deployment, such as managing Google Cloud Run.

The service account key is a JSON file containing the credentials used for authentication. We securely store this key as a GitHub secret to protect sensitive information.

Step 1: Open the Service Accounts Page

Here are the steps you can follow to set up your service account and get your key:

First, visit the Google Cloud Console at https://console.cloud.google.com/. Ensure you’ve selected the correct project (e.g. gcr-ci-cd-project). To change projects, click the drop-down menu next to the Google Cloud logo at the top-left corner and select your project.

Then navigate to the Navigation Menu (three horizontal lines in the top-left corner) and click on IAM & Admin > Service Accounts.

Step 2: Create a New Service Account

Click on the "Create Service Account" button. This will open a form where you’ll define your service account details.

Next, enter the Service Account details:

• Name: Enter a descriptive name (for example, ci-cd-sa).

• ID: This will auto-fill based on the name.

• Description: Add a description to help identify its purpose, such as “Used for deploying Node.js app to Cloud Run.”

• Click Create and Continue to proceed.

Step 3: Assign Necessary Roles (Permissions)

On the next screen, you’ll assign roles to the service account. Add the following roles one by one:

• Cloud Run Admin: Allows management of Cloud Run services.

• Service Account User: Grants the ability to use service accounts.

• Service Usage Admin: Enables control over enabling APIs.

• Viewer: Provides read-only access to view resources.

To add a role:

• Click on "Select a Role".

• Use the search bar to type the role name (for example, "Cloud Run Admin") and select it.

• Repeat for all four roles.

Your screen should look similar to this:

After assigning the roles, click Continue.

Step 4: Skip Granting Users Access to the Service Account

On the next screen, you’ll see an option to grant additional users access to this service account. Click Done to complete the creation process.

Step 5: Generate a Service Account Key 🔑

You should now see your newly created service account in the list. Find the row for your service account (for example, ci-cd-sa) and click the three vertical dots under the “Actions” column. Select "Manage Keys" from the drop-down menu.

To add a new key:

• Click on "Add Key" > "Create New Key".

• In the pop-up dialog, select JSON as the key type.

• Click Create.

  

Now, download the key file. A JSON file will automatically be downloaded to your computer. This file contains the credentials needed to authenticate with Google Cloud.

Make sure you keep the key secure and store it in a safe location. Don’t share it – treat it as sensitive information.

Step 6: Add the Service Account Key to GitHub Secrets 🔒

Start by opening the downloaded JSON file using a text editor (like Notepad or VS Code). Then select and copy the entire contents of the file.

Then navigate to the repository you created for this project on GitHub. Click on the Settings tab at the top of the repository. Scroll down and find the Secrets and variables > Actions section.

Now you need to add a new secret. Click the "New repository secret" button. In the Name field, enter GCP_SERVICE_ACCOUNT. In the Value field, paste the JSON content you copied earlier. Click Add secret to save it.

Do the same for the GCP_PROJECT_ID secret, but now add your Google Project ID as the value. To get your project ID, follow these steps:

1. Navigate to the Google Cloud Console: Open Google Cloud Console at https://console.cloud.google.com/.

2. Locate the Project Dropdown: At the top-left of the screen, next to the Google Cloud logo, you will see a drop-down that shows the name of your current project.

3. View the Project ID: Click the drop-down, and you'll see a list of all your projects. Your Project ID will be displayed next to the project name. It is a unique identifier used by Google Cloud.

4. Copy the Project ID: Copy the Project ID that is displayed, and add it as the value of the GCP_PROJECT_ID secret.

Step 7: Adding External Variables to the GitHub Repository 🔧

Before proceeding with deployment, we need to define some external variables that were referenced in the CD workflow. These variables ensure that the pipeline knows critical details about your Google Cloud Run services and Docker container registry.

Here are the steps you’ll need to follow to do this:

1. First, go to your repository on GitHub.

2. Click the Settings tab at the top of the repository. Scroll down to Secrets and variables > Actions.

3. Click on the Variables tab next to Secrets. Click "New repository variable" for each variable. Then you’ll need to define these variables:

   • GCR_PROJECT_NAME: Set this to the name of your Cloud Run service for the production/live environment. For example, gcr-ci-cd-app.

   • GCR_STAGING_PROJECT_NAME: Set this to the name of your Cloud Run service for the staging/test environment. For example, gcr-ci-cd-staging.

   • GCR_REGION: Enter the region where you’d like to deploy the services. For this tutorial, set it to us-central1.

   • IMAGE: Specify the name of the Docker image/container registry where the published image will be uploaded. For example, <dockerhub-username>/ci-cd-tutorial-app.

4. After entering each variable name and value, click Add variable.

Enabling the Service Usage API on the Google Cloud Project 🌐

To deploy your application, the Service Usage API must be enabled in your Google Cloud project. This API allows you to manage Google Cloud services programmatically, including enabling/disabling APIs and monitoring their usage.

Follow these steps to enable it:

1. First, visit the Google Cloud Console at https://console.cloud.google.com/.

2. Then make sure you’re in the correct project. Click the project drop-down menu near the Google Cloud logo at the top-left corner. Select gcr-ci-cd-project , or the name you gave your project from the list of projects.

3. Next you’ll need to access the API library. Open the Navigation Menu (three horizontal lines in the top-left corner). Select APIs & Services > Library from the menu.

4. In the API Library, use the search bar to search for "Service Usage API".

5. Click on the Service Usage API from the search results. On the API’s details page, click Enable.

6. To verify, go to APIs & Services > Enabled APIs & Services in the Google Cloud Console. Confirm that the Service Usage API appears in the list of enabled APIs.

   

Create the Staging Branch and Merge the Feature Branch into It (Continuous Integration and Continuous Delivery) 🌟

When changes from the feature/ci-cd-pipeline branch are merged into the staging branch, we complete the Continuous Integration (CI) process, and the workflow ci-pipeline.yml will run. This ensures that the changes made in the feature branch are tested and integrated into a shared branch.

Once the pull request (PR) is merged into staging, the Continuous Delivery (CD) pipeline automatically triggers, deploying the application to the staging environment. This simulates how updates are tested in a safe environment before being pushed to production.

Create the staging Branch on the Remote Repository

To enable the CI/CD pipeline, we’ll first create a staging branch on the remote GitHub repository. This branch will serve as the test environment where changes are deployed before they reach the production environment.

To create the staging branch directly on GitHub, follow these steps:

1. First, navigate to your repository on GitHub. Open your web browser and go to the GitHub repository where you want to create the new staging branch.

2. Then, switch to the main branch. On the top of the repository page, locate the Branch dropdown (usually labelled as main or the current branch name). Click on the dropdown and make sure you are on the main branch.

3. Next, create the staging branch. In the same dropdown where you see the main branch, type staging into the text box. Once you start typing, GitHub will offer you the option to create a new branch called staging. Select the Create branch: staging option from the dropdown.

4. Finally, verify the branch**.** After creating the staging branch, GitHub will automatically switch to it. You should now see staging in the branch dropdown, confirming the new branch was created.

   

Merge Your Feature Branch into the Staging Branch via a Pull Request (PR)

This process combines both Continuous Integration (CI) and Continuous Delivery (CD). You will commit changes from your feature branch, push them to the remote feature branch, and then open a PR to merge those changes into the staging branch. Here's how to do it:

Step 1: Commit Local Changes on Your Feature Branch

First, you’ll want to make sure that you are on the correct branch (the feature branch) by running:

git status

If you are not on the feature/ci-cd-pipeline branch, switch to it by running:

git checkout feature/ci-cd-pipeline

Now, it’s time to add your changes you made for the commit:

git add .

This stages all changes, including new files, modified files, and deleted files.

Next, commit your changes with a clear and descriptive message:

git commit -m "Set up CI/CD pipelines for the project"

Then you can verify your commit by running:

git log

This will display your most recent commits, and you should see the commit message you just added.

Step 2: Push Your Feature Branch Changes to the Remote Repository

After committing your changes, push them to the remote repository:

git push origin feature/ci-cd-pipeline

This pushes your local changes on the feature/ci-cd-pipeline branch to the remote GitHub repository.

Once the push is successful, visit your GitHub repository in a web browser, and confirm that the feature/ci-cd-pipeline branch is updated with your new commit.

Step 3: Create a Pull Request to Merge the Feature Branch into Staging

Go to your repository on GitHub and ensure that you are on the main page of the repository.

You should see an alert at the top of the page suggesting you create a pull request for the recently pushed branch (feature/ci-cd-pipeline). Click the Compare & Pull Request button next to the alert.

Now, it’s time to choose the base and compare branches. On the PR creation page, make sure the base branch is set to staging (this is the branch you want to merge your changes into). The compare branch should already be set to feature/ci-cd-pipeline (the branch you just pushed). If they’re not selected correctly, use the dropdowns to change them.

You’ll want to come up with a good PR description for this. Write a clear title and description for the pull request, explaining what changes you're merging and why. For example:

• Title: "Merge CI/CD setup changes from feature branch"

• Description: "This pull request adds the CI/CD pipelines for GitHub Actions and Docker Hub integration to the project. It includes the configurations for both CI and CD workflows."

Now GitHub will show a list of all the changes that will be merged. Take a moment to review them and ensure everything looks correct.

If all looks good after reviewing, click on the Create pull request button. This will create the PR and notify team members (if any) that changes are ready to be reviewed and merged.

Wait a few seconds, and you should see a message indicating that all the checks have passed. Click on the link with the description "CI Pipeline to staging/production environment...". This should direct you to the Continuous Integration workflow, where you can view the steps that ran

The Continuous Integration (CI) Process

The CI process begins when a Pull Request is made to the staging branch. It triggers the GitHub Actions workflow defined in the .github/workflows/ci-pipeline.yml file. The workflow runs the necessary steps to set up the environment, install dependencies, and build the Node.js application.

It then runs automated tests (using npm test) to ensure that the changes do not break any functionality in the codebase. If all these steps are completed successfully, the CI pipeline confirms that the feature branch is stable and ready to be merged into the staging branch for further testing and deployment.

Step 4: Merge the Pull Request

If your team or collaborators are part of the project, they may review your PR. This step may involve discussing any changes or improvements. If everything looks good, a reviewer will merge the PR.

Once the PR has been reviewed and approved, you can merge the PR. To do this, just click on the Merge pull request button. Choose Confirm merge when prompted.

After merging, you can go to the staging branch to verify that the changes were successfully merged.

Navigating to the Actions Page After Merging the PR

Once you have successfully merged your pull request from the feature/ci-cd-pipeline branch into the staging branch, the Continuous Delivery (CD) pipeline will be triggered. To view the progress of the CD pipeline, navigate to the Actions tab in your GitHub repository. Here's how to do it:

1. Go to your GitHub repository.

2. At the top of the page, you will see the Actions tab next to the Code tab. Click on it.

3. On the Actions page, you will see a list of workflows that have been triggered. Look for the one labelled CD Pipeline to Google Cloud Run (staging and production). It should appear as a new run after the PR merge.

4. Click on the workflow run to view its progress and see the detailed logs for each step.

This will allow you to monitor the status of the CD pipeline and check if there are any issues during deployment.

If you look at the CD steps and workflow, you'll see that the step to deploy the application to the production environment was skipped, while the step to deploy to the staging environment was executed.

Continuous Delivery (CD) pipeline – what’s going on:

The Continuous Delivery (CD) Pipeline automates the process of deploying the application to Google Cloud Run (testing environment). This workflow is triggered by a push to the staging branch, which happens after the changes from the feature branch are merged into staging. It can also be manually triggered via workflow_dispatch or upon a new release being published.

The pipeline consists of multiple stages:

1. Test Job: The pipeline begins by setting up the environment and running tests using the npm test command. If the tests pass, the process moves forward.

2. Build Job: The next step builds the Docker image of the Node.js application, tags it, and then pushes it to Docker Hub.

3. Deployment to GCP: After the image is pushed, the workflow authenticates to Google Cloud and deploys the application. If the event is a release (that is, a push to the main branch), the application is deployed to the production environment. If the event is a push to staging, the app is deployed to the staging environment.

The CD process ensures that any changes made to the staging branch are automatically tested, built, and deployed to the staging environment, ready for further validation. When a release is published, it will trigger deployment to production, ensuring your app is always up to date.

Accessing the Deployed Application in the Staging Environment on Google Cloud Run 🌐

Once the deployment to Google Cloud Run is successfully completed, you'll want to access your application running in the staging environment. Follow these steps to find and visit your deployed application:

1. Navigate to the Google Cloud Console

Open the Google Cloud Console in your browser by visiting https://console.cloud.google.com. If you're not already signed in, make sure you log in with your Google account.

2. Go to the Cloud Run Dashboard

In the Google Cloud Console, use the Search bar at the top or navigate through the left-hand menu: Go to Cloud Run (you can type this into the search bar, or find it under Products & services > Compute > Cloud Run). Click on Cloud Run to open the Cloud Run dashboard.

3. Select Your Staging Service

In the Cloud Run dashboard, you should see a list of all your services deployed across various environments. Find the service associated with the staging environment. The name should be similar to what you defined in your workflow (for example, gcr-ci-cd-staging).

4. Access the Service URL

Once you've selected your staging service, you’ll be taken to the Service details page. This page provides all the important information about your deployed service.
On this page, look for the URL section under the Service URL heading. The URL will look something like: https://gcr-ci-cd-staging-<unique-id>.run.app.

5. Visit the Application

Click on the Service URL, and it will open your staging environment in a new tab in your browser. You can now interact with your application as if it were live, but in the staging environment.

Merge the Staging Branch into the Main Branch (Continuous Integration and Continuous Deployment) 🌐

In this section, we'll take the updates in the staging branch, merge them into the main branch, and trigger the CI/CD pipeline. This process not only ensures your changes are production-ready but also deploys them to the production/live environment. 🚀

Step 1: Push Local Changes and Open a Pull Request

Why? The first step involves merging the staging branch into the main branch. Just like in the previous Continuous Delivery process, this ensures the integration of thoroughly tested updates.

Here’s how to do it:

First, visit the GitHub repository where your project is hosted.

Then go to the Pull Requests tab. Click New Pull Request. Choose staging as the source branch (base branch) and main as the target branch. Add a clear title and description for the Pull Request, explaining why these updates are ready for production deployment.

Step 2: Continuous Integration (CI) Pipeline Execution

After merging the pull request, the Continuous Integration (CI) pipeline will automatically execute to validate that the changes are still stable when integrated into the main branch.

Pipeline Steps:

• Code Checkout: The workflow fetches the latest code from the main branch.

• Dependency Installation: The pipeline installs all required dependencies.

• Testing: Automated tests are run to validate the application's stability.

Step 3: Create a New Release

The Continuous Deployment (CD) workflow to deploy to the production environment is triggered by the creation of a new release from the main branch.

Let’s walk through the steps to create a release.

On your GitHub repository page, click on the Releases section (located under the Code tab).

Next, click Draft a new release. Set the Target branch to main. Enter a Tag version (for example, v1.0.0) following semantic versioning. Add a Release title and an optional description of the changes.

Then, click Publish Release to finalize.

Why run the Continuous Deployment pipeline on release instead of on push? 🤔

In our setup, we decided not to trigger the Continuous Deployment (CD) pipeline every time changes are pushed to the main branch. Instead, we trigger it only when a new release is created. This gives the team more control over when updates are deployed to the production environment.

Imagine a scenario where developers are working on new features—they may push changes to the main branch as part of their regular workflow, but these features might not be complete or ready for users yet. Automatically deploying every push could accidentally expose unfinished features to your users, which can be confusing or disruptive.

By requiring a release to trigger the deployment, the team gets a chance to finalize and polish all changes before they go live.

For example, developers can test new features in the staging environment, fix any issues, and merge those changes into the main branch without worrying about them immediately appearing in production. This workflow ensures that only well-tested and complete features make their way to your end users.

Ultimately, this approach helps maintain a smooth user experience. Instead of seeing half-built features or unexpected changes, users only see updates that are ready and functional. It also gives the team the flexibility to push changes to the main branch frequently—preventing merge conflicts and making collaboration easier—while keeping control over what gets deployed live. 🚀

Step 4: Navigate to the Actions Page

After the release is published, the CD pipeline for the production environment is triggered. To monitor this repeat the process taken for the Continuous Delivery workflow, follow these steps:

1. Go to the GitHub Actions tab: In your GitHub repository, click on the Actions tab.

2. Locate the deployment workflow: Look for the CD Pipeline to Google Cloud Run (staging and production) workflow. You’ll notice that the workflow has been triggered on the main branch due to the push event.

3. Open the workflow details: Click on the workflow to view detailed steps, logs, and statuses for each part of the deployment process.

This time, the Continuous delivery workflow deploys the application to the production/live environment.

Step 5: Access the Live Application

Once the deployment is complete, go to Google Cloud Console at https://console.cloud.google.com.

Navigate to Cloud Run from the menu. Select the service corresponding to the production environment (for example, gcr-ci-cd-app).

Locate the Service URL in the service details page. Open the URL in your browser to access the live application.

And now, congratulations – you’re done!

Conclusion 🌟

In this article, we explored how to build and automate a CI/CD pipeline for a Node.js application, using GitHub Actions, Docker Hub, and Google Cloud Run.

We set up workflows to handle Continuous Integration by testing and integrating code changes and Continuous Delivery to deploy those changes to a staging environment. We also containerized our app using Docker and deployed it seamlessly to Google Cloud Run.

Finally, we implemented Continuous Deployment, ensuring updates to the production environment happen only when a release is created from the main branch.

This approach gives teams the flexibility to push and test incomplete features without impacting end users. By following these steps, you've built a robust pipeline that makes deploying your application smoother, faster, and more reliable.

Study Further 📚

If you would like to learn more about Continuous Integration, Delivery, and Deployment you can check out the courses below:

• Continuous Integration and Continuous Delivery (CI/CD) (from IBM Coursera)

• GitHub Actions - The Complete Guide (from Udemy)

• Learn CI/CD by buliding a project (freeCodeCamp tutorial)

About the Author 👨‍💻

Hi, I’m Prince! I’m a software engineer passionate about building scalable applications and sharing knowledge with the tech community.

If you enjoyed this article, you can learn more about me by exploring more of my blogs and projects on my LinkedIn profile. You can find my LinkedIn articles here. And you can visit my website to read more of my articles as well. Let’s connect and grow together! 😊

In this guide, we’ll walk through setting up automated GitHub workflows for a Python backend (using Flask or Django) and a React frontend. These workflows help test and validate code changes automatically, making sure any issues are caught early.

We’ll assume:

• You’ve already written unit tests for your React components and backend routes.

• Your project is set up as a monorepo, with separate directories for frontend and backend.

• You’re familiar with GitHub Actions, the platform we’ll use for automation, and that you’re using the ubuntu-latest environment provided by GitHub.

Step 1: Create GitHub Actions Workflows

In this step, we’ll define two GitHub Actions workflows, one for the frontend and another for the backend. These workflows will run tests automatically whenever changes are pushed to the main branch.

What is a GitHub Action Workflow?

A GitHub Action workflow is a set of instructions that tell GitHub how to automatically execute tasks based on certain events.

Here, our workflows will run tests and deploy the app only if the tests pass. Workflows are triggered by events, such as a push to a branch, and consist of jobs that define the tasks we want to automate.

Frontend CI/CD Pipeline

Let’s start by creating a new file in your repository at .github/workflows/frontend.yml. This file will set up an automated pipeline to handle the frontend testing and deployment. Then, define the workflow with the following content:

name: Frontend CI/CD Pipeline

on:
  push:
    branches:
      - main  

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v3

      - name: Cache Node.js modules
        uses: actions/cache@v3
        with:
          path: ./frontend/node_modules
          key: ${{ runner.os }}-node-${{ hashFiles('./frontend/package-lock.json') }}
          restore-keys: |
            ${{ runner.os }}-node-

      - name: Set up Node.js
        uses: actions/setup-node@v3
        with:
          node-version: '20'

      - name: Install frontend dependencies
        run: yarn install
        working-directory: ./frontend 

      - name: Run frontend tests
        run: yarn test
        working-directory: ./frontend

Here’s a breakdown of what each part does:

1. on: push: This triggers the workflow whenever there’s a push to the main branch.

2. Checkout code: This step uses the GitHub Action to check out the repository code.

3. Cache Node.js modules: Caches node_modules to speed up workflow execution on subsequent runs.

4. Set up Node.js: Sets up the Node.js environment for dependency installation and testing.

5. Install dependencies and run tests: Installs packages with Yarn and then runs the pre-written tests to verify that the frontend works as expected.

Backend CI/CD Pipeline

Now, let’s create a separate file for the backend workflow at .github/workflows/backend.yml. This file will automate testing and deployment for the Python backend.

name: Backend CI/CD Pipeline

on:
  push:
    branches:
      - main  

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v3

      - name: Cache Python packages
        uses: actions/cache@v3
        with:
          path: ~/.cache/pip
          key: ${{ runner.os }}-pip-${{ hashFiles('./backend/requirements.txt') }}
          restore-keys: |
            ${{ runner.os }}-pip-

      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.8'  

      - name: Create Virtual Environment
        run: python3 -m venv venv
        working-directory: ./backend

      - name: Install backend dependencies
        run: |
          source venv/bin/activate
          pip install -r requirements.txt  
        working-directory: ./backend

      - name: Configure DATABASE_URL securely
        env:
          DATABASE_URL: ${{ secrets.DATABASE_URL }}
        run: |
          if [ -z "$DATABASE_URL" ]; then
            echo "DATABASE_URL is missing" >&2
            exit 1
          fi

      - name: Run tests with pytest
        run: |
          source venv/bin/activate
          pytest tests/ --doctest-modules -q --disable-warnings
        working-directory: ./backend  

      - name: Deploy to Production
        if: ${{ success() }}
        run: |
          echo "Deploying to production..."

      - name: Notify on Failure
        if: ${{ failure() }}
        run: |
          echo "Build failed! Sending notification..."

Here’s what this workflow does:

1. Checks out code and caches Python packages for faster execution on repeated runs.

2. Sets up Python and creates a virtual environment to isolate dependencies.

3. Installs dependencies in the virtual environment from requirements.txt.

4. Configures environment variables securely with GitHub Secrets. In this example, we’re using a database URL that’s stored in a GitHub secret for secure access.

5. Runs backend tests with pytest, which checks that the backend routes and functions work correctly.

Step 2: Configure Secrets

For security, let’s set up GitHub Secrets to store sensitive information, like database connection strings.

1. Go to your GitHub repository and select Settings.

2. In the sidebar, select "Secrets and variables" from the sidebar, then click on "Actions".

3. Add a new repository secret:

   • Name: DATABASE_URL

   • Value: Your actual database connection string.

Using GitHub Secrets keeps sensitive data safe and prevents it from appearing in your codebase.

Step 3: Commit and Push Changes

Once your workflow files are ready, commit and push the changes to the main branch. Each time you push changes to main, GitHub Actions will trigger these workflows automatically, ensuring your code is thoroughly tested.

Step 4: Monitor Workflow Runs

After pushing your changes, navigate to the Actions tab in your GitHub repository to monitor the workflow runs. Here’s what you’ll find:

• Workflow runs: This page lists each time a workflow is triggered. You can see if the workflow succeeded, failed, or is in progress.

• Logs: Click on a specific workflow run to view detailed logs. Logs are divided by steps, so you can see exactly where an issue occurred if something goes wrong.

Identifying Issues in Logs

Each step’s log provides insights into any problems:

• If dependencies fail to install, you’ll see error messages specifying which package caused the issue.

• If tests fail, logs will list the specific tests and reasons for the failure, helping you debug quickly.

• For workflows that use secrets, errors related to missing secrets will appear in the environment setup steps, allowing you to fix any configuration issues.

By understanding how to interpret these logs, you can address issues proactively and ensure smooth, reliable deployments.

Conclusion

By following these steps, you’ve set up automated GitHub workflows for both the frontend and backend of your application.

This setup ensures your tests run automatically with each push to the main branch, helping maintain high code quality and reliability.

With automated workflows, you can focus more on building features and less on manually testing code, knowing that your workflows will alert you to any issues early on.

In this tutorial we'll learn how to automatically publish machine learning models in a Gitlab package registry and make them available for your teammates to use. You can also use this technique to share a packaged version of your code as a binary.

If you are a beginner Gitlab user and are unfamiliar with CI/CD techniques, this tutorial is for you! A basic understanding of how machine-learning and deep learning is a plus, but it isn't a requirement to understand the CI/CD publishing part.

Here's what we'll cover:

• Gitlab Code Setup
• Deep Convolutional Neural Network Code
• Image Recognition Code
• Branching Methodology
• CI/CD Uploading
• Conclusion

First, Some Background

At some point during your machine learning engineer career you might need to share a model you've trained with other developers. There are multiple ways of doing this.

Give access to the repository

If you don't mind showing your whole code, this is a very viable option.

If you use a good branching methodology your colleagues will only need to look at the main branch in order to figure out what's the most up to date model they can use. Then they can check the README.md to learn how to use it.

However, giving full access to the repository might not be a viable option for you.

Share the latest model manually

Another way would be to extract the relevant code that you want to make public and send it to them manually.

This can become a bit of a mess if you are working with more than one person because the model you send might not be up to date. It also puts the burden on you to make sure that people are always using the latest version of your model.

Share the latest model automatically

A simpler solution, even in the case where the repository code is available, is to put the packaging burden on a CI/CD pipeline.

This is the topic of this tutorial, and our setup will look like this:

• The code repository, CI/CD tool set, and package registry will be on Gitlab
• The code we'll be packaging will be a simple trained PyTorch neural network on the MNIST dataset for digit recognition.
• All the instructions and the requirements will be available in the package.

🚨 Disclaimer 🚨: This isn't how you should deploy a PyTorch production-ready model! To learn how to do this, check out this tutorial on TorchScript.

Let's get started.

Gitlab Code Setup

For this tutorial we will bundle four files:

• model.pth: which is a pickled version of the latest version of the trained model.
• run_mnist.py: simple Python script to run the model to detect a digit from a png image.
• requirements.txt: text file containing all the dependencies required to run the model.
• INSTRUCTION.md: step by step instructions to use the package.

The package can then be used freely by anyone who has access to the package registry and will be automatically updated.

The package will then look like this on Gitlab Package Registry!

Let's jump into the neural network code, which is a modified version of this comprehensive article on digit recognition. The modified code can be found over at my public Gitlab repository.

Deep Convolutional Neural Network Code

In the section below, you will see quite a lot of terminology about deep neural networks. This isn't a tutorial on neural networks, so if you feel a bit overwhelmed by the specifics you can jump directly to the Branching Methodology section.

Just keep in mind that we've trained some sort of image recognition program that, given a .png file representing a digit, will be able to tell you what number it contains.

However, for those that want to get a better understanding about how Deep Neural Networks work under the hood, you can take a look at my tutorial where I build one from scratch or checkout the code directly in my Github.

Neural Network Definition

The network definition code is very straightforward since the network we will use is simple. It has the following characteristics:

• 2 convolutional layers.
• Dropout is applied on the second convolutional layer.
• Relu activation functions applied on all neurons.
• 2 fully connected layers at the end for inference.

import torch
import torchvision

import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim


# Define the network
# It's a 2 convolutional layer with dropout at the 2nd and finally 2 fully connected layer
# All layers use relu
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 10, kernel_size=5)
        self.conv2 = nn.Conv2d(10, 20, kernel_size=5)
        self.conv2_drop = nn.Dropout2d()
        self.fc1 = nn.Linear(320, 50)
        self.fc2 = nn.Linear(50, 10)

    def forward(self, x):
        x = F.relu(F.max_pool2d(self.conv1(x), 2))
        x = F.relu(F.max_pool2d(self.conv2_drop(self.conv2(x)), 2))
        x = x.view(-1, 320)
        x = F.relu(self.fc1(x))
        x = F.dropout(x, training=self.training)
        x = self.fc2(x)
        return F.log_softmax(x, dim=1)

Training Function

We then created a utility training function in order to iteratively improve our defined network using gradient descent. If you want to learn more about how gradient descent works check out my short tutorial on it.

This training regimen will do the following:

• Iterate on batches of training data representing 28 by 28 digits.
• Use the negative log likelihood cost function to calculate the loss.
• Calculate gradients.
• Optimize the weights of the network using gradient descent.
• Save the model at fixed intervals.

def train(network, optimizer, train_loader, epoch_id, log_interval=10):
  """Run the training regiment on the training set using train_loader

    Args:
        network: The instantiated network.
        optimizer: The optimizer used to change the weights.
        train_loader: the loader for the training set already setup
        epoch_id: the current id of the epoch used for cosmetic reason.
        log_interval: interval at which we print an output

    Returns:
        nothing, will save directly at root level the model and the optimizer state

  """

  # Set the network in training mode
  network.train()

  # Iterate over the full training set
  for batch_idx, (data, target) in enumerate(train_loader):

    # Calculate the gradients for this batch of data
    optimizer.zero_grad()
    output = network(data)
    loss = F.nll_loss(output, target)
    loss.backward()

    # Optimize the network
    optimizer.step()

    # Log and save every selected interval
    if batch_idx % log_interval == 0:

      print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
        epoch_id, batch_idx * len(data), len(train_loader.dataset),
        100. * batch_idx / len(train_loader), loss.item()))

      # This will save the state as a pickled object
      torch.save(network.state_dict(), './model.pth')
      torch.save(optimizer.state_dict(), './optimizer.pth')

The data for training can be found over here on the Yan LeCun website. Here we are using the datasets formatted as 28 by 28 PyTorch tensors for training.

Testing Function

The next function we create is a testing function to validate if our network has learned something without reusing the same training data. This function is simple in the sense that it will just tally the correct and incorrect predictions.

def test(network, test_loader):
  """Run the testing regiment on the test set using test_loader

    Args:
        network: The instantiated and trained network.
        test_loader: the loader for the testing set already setup

    Returns:
        nothing, will only print result

  """

  # Variable instantiation
  test_loss = 0
  correct = 0

  # Move the network to evaluate mode instead of training
  network.eval()

  # setup torch so to not track any  gradient
  with torch.no_grad():

    # Iterate on all the test data and accumulate the loss
    for data, target in test_loader:
      output = network(data)
      test_loss += F.nll_loss(output, target, size_average=False).item()
      pred = output.data.max(1, keepdim=True)[1]
      correct += pred.eq(target.data.view_as(pred)).sum()

  # Average loss calculation and printing   
  test_loss /= len(test_loader.dataset)
  print('\nTest set: Avg. loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\n'.format(
    test_loss, correct, len(test_loader.dataset),
    100. * correct / len(test_loader.dataset)))

This function will be useful to check how well our network has learned after each training iteration.

Training Regimen

Finally, we can tie all of the above together with the main body of the training script! A few things are happening, but the most important points are the following:

• We set our hyper parameters statically. A better way to define them would be to use a validation set to figure them out based on the data.
• We create our data loader which will ingest data and spit out tensors in the right shape for the network. These loader will transform the data by normalizing them with the global mean and standard deviation for the MNIST datasets.
• We use stochastic gradient descent with momentum as the optimization method, which is one of the many flavors of gradient descent we can use.
• We loop through the full training dataset's "epoch", the amount of time to train the network while testing on the held-out test datasets.

# Experimental Parameters that we can tweak
n_epochs = 3
batch_size_train = 64
batch_size_test = 1000
learning_rate = 0.01
momentum = 0.5

# Variable from the dataset that should stay as is
global_mean_mnist = 0.1307
global_std_mnist = 0.3081


# Random Seed for Reproducible Experimentation
random_seed = 42
torch.backends.cudnn.enabled = False
torch.manual_seed(random_seed)


# Data Loader to gather the data and then normalize them
train_loader = torch.utils.data.DataLoader(
  torchvision.datasets.MNIST('./data/', train=True, download=True,
                             transform=torchvision.transforms.Compose([
                               torchvision.transforms.ToTensor(),
                               torchvision.transforms.Normalize(
                                 (global_mean_mnist,), (global_std_mnist,))
                             ])),
  batch_size=batch_size_train, shuffle=True)

test_loader = torch.utils.data.DataLoader(
  torchvision.datasets.MNIST('./data/', train=False, download=True,
                             transform=torchvision.transforms.Compose([
                               torchvision.transforms.ToTensor(),
                               torchvision.transforms.Normalize(
                                 (global_mean_mnist,), (global_std_mnist,))
                             ])),
  batch_size=batch_size_test, shuffle=True)

# Initialize network and optimizer
network = Net()
optimizer = optim.SGD(network.parameters(), lr=learning_rate,
                      momentum=momentum)

# Test first to show that the model didn't learn a thing
test(network, test_loader)

# Train on the whole dataset multiple time and test
for epoch_id in range(1, n_epochs + 1):
  train(network, optimizer, train_loader, epoch_id)
  test(network, test_loader)

Note that it's very important to test your network on a held-out set to avoid over-fitting on the training data.

All of the above scripts can be found in the file train_mnist.py in the repository.

At this point, we can train a model and have it saved at regular intervals in a pickle format.

We can now use that saved trained mode to evaluate a digit in a .png file.

Image Recognition Code

Let's say we have as an input the following image:

a small 0 digit

or this one:

a bigger 7 digit

How can we make our network, which works on a 28 by 28 PyTorch tensor, evaluate the numbers?

It's fairly straightforward if we follow roughly the same process that the training datasets went through, which is:

• Have grayscale images (no color or alpha channels)
• Resize the images to be 28 by 28 pixels
• Normalize the images using the mean and standard deviation of the MNIST datasets.

if __name__ == "__main__":

    # Variable iniatilization
    global_mean_mnist = 0.1307
    global_std_mnist = 0.3081

    # Loading of the network with right weight
    result_path = './model.pth'
    model = Net()
    model.load_state_dict(torch.load(result_path))
    model.eval()

    # Setup the transform from image to normalized tensors
    transform = transforms.Compose([
                        transforms.Resize((28,28)),
                        transforms.ToTensor(),
                        transforms.Normalize(
                            (global_mean_mnist,), (global_std_mnist,))
                        ])

    # Parse the input from the user which should be a filename with the --image flag
    parser = OptionParser()
    parser.add_option("--image", dest = "input_image_path",
                      help = "Input Image Path")
    (options, args) = parser.parse_args()

    # Get the path to the image to decode
    input_image_path = str(options.input_image_path)

    # Open the image(s) and do the inference
    images=glob.glob(input_image_path)
    for image in images:

        # Convert the image to grayscale
        img = Image.open(image).convert('L')

        # Transform the image to a normalized tensor
        img_tensor = transform(img).unsqueeze(0)

        # Make and print the prediction
        output = model(img_tensor).data.max(1, keepdim=True)[1][0][0]
        print(f"Image is a {int(output)}")

As you can see, we use a parser to accept an image path on the command line before applying our transformations. Once they are applied we can feed that to our loaded model and collect the output prediction.

⚠️ Don't forget to include the definition of the network in the script (by importing or copy pasting), otherwise the pickled model will not be able to load properly.

We can now run our code like this:

python run_mnist.py --image NAME_OF_IMAGE.png

This will simply print the model's inference about what that particular image contains.

Now that we have the basic training and evaluation code set up, let's discuss a bit more about how to use git branching to our advantage to publish this model to the package registry.

Branching Methodology

If you are working alone on a project, it is very tempting to simply commit to master/main and be done with it. However, this way of working is very difficult to maintain and it makes incorporating proper CI/CD tools a pain.

A main / develop branch strategy as shown below is more maintainable:

Image from: https://nvie.com/posts/a-successful-git-branching-model/

By always keeping the main branch clean, we can easily flag our CI/CD pipeline to be triggered as soon as we push to the main. We will be also free to commit as much as we need in the develop branch while we improve our models.

When we are ready for a new deploy we will only need to merge with the main branch (or better yet do a merge-request / pull-request and then merge).

This merge to main should trigger Gitlab to upload the new version of our model to the package registry.

Let's take a look at the simple way to automate publishing to the package registry using the .gitlab-ci.yml file.

CI/CD Pipeline

The .gitlab-ci.yml file is a special file in your repository used by Gitlab to define what the Gitlab server should do when you push to a repository.

To learn more about how CI/CD works in Gitlab, head over to this Gitlab CI/CD crash course.

In this tutorial our .gitlab-ci.yml file looks like this:

image: pytorch/pytorch

variables:
  VERSION: "0.0.4" # To Change if needs be

stages:
  - upload

upload:
  stage: upload
  only:
    - master
  script:
    - apt-get update
    - apt-get install -y curl wget
    - 'curl --header "JOB-TOKEN: $CI_JOB_TOKEN" --upload-file ./model.pth "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/generic/example-ml-packaging-pipeline/${VERSION}/model.pth"'
    - 'curl --header "JOB-TOKEN: $CI_JOB_TOKEN" --upload-file ./run_mnist.py "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/generic/example-ml-packaging-pipeline/${VERSION}/run_mnist.py"'
    - 'curl --header "JOB-TOKEN: $CI_JOB_TOKEN" --upload-file ./requirements.txt "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/generic/example-ml-packaging-pipeline/${VERSION}/requirements.txt"'
    - 'curl --header "JOB-TOKEN: $CI_JOB_TOKEN" --upload-file ./INSTRUCTION.md "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/generic/example-ml-packaging-pipeline/${VERSION}/INSTRUCTION.md"'

The anatomy of this .yml file is very bare bones. We have only one stage in our pipeline which is the upload stage.

In the upload stage, we will run the script section only when the master branch gets updated. The script that we ran is simply using curl to transfer the data from this repository (4 files) into the package registry.

Let's take a look at the anatomy of the curl command we are using:

 - 'curl --header "JOB-TOKEN: $CI_JOB_TOKEN" --upload-file ./NAME_OF_FILE "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/generic/example-ml-packaging-pipeline/${VERSION}/NAME_OF_FILE"'

• --header is used to tell curl that you will be including an extra header to the request.
• JOB-TOKEN is our header and $CI_JOB_TOKEN is its value. It's a variable that lives within Gitlab servers when a job is created
• --upload-file is a flag to tell that we will transfer a local file to the remote URL.
• ./NAME_OF_FILE is the name of the local file we want to transfer.
• ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/generic/example-ml-packaging-pipeline/${VERSION}/NAME_OF_FILE is the location of the remote URL that we want to transfer a file.

Here $CI_API_V4_URL is the URL of the Gitlab API we are using, $CI_PROJECT_ID is defined within Gitlab CI as the id for our project, and finally VERSION is the version number we defined at the top of the .yml file.

That's it! When you update the main branch to the remote repository on Gitlab it will fire up a pipeline that will run your packaging job.

The job will then be available and you will be able to check the trace on Gitlab!

You and your teammates will be able to see the document in the package registry section and get the right versioned files in the package:

This is our v.0.0.5 of the example package!

To get a more complete idea of what is possible with the Packages API, head over to the official documentation.

Conclusion

In this tutorial you've learn how to bundle, upload, and automatize a machine learning model packaging using Gitlab CI/CD.

Congratulation! 🎉🎉🎉

There is still a lot more you can do with Gitlab CI/CD, for instance:

• Add a testing stage before the bundling in order to make sure that there is no regression in the code.
• Add a testing stage after the bundling to make sure that the performance of your model is satisfactory in terms of inference latency.
• Use a more optimized version of the model with TorchScript.
• Add automatic social notification of new release after the upload step.

To learn more about Gitlab CI/CD the official docs is a great place to start out, and the get started section is very beginner friendly.

If you want to read more of this type of content, check out my mechanical/software engineering articles. If you want to discuss any of this feel free to send me a DM on LinkedIn or Twitter :)

But automation can be tough and can sometimes prove to be costly. How can Github Actions help harden our code and give us more time to work on features instead of bugs?

• What are Github Actions?
• What is CI/CD?
• What are we going to build?
• Part 0: Setting up a project
• Part 1: Automating tests
• Part 2: Post new pull requests to Slack

What are Github Actions?

Actions are a relatively new feature to Github that allow you to set up CI/CD workflows using a configuration file right in your Github repo.

Previously, if you wanted to set up any kind of automation with tests, builds, or deployments, you would have to look to services like Circle CI and Travis or write your own scripts. But with Actions, you have first class support to powerful tooling to automate your workflow.

What is CI/CD?

CD/CD stands for Continuous Integration and Continuous Deployment (or can be Continuous Delivery). They're both practices in software development that allow teams to build projects together quickly, efficiently, and ideally with less errors.

Continuous Integration is the idea that as different members of the team work on code on different git branches, the code is merged to a single working branch which is then built and tested with automated workflows. This helps to constantly make sure everyone's code is working properly together and is well-tested.

Continuous Deployment takes this a step further and takes this automation to the deployment level. Where with the CI process, you automate the testing and the building, Continuous Deployment will automate deploying the project to an environment.

The idea is that the code, once through any building and testing processes, is in a deployable state, so it should be able to be deployed.

What are we going to build?

We're going to tackle two different workflows.

The first will be to simply run some automated tests that will prevent a pull request from being merged if it is failing. We won't walk through building the tests, but we'll walk through running tests that already exist.

In the second part, we'll set up a workflow that sends a message to slack with a link to a pull request whenever a new one is created. This can be super helpful when working on open source projects with a team and you need a way to keep track of requests.

Part 0: Setting up a project

For this guide, you can really work through any node-based project as long as it has tests you can run for Part 1.

If you'd like to follow along with a simpler example that I'll be using, I've set up a new project that you can clone with a single function that has two tests that are able to run and pass.

A function with two tests

If you'd like to check out this code to get started, you can run:

git clone --single-branch --branch start git@github.com:colbyfayock/my-github-actions.git

Once you have that cloned locally and have installed the dependencies, you should be able to run the tests and see them pass!

Passing tests

It should also be noted that you'll be required to have this project added as a new repository on Github in order to follow along.

Follow along with the commit!

Part 1: Automating tests

Tests are an important part of any project that allow us to make sure we're not breaking existing code while we work. While they're important, they're also easy to forget about.

We can remove the human nature out of the equation and automate running our tests to make sure we can't proceed without fixing what we broke.

Step 1: Creating a new action

The good news, is Github actually makes it really easy to get this workflow started as it comes as one of their pre-baked options.

We'll start by navigating to the Actions tab on our repository page.

Github Actions starting page

Once there, we'll immediately see some starter workflows that Github provides for us to dive in with. Since we're using a node project, we can go ahead and click Set up this workflow under the Node.js workflow.

Setting up a Node.js Github Action workflow

After the page loads, Github will land you on a new file editor that already has a bunch of configuration options added.

We're actually going to leave this "as is" for our first step. Optionally, you can change the name of the file to tests.yml or something you'll remember.

Adding a new Github Action workflow file

You can go ahead and click Start commit then either commit it directory to the master branch or add the change to a new branch. For this walkthrough, I'll be committing straight to master.

To see our new action run, we can again click on the Actions tab which will navigate us back to our new Actions dashboard.

Viewing Github Action workflow events

From there, you can click on Node.js CI and select the commit that you just made above and you'll land on our new action dashboard. You can then click one of the node versions in the sidebar via build (#.x), click the Run npm test dropdown, and we'll be able to see the output of our tests being run (which if you're following along with me, should pass!).

Viewing logs of a Github Action workflow

Follow along with the commit!

Step 2: Configuring our new action

So what did we just do above? We'll walk through the configuration file and what we can customize.

Github Action Node.js workflow file

Starting from the top, we specify our name:

name: Node.js CI

This can really be whatever you want. Whatever you pick should help you remember what it is. I'm going to customize this to "Tests" so I know exactly what's going on.

on:
  push:
    branches: [ master ]
  pull_request:
    branches: [ master ]

The on key is how we specify what events trigger our action. This can be a variety of things like based on time with cron. But here, we're saying that we want this action to run any time someone pushes commits to master or someone creates a pull request targeting the master branch. We're not going to make a change here.

jobs:
  build:
    runs-on: ubuntu-latest

This next bit creates a new job called build. Here we're saying that we want to use the latest version of Ubuntu to run our tests on. Ubuntu is common, so you'll only want to customize this if you want to run it on a specific environment.

    strategy:
      matrix:
        node-version: [10.x, 12.x, 14.x]

Inside of our job we specify a strategy matrix. This allows us to run the same tests on a few different variations.

In this instance, we're running the tests on 3 different versions of node to make sure it works on all of them. This is definitely helpful to make sure your code is flexible and future proof, but if you're building and running your code on a specific node version, you're safe to change this to only that version.

    steps:
    - uses: actions/checkout@v2
    - name: Use Node.js ${{ matrix.node-version }}
      uses: actions/setup-node@v1
      with:
        node-version: ${{ matrix.node-version }}
    - run: npm ci
    - run: npm run build --if-present
    - run: npm test

Finally, we specify the steps we want our job to run. Breaking this down:

• uses: actions/checkout@v2: In order for us to run our code, we need to have it available. This checks out our code on our job environment so we can use it to run tests.
• uses: actions/setup-node@v1: Since we're using node with our project, we'll need it set up on our environment. We're using this action to do that setup for us for each version we've specified in the matrix we configured above.
• run: npm ci: If you're not familiar with npm ci, it's similar to running npm install but uses the package-lock.json file without performing any patch upgrades. So essentially, this installs our dependencies.
• run: npm run build --if-present: npm run build runs the build script in our project. The --if-present flag performs what it sounds like and only runs this command if the build script is present. It doesn't hurt anything to leave this in as it won't run without the script, but feel free to remove this as we're not building the project here.
• run: npm test: Finally, we run npm test to run our tests. This uses the test npm script set up in our package.json file.

And with that, we've made a few tweaks, but our tests should run after we've committed those changes and pass like before!

Logs of passing tests in Github Action workflow

Follow along with the commit!

Step 3: Testing that our tests fail and prevent merges

Now that our tests are set up to automatically run, let's try to break it to see it work.

At this point, you can really do whatever you want to intentionally break the tests, but here's what I did:

Code diff - https://github.com/colbyfayock/my-github-actions/pull/1

I'm intentionally returning different expected output so that my tests will fail. And they do!

Failing status checks on pull request

In my new pull request, my new branch breaks the tests, so it tells me my checks have failed. If you noticed though, it's still green to merge, so how can we prevent merges?

We can prevent pull requests from being merged by setting up a Protected Branch in our project settings.

First, navigate to Settings, then Branches, and click Add rule.

Github branch protection rules

We'll then want to set the branch name pattern to *, which means all branches, check the Require status checks to pass before merging option, then select all of our different status checks that we'd like to require to pass before merging.

Setting up a branch protection rule in Github

Finally, hit Create at the bottom of the page.

And once you navigate back to the pull request, you'll notice that the messaging is a bit different and states that we need our statuses to pass before we can merge.

Failing tests preventing merge in pull request

Note: as an administrator of a repository, you'll still be able to merge, so this technically only prevents non-administrators from merging. But will give you increased messaging if the tests fail.

And with that, we have a new Github Action that runs our tests and prevents pull requests from merging if they fail.

Follow along with the pull request!

Note: we won't be merging that pull request before continuing to Part 2.

Part 2: Post new pull requests to Slack

Now that we're preventing merge requests if they're failing, we want to post a message to our Slack workspace whenever a new pull request is opened up. This will help us keep tabs on our repos right in Slack.

For this part of the guide, you'll need a Slack workspace that you have permissions to create a new developer app with and the ability to create a new channel for the bot user that will be associated with that app.

Step 1: Setting up Slack

There are a few things we're going to walk through as we set up Slack for our workflow:

• Create a new app for our workspace
• Assign our bot permissions
• Install our bot to our workspace
• Invite our new bot to our channel

To get started, we'll create a new app. Head over to the Slack API Apps dashboard. If you already haven't, log in to your Slack account with the Workspace you'd like to set this up with.

Creating a new Slack app

Now, click Create New App where you'll be prompted to put in a name and select a workspace you want this app to be created for. I'm going to call my app "Gitbot" as the name, but you can choose whatever makes sense for you. Then click Create App.

Configuring a new Slack app

Once created, navigate to the App Home link in the left sidebar. In order to use our bot, we need to assign it OAuth scopes so it has permissions to work in our channel, so select Review Scopes to Add on that page.

Reviewing Slack app scopes

Scroll own and you'll see a Scopes section and under that a Bot Token section. Here, click Add an OAuth Scope. For our bot, we don't need a ton of permissions, so add the channels:join and chat:write scopes and we should be good to go.

Adding scopes for a Slack app Bot Token

Now that we have our scopes, let's add our bot to our workspace. Scroll up on that same page to the top and you'll see a button that says Install App to Workspace.

Installing Slack app to a workspace

Once you click this, you'll be redirected to an authorization page. Here, you can see the scopes we selected for our bot. Next, click Allow.

Allowing permission for Slack app to be installed to workspace

At this point, our Slack bot is ready to go. At the top of the OAuth & Permissions page, you'll see a Bot User OAuth Access Token. This is what we'll use when setting up our workflow, so either copy and save this token or remember this location so you know how to find it later.

Note: this token is private - don't give this out, show it in a screencast, or let anyone see it!

Copying OAuth Access Token for Slack bot user

Finally, we need to invite our Slack bot to our channel. If you open up your workspace, you can either use an existing channel or create a new channel for these notifications, but you'll want to enter the command /invite @[botname] which will invite our bot to our channel.

Inviting Slack bot user to channel

And once added, we're done with setting up Slack!

Slack bot was added to channel

Create a Github Action to notify Slack

Our next step will be somewhat similar to when we created our first Github Action. We'll create a workflow file which we'll configure to send our notifications.

While we can use our code editors to do this by creating a file in the .github directory, I'm going to use the Github UI.

First, let's navigate back to our Actions tab in our repository. Once there, select New workflow.

Setting up a new Github Action workflow

This time, we're going to start the workflow manually instead of using a pre-made Action. Select set up a workflow yourself at the top.

Setting up a Github Action workflow manually

Once the new page loads, you'll be dropped in to a new template where we can start working. Here's what our new workflow will look like:

name: Slack Notifications

on:
  pull_request:
    branches: [ master ]

jobs:
  notifySlack:

    runs-on: ubuntu-latest

    steps:
    - name: Notify slack
      env:
        SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }}
      uses: abinoda/slack-action@master
      with:
        args: '{\"channel\":\"[Channel ID]\",\"blocks\":[{\"type\":\"section\",\"text\":{\"type\":\"mrkdwn\",\"text\":\"*Pull Request:* ${{ github.event.pull_request.title }}\"}},{\"type\":\"section\",\"text\":{\"type\":\"mrkdwn\",\"text\":\"*Who?:* ${{ github.event.pull_request.user.login }}\n*Request State:* ${{ github.event.pull_request.state }}\"}},{\"type\":\"section\",\"text\":{\"type\":\"mrkdwn\",\"text\":\"<${{ github.event.pull_request.html_url }}|View Pull Request>\"}}]}'

So what's happening in the above?

• name: we're setting a friendly name for our workflow
• on: we want our workflow to trigger when there's a pull request is created that targets our master branch
• jobs: we're creating a new job called notifySlack
• jobs.notifySlack.runs-on: we want our job to run on a basic setup of the latest Unbuntu
• jobs.notifySlack.steps: we really only have one step here - we're using a pre-existing Github Action called Slack Action and we're configuring it to publish a notification to our Slack

There are two points here we'll need to pay attention to, the env.SLACK_BOT_TOKEN and the with.args.

In order for Github to communicate with Slack, we'll need a token. This is what we're setting in env.SLACK_BOT_TOKEN. We generated this token in the first step. Now that we'll be using this in our workflow configuration, we'll need to add it as a Git Secret in our project.

_Github secrets including SLACK_BOTTOKEN

The with.args property is what we use to configure the payload to the Slack API that includes the channel ID (channel) and our actual message (blocks).

The payload in the arguments is stringified and escaped. For example, when expanded it looks like this:

{
  "channel": "[Channel ID]",
  "blocks": [{
    "type": "section",
    "text": {
      "type": "mrkdwn",
      "text": "*Pull Request:* ${{ github.event.pull_request.title }}"
    }
  }, {
    "type": "section",
    "text": {
      "type": "mrkdwn",
      "text": "*Who?:*n${{ github.event.pull_request.user.login }}n*State:*n${{ github.event.pull_request.state }}"
    }
  }, {
    "type": "section",
    "text": {
      "type": "mrkdwn",
      "text": "<${{ github.event.pull_request._links.html.href }}|View Pull Request>"
    }
  }]
}

Note: this is just to show what the content looks like, we need to use the original file with the stringified and escaped argument.

Back to our configuration file, the first thing we set is our channel ID. To find our channel ID, you'll need to use the Slack web interface. Once you open Slack in your browser, you want to find your channel ID in the URL:

https://app.slack.com/client/[workspace ID]/[channel ID]

Channel ID in Slack web app URL

With that channel ID, you can modify our workflow configuration and replace [Channel ID] with that ID:

with:
  args: '{\"channel\":\"C014RMKG6H2\",...

The rest of the arguments property is how we set up our message. It includes variables from the Github event that we use to customize our message.

We won't go into tweaking that here, as what we already have will send a basic pull request message, but you can test out and build your own payload with Slack's Block Kit Builder.

Follow along with the commit!

Test out our Slack workflow

So now we have our workflow configured with our Slack app, finally we're ready to use our bot!

For this part, all we need to do is create a new pull request with any change we want. To test this out, I simply created a new branch where I added a sentence to the README.md file.

Code diff - https://github.com/colbyfayock/my-github-actions/pull/2

Once you create that pull request, similar to our tests workflow, Github will run our Slack workflow! You can see this running in the Actions tab just like before.

As long as you set everything up correctly, once the workflow runs, you should now have a new message in Slack from your new bot.

Slack bot automated message about new pull request

Note: we won't be merging that pull request in.

What else can we do?

Customize your Slack notifications

The message I put together is simple. It tells us who created the pull request and gives us a link to it.

To customize the formatting and messaging, you can use the Github Block Kit Builder to create your own.

If you'd like to include additional details like the variables I used for the pull request, you can make use of Github's available contexts. This lets you pull information about the environment and the job to customize your message.

I couldn't seem to find any sample payloads, so here's an example of a sample github context payload you would expect in the event.

Sample github context

More Github actions

With our ability to create new custom workflows, that's not a lot we can't automate. Github even has a marketplace where you can browse around for one.

If you're feeling like taking it a step further, you can even create your own! This lets you set up scripts to configure a workflow to perform whatever tasks you need for your project.

Join in the conversation!

What do you use Github actions for?

Share with me on Twitter!

Resources. When they are unlimited they are not important. But when they're limited, boy do you have challenges!

Recently, my team has faced such a challenge ourselves: we realised that we needed to upgrade the Node version on one of our Jenkins agents so we could build and properly test our Angular 7 app. However, we learned that we would also lose the ability to build our legacy AngularJS apps which require Node 8.

What were we to do?

Apart from eliminating the famous "It works on my machine" problem, Docker came in handy to tackle such a problem. However, there were certain challenges that needed to be addressed, such as Docker in Docker.

For this purpose, after a long period of trial and error, we built and published a docker file that fit our team's needs. It helps run our builds, and it looks like the following:

1. Install dependencies
2. Lint the code
3. Run unit tests
4. Run SonarQube analysis
5. Build the application
6. Build a docker image which would be deployed
7. Run the docker container
8. Run cypress tests
9. Push docker image to the repository
10. Run another Jenkins job to deploy it to the environment
11. Generate unit and functional test reports and publish them
12. Stop any running containers
13. Notify chat/email about the build

The docker image we needed

Our project is an Angular 7 project, which was generated using the angular-cli. We also have some dependencies that need Node 10.x.x. We lint our code with tslint, and run our unit tests with Karma and Jasmine. For the unit tests we need a Chrome browser installed so they can run with headless Chrome.

This is why we decided to use the cypress/browsers:node10.16.0-chrome77 image. After we installed the dependencies, linted our code and ran our unit tests, we ran the SonarQube analysis. This required us to have Openjdk 8 as well.

FROM cypress/browsers:node10.16.0-chrome77

# Install OpenJDK-8
RUN apt-get update && \
    apt-get install -y openjdk-8-jdk && \
    apt-get install -y ant && \
    apt-get clean;

# Fix certificate issues
RUN apt-get update && \
    apt-get install ca-certificates-java && \
    apt-get clean && \
    update-ca-certificates -f;

# Setup JAVA_HOME -- useful for docker commandline
ENV JAVA_HOME /usr/lib/jvm/java-8-openjdk-amd64/
RUN export JAVA_HOME

Once the sonar scan was ready, we built the application. One of the strongest principles in testing is that you should test the thing that will be used by your users.
That is the reason that we wanted to test the built code in exactly the same docker container as it would be in production.

We could, of course serve the front-end from a very simple nodejs static server.
But that would mean that everything an Apache HTTP server or an NGINX server usually did would be missing (for example all the proxies, gzip or brotli).

Now while this is a strong principle, the biggest problem was that we were already running inside a Docker container. That is why we needed DIND (Docker in Docker).

After spending a whole day with my colleague researching, we found a solution which ended up working like a charm. The first and most important thing is that our build container needed the Docker executable.

# Install Docker executable
RUN apt-get update && apt-get install -y \
        apt-transport-https \
        ca-certificates \
        curl \
        gnupg2 \
        software-properties-common \
    && curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add - \
    && add-apt-repository \
        "deb [arch=amd64] https://download.docker.com/linux/debian \
        $(lsb_release -cs) \
        stable" \
    && apt-get update \
    && apt-get install -y \
        docker-ce

RUN usermod -u 1002 node && groupmod -g 1002 node && gpasswd -a node docker

As you can see we installed the docker executable and the necessary certificates, but we also added the rights and groups for our user. This second part is necessary because the host machine, our Jenkins agent, starts the container with -u 1002:1002. That is the user ID of our Jenkins agent which runs the container unprivileged.

Of course this isn't everything. When the container starts, the docker daemon of the host machine must be mounted. So we needed to start the build container
with some extra parameters. It looks like the following in a Jenkinsfile:

pipeline {
  agent {
    docker {
     image 'btapai/pipelines:node-10.16.0-chrome77-openjdk8-CETtime-dind'
     label 'frontend'
     args '-v /var/run/docker.sock:/var/run/docker.sock -v /var/run/dbus/system_bus_socket:/var/run/dbus/system_bus_socket -e HOME=${workspace} --group-add docker'
    }
  }

// ...
}

As you can see, we mounted two Unix sockets. /var/run/docker.sock mounts the docker daemon to the build container.

/var/run/dbus/system_bus_socket is a socket that allows cypress to run inside our container.

We needed -e HOME=${workspace} to avoid access rights issues during the build.

--group-add docker passes the host machines docker group down, so that inside the container our user can use the docker daemon.

With these proper arguments, we were able to build our image, start it up and run our cypress tests against it.

But let's take a deep breath here. In Jenkins, we wanted to use multi-branch pipelines. Multibranch pipelines in Jenkins would create a Jenkins job for each branch that contained a Jenkinsfile. This meant that when we developed multiple branches they would have their own views.

There were some problems with this. The first problem was that if we built our image with the same name in all the branches, there would be conflicts (since our docker daemon was technically not inside our build container).

The second problem arose when the docker run command used the same port in every build (because you can't start the second container on a port that is already taken).

The third issue was getting the proper URL for the running application, because Dorothy, you are not in Localhost anymore.

Let's start with the naming. Getting a unique name is pretty easy with git, because commit hashes are unique. However, to get a unique port we had to use a little trick when we declared our environment variables:

pipeline {

// ..

  environment {
    BUILD_PORT = sh(
        script: 'shuf -i 2000-65000 -n 1',
        returnStdout: true
    ).trim()
  }

// ...

    stage('Functional Tests') {
      steps {
        sh "docker run -d -p ${BUILD_PORT}:80 --name ${GIT_COMMIT} application"
        // be patient, we are going to get the url as well. :)
      }
    }

// ...

}

With the shuf -i 2000-65000 -n 1 command on certain Linux distributions you can generate a random number. Our base image uses Debian so we were lucky here.
The GIT_COMMIT environment variable was provided in Jenkins via the SCM plugin.

Now came the hard part: we were inside a docker container, there was no localhost, and the network inside docker containers can change.

It was also funny that when we started our container, it was running on the host machine's docker daemon. So technically it was not running inside our container. We had to reach it from the inside.

After several hours of investigation my colleague found a possible solution:
docker inspect --format "{{ .NetworkSettings.IPAddress }}"

But it did not work, because that IP address was not an IP address inside the container, but rather outside it.

Then we tried the NetworkSettings.Gateway property, which worked like a charm.
So our Functional testing stage looked like the following:

stage('Functional Tests') {
  steps {
    sh "docker run -d -p ${BUILD_PORT}:80 --name ${GIT_COMMIT} application"
    sh 'npm run cypress:run -- --config baseUrl=http://`docker inspect --format "{{ .NetworkSettings.Gateway }}" "${GIT_COMMIT}"`:${BUILD_PORT}'
  }
}

It was a wonderful feeling to see our cypress tests running inside a docker container.

But then some of them failed miserably. Because the failing cypress tests expected to see some dates.

cy.get("created-date-cell")
  .should("be.visible")
  .and("contain", "2019.12.24 12:33:17")

But because our build container was set to a different timezone, the displayed date on our front-end was different.

Fortunately, it was an easy fix, and my colleague had seen it before. We installed the necessary time zones and locales. In our case we set the build container's timezone to Europe/Budapest, because our tests were written in this timezone.

# SETUP-LOCALE
RUN apt-get update \
    && apt-get install --assume-yes --no-install-recommends locales \
    && apt-get clean \
    && sed -i -e 's/# en_US.UTF-8 UTF-8/en_US.UTF-8 UTF-8/' /etc/locale.gen \
    && sed -i -e 's/# hu_HU.UTF-8 UTF-8/hu_HU.UTF-8 UTF-8/' /etc/locale.gen \
    && locale-gen

ENV LANG="en_US.UTF-8" \
    LANGUAGE= \
    LC_CTYPE="en_US.UTF-8" \
    LC_NUMERIC="hu_HU.UTF-8" \
    LC_TIME="hu_HU.UTF-8" \
    LC_COLLATE="en_US.UTF-8" \
    LC_MONETARY="hu_HU.UTF-8" \
    LC_MESSAGES="en_US.UTF-8" \
    LC_PAPER="hu_HU.UTF-8" \
    LC_NAME="hu_HU.UTF-8" \
    LC_ADDRESS="hu_HU.UTF-8" \
    LC_TELEPHONE="hu_HU.UTF-8" \
    LC_MEASUREMENT="hu_HU.UTF-8" \
    LC_IDENTIFICATION="hu_HU.UTF-8" \
    LC_ALL=

# SETUP-TIMEZONE
RUN apt-get update \
    && apt-get install --assume-yes --no-install-recommends tzdata \
    && apt-get clean \
    && echo 'Europe/Budapest' > /etc/timezone && rm /etc/localtime \
    && ln -snf /usr/share/zoneinfo/'Europe/Budapest' /etc/localtime \
    && dpkg-reconfigure -f noninteractive tzdata

Since every crucial part of the build was now resolved, pushing the built image to the registry was just a docker push command. You can check out the whole dockerfile here.

One thing remained, which was to stop running containers when the cypress tests failed. We did this easily using the always post step.

post {
  always {
    script {
      try {
        sh "docker stop ${GIT_COMMIT} && docker rm ${GIT_COMMIT}"
      } catch (Exception e) {
        echo 'No docker containers were running'
      }
    }
  }
}

Thank you very much for reading this blog post. I hope it helps you.

The original article can be read on my blog: