These terms might seem complex or interchangeable at first glance, leaving you wondering: What do they actually mean? How do they differ from one another? 🤔

In this handbook, I’ll break down these concepts in a clear and approachable way, drawing on relatable analogies to make each term easier to understand. 🧠💡 Beyond just theory, we’ll dive into a hands-on tutorial where you’ll learn how to set up a CI/CD workflow step by step.

Together, we’ll:

• Set up a Node.js project. ✨

• Implement automated tests using Jest and Supertest. 🛠️

• Set up a CI/CD workflow using GitHub Actions, triggered on push, and pull requests, or after a new release. ⚙️

• Build and publish a Docker image of your application to Docker Hub. 📦

• Deploy your application to a staging environment for testing. 🚀

• Finally, roll it out to a production environment, making it live! 🌐

By the end of this guide, not only will you understand the difference between CI/CD concepts, but you’ll also have practical experience in building your own automated pipeline. 😃

Table of Contents

1. What is Continuous Integration, Deployment, and Delivery?

2. Differences Between Continuous Integration, Continuous Delivery, and Continuous Deployment

3. How to Set Up a Node.js Project with a Web Server and Automated Tests

4. How to Create a GitHub Repository to Host Your Codebase

5. How to Set Up the CI and CD Workflows Within Your Project

6. Set Up a Docker Hub Repository for the Project's Image and Generate an Access Token for Publishing the Image

7. Create a Google Cloud Account, Project, and Billing Account

8. Create a Google Cloud Service Account to Enable Deployment of the Node.js Application to Google Cloud Run via the CD Pipeline

9. Create the Staging Branch and Merge the Feature Branch into It (Continuous Integration and Continuous Delivery)

10. Merge the Staging Branch into the Main Branch (Continuous Integration and Continuous Deployment)

11. Conclusion

What is Continuous Integration, Deployment, and Delivery? 🤔

Continuous Integration (CI)

Imagine you’re part of a team of six developers, all working on the same project. Without a proper system, chaos would ensue.

Let’s say Mr. A is building a new login feature, Mrs. B is fixing a bug in the search bar, and Mr. C is tweaking the dashboard UI—all at the same time. If everyone is editing the same "folder" or codebase directly, things could go horribly wrong: "Hey! Who just broke the app?!" 😱

To keep everything in order, teams use Version Control Systems (VCS) like GitHub, GitLab, or BitBucket. Think of it as a digital workspace where everyone can safely collaborate without stepping on each other’s toes. 🗂️✨

Here’s how Continuous Integration fits into this process step-by-step:

1. The Main Branch: The General Folder ✨

At the heart of every project is the main branch—the ultimate source of truth. It contains the stable codebase that powers your live app. It’s where every team member contributes their work, but with one important rule: only tested and approved code gets merged here. 🚀

2. Feature Branches: Personal Workspaces 🔨

When someone like Mr. A wants to work on a new feature, they create a feature branch. This branch is essentially a personal copy of the main branch where they can tinker, write code, and test without affecting others. Mrs. B and Mr. C are also working on their own branches. Everyone’s experiments stay neatly organized. 🧪💡

3. Merging Changes: The CI Workflow 🎉

When Mr. A is satisfied with his feature, he doesn’t just shove it into the main branch—CI ensures it’s done safely:

• Automated Tests: Before merging, CI tools automatically run tests on Mr. A’s code to check for bugs or errors. Think of it as a bouncer guarding the main branch, ensuring no bad code gets in. 🕵️‍♂️

• Build Verification: The feature branch code is also "built" (converted into a deployable version of the app) to confirm it works as intended.

Once these checks are passed, Mr. A’s feature branch is merged into the main branch. This frequent merging of changes is what we call Continuous Integration.

Continuous Delivery (CD)

Continuous Delivery (CD) often gets mixed up with Continuous Deployment, and while they share similarities, they serve distinct purposes in the development lifecycle. Let’s break it down! 🧐

The Need for a Staging Area 🌉

In the Continuous Integration (CI) process we discussed above, we primarily dealt with feature branches and the main branch. But directly merging changes from feature branches into the main branch (which powers the live product) can be risky. Why? 🛑

While automated tests and builds catch many errors, they’re not foolproof. Some edge cases or bugs might slip through unnoticed. This is where the staging branch and staging environment come into play! 🎭

Think of the staging branch as a “trial run.” Before unleashing changes to real customers, the codebase from feature branches is merged into the staging branch and deployed to a staging environment. This environment is an exact replica of the production environment, but it’s used exclusively by the Quality Assurance (QA) team for testing.

The QA team takes the role of a “test driver,” running the platform through its paces just as a real user would. They check for usability issues, edge cases, or bugs that automated tests might miss, and provide feedback to developers for fixes. 🚦 If everything passes, the codebase is cleared for deployment to production.

Continuous Delivery in Action 📦

The process of merging changes into the staging branch and deploying them to the staging environment is what we call Continuous Delivery. 🛠️ It ensures that the application is always in a deployable state, ready for the next step in the pipeline.

Unlike Continuous Deployment (which we’ll discuss later), Continuous Delivery doesn’t automatically push changes to production (live platform). Instead, it pauses to let humans—namely the QA team or stakeholders—decide when to proceed. This adds an extra layer of quality assurance, reducing the chances of errors making it to the live product. 🕵️‍♂️

Continuous Deployment (CD)

Continuous Deployment (CD) takes automation to its peak. While it shares similarities with Continuous Delivery, the key difference lies in the final step: there’s no manual approval required. The final process—merging the codebase and deploying it live for end users (the QA testers or the team lead could do this).

Let’s explore what makes Continuous Deployment so powerful (and a little scary)! 😅

The Last Mile of the CI/CD Pipeline 🛣️

Imagine you’ve gone through the rigorous process of Continuous Integration: teammates have merged their feature branches, automated tests were run, and the codebase was successfully deployed to the staging environment during Continuous Delivery.

Now, you’re confident that the application is free of bugs and ready to shine in the production environment—the live version of your platform used by real customers.

In Continuous Deployment, this final step of deploying changes to the live environment happens automatically. The pipeline triggers whenever specific events occur, such as:

• A Pull Request (PR) is merged into the main branch.

• A new release version is created.

• A commit is pushed directly to the production branch (though this is rare for most teams).

Once triggered, the pipeline springs into action, building, testing, and finally deploying the updated codebase to the production environment. 📡

Differences Between Continuous Integration, Continuous Delivery, and Continuous Deployment 🔍

Now that we’ve untangled the mysteries of Continuous Integration, Continuous Delivery, and Continuous Deployment, it’s time to roll up our sleeves and put theory into practice 😁.

How to Set Up a Node.js Project with a Web Server and Automated Tests ✨

In this hands-on section, we’ll build a Node.js web server with automated tests using Jest. From there, we’ll create a CI/CD pipeline with GitHub Actions that automates testing for every pull request to the staging and main branches. Finally, we’ll publish an Image of our application to DockerHub and deploy the image to Google Cloud Run, first to a staging environment for testing and later to the production environment for live use.

Ready to bring your project to life? Let’s get started! 🚀✨

Step 1: Install Node.js 📥

To get started, you’ll need to have Node.js installed on your machine. Node.js provides the JavaScript runtime we’ll use to create our web server.

1. Visit https://nodejs.org/en/download/package-manager

2. Choose your operating system (Windows, macOS, or Linux) and download the installer.

3. Follow the installation instructions to complete the setup.

To verify that Node.js was installed successfully, open your terminal and run node -v. This should display the installed version of Node.js

Step 2: Clone the Starter Repository 📂

The next step is to grab the starter code from GitHub. If you don’t have Git installed, you can download it at https://git-scm.com/downloads. Choose your OS and follow the instructions to install Git. Once you’re set, it’s time to clone the repository.

Run the following command in your terminal to clone the boilerplate code:

git clone --single-branch --branch initial https://github.com/onukwilip/ci-cd-tutorial

This will download the project files from the initial branch, which contains the starter template for our Node.js web server.

Navigate into the project directory:

cd ci-cd-tutorial

Step 3: Install Dependencies 📦

Once you’re in the project directory, install the required dependencies for the Node.js project. These are the packages that power the application:

npm install --force

This will download and set up all the libraries specified in the project. Alright, dependencies installed? You’re one step closer!

Step 4: Run Automated Tests ✅

Before diving into the code, let’s confirm that the automated tests are functioning correctly. Run:

npm test

You should see two successful test results in your terminal. This indicates that the starter project is correctly configured with working automated tests.

Step 5: Start the Web Server 🌐

Finally, let’s start the web server and see it in action. Run the following command:

npm start

Wait for the application to start running. Open your browser and visit http://localhost:5000. 🎉 You should see the starter web server up and running, ready for your CI/CD magic:

How to Create a GitHub Repository to Host Your Codebase 📂

Step 1: Sign In to GitHub

1. Go to GitHub: Open your browser and visit GitHub - https://github.com.

2. Sign In: Click on the Sign In button in the top-right corner and enter your username and password to log in, OR create an account if you don’t have one by clicking the Sign up button.

Step 2: Create a New Repository

Once you're signed in, on the main GitHub page, you’ll see a "+" sign in the top-right corner next to your profile picture. Click on it, and select “New repository” from the dropdown.

Now it’s time to set the repository details. You’ll include:

• Repository Name: Choose a name for your repository. For example, you can call it ci-cd-tutorial.

• Description (Optional): You can add a short description, like “A tutorial project for CI/CD with Docker and GitHub Actions.”

• Visibility: Choose whether you want your repository to be public (accessible by anyone) or private (only accessible by you and those you invite). For the sake of this tutorial, make it public.

• Do Not Check the Add a README File Box: Important: Make sure you do not check the option to Add a README file. This will automatically create a README.md file in your repository, which could cause conflicts later when you push your local files. We'll add the README file manually if needed later.

After filling out the details, click on “Create repository”.

Step 3: Change the Remote Destination and Push to Your New Repository

Update the Remote Repository URL:

Since you've already cloned the codebase from my repository, you need to update the remote destination to point to your newly created GitHub repository.

Copy your repository URL (the URL of the page you were redirected to after creating the repository). It should look similar to this: https://github.com/<username>/<repo-name>.

Open your terminal in the project directory and run the following commands:

git remote set-url origin <your-repo-url>

Replace <your-repo-url> with your GitHub repository URL which you copied earlier.

Rename the Current Branch to main:

If your branch is named something other than main, you can rename it to main using:

git branch -M main

Push to Your New Repository:

Finally, commit any changes you’ve made and push your local repository to the new remote GitHub repository by running:

git add .
git commit -m 'Created boilerplate'
git push -u origin main

Now your local codebase is linked to your new GitHub repository, and the files are successfully pushed there. You can verify by visiting your repository on GitHub.

How to Set Up the CI and CD Workflows Within Your Project ⚙️

Now it’s time to create the CI and CD workflows for our project! These workflows won’t run on your local PC but will be automatically triggered and executed in the cloud once you push your changes to the remote repository. GitHub Actions will detect these workflows and run them based on the triggers you define.

Step 1: Prepare the Workflow Directory 📂

Before adding the CI/CD pipelines, it's a good practice to first create a feature branch. This step mirrors the workflow commonly used in teams, where new features or changes are made in separate branches before they are merged into the main codebase.

To create and switch to a new branch, run the following command:

git checkout -b feature/ci-cd-pipeline

This will create a new branch called feature/ci-cd-pipeline and switch to it. Now, you can safely add and test the CI/CD workflows without affecting the main branch.

Once you finish, you’ll be able to merge this feature branch back into main or staging as part of the pull request process.

In the project’s root directory, create a folder named .github. Inside .github, create another folder called workflows.

Any YAML file placed in the .github/workflows directory is automatically recognized as a GitHub Actions workflow. These workflows will execute based on specific triggers, such as pull requests, pushes, or releases.

Step 2: Create the Continuous Integration Workflow 🚀

We’ll now create a CI workflow that automatically tests the application whenever a pull request is made to the main or staging branches.

First, inside the workflows directory, create a file named ci-pipeline.yml.

Paste the following code into the file:

name: CI Pipeline to staging/production environment
on:
  pull_request:
    branches:
      - staging
      - main
jobs:
  test:
    runs-on: ubuntu-latest
    name: Setup, test, and build project
    env:
      PORT: 5001
    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Install dependencies
        run: npm ci

      - name: Test application
        run: npm test

      - name: Build application
        run: |
          echo "Run command to build the application if present"
          npm run build --if-present

Explanation of the CI Workflow

Here’s a breakdown of each section in the workflow:

1. name: CI Pipeline to staging/production environment: This is the title of your workflow. It helps you identify this pipeline in GitHub Actions.

2. on: The on parameter is what determines the events that trigger your workflow. When the workflow YAML file is pushed to the remote GitHub repository, GitHub Actions automatically registers the workflow using the configured triggers in the on field. These triggers act as event listeners that tell GitHub when to execute the workflow

   For example:

   If we set pull_request as the value for the on parameter and specify the branches we want to monitor using the branches key, GitHub sets up event listeners for pull requests to those branches.

    on:
      pull_request:
        branches:
          - main
          - staging
   

   This configuration means that GitHub will trigger the workflow whenever a pull request is made to the main or staging branches.

   Multiple Triggers:
   You can define multiple event listeners in the on parameter. For instance, in addition to pull requests, you can add a listener for push events.

    on:
      pull_request:
        branches:
          - main
          - staging
      push:
        branches:
          - main
   

   This configuration ensures that the workflow is triggered when:

   • A pull request is made to either the main or staging branch.

   • A push is made directly to the main branch.

📘 Learn more about triggers: Check out the official GitHub documentation here.

3. jobs: The jobs section outlines the specific tasks (or jobs) that the workflow will execute. Each job is an independent unit of work that runs on a separate virtual machine (VM). This isolation ensures a clean, unique environment for every job, avoiding potential conflicts between tasks.

   Key Points About Jobs:

   1. Clean VM for Each Job: When GitHub Actions runs a workflow, it assigns a dedicated VM instance to each job. This means the environment is reset for every job, ensuring there’s no overlap or interference between tasks.

   2. Multiple Jobs: Workflows can have multiple jobs, each responsible for a specific task. For example:

      • A Test job to install dependencies and run automated tests.

      • A Build job to compile the application.

   3. Job Organization: Jobs can be organized to run:

      • Sequentially: Ensures one job is completed before the next starts, for example the Test job must finish before the Build job. This sequential flow mimics the "pipeline" structure.

      • Simultaneously: Multiple jobs can run in parallel to save time, especially if the jobs are independent of one another.

   4. Single Job in This Workflow: In our current workflow, there is only one job, test, which:

      • Installs dependencies.

      • Runs automated tests.

      • Builds the application.

📘 Learn more about jobs: Dive into the GitHub Actions jobs documentation here.

4. runs-on: ubuntu-latest: Specifies the operating system the job will run on. GitHub provides pre-configured virtual environments, and we’re using the latest Ubuntu image.

5. env: Sets environment variables for the job. Here, we define the PORT variable used by our application.

6. Steps: Steps define the individual actions to execute within a job:

   • Checkout: Uses the actions/checkout action to clone the repository containing the codebase in the feature branch into the virtual machine instance environment. This step ensures the pipeline has access to the project files.

   • Install dependencies: Runs npm ci to install the required Node.js packages.

   • Test application: Runs the automated tests using the npm test command. This validates the codebase for errors or failing test cases.

   • Build application: Builds the application if a build script is defined in the package.json. The --if-present flag ensures this step doesn’t fail if no build script is present.

Now that we’ve completed the CI pipeline, which runs on pull requests to the main or staging branches, let’s move on to setting up the Continuous Delivery (CD) and Continuous Deployment pipelines. 🚀

Step 3: The Continuous Delivery and Deployment Workflow

First, create the Pipeline File:
In the .github/workflows folder, create a new file called cd-pipeline.yml. This file will define the workflows for automating delivery and deployment.

Next, paste the configuration:
Copy and paste the following configuration into the cd-pipeline.yml file:

name: CD Pipeline to Google Cloud Run (staging and production)
on:
  push:
    branches:
      - staging
  workflow_dispatch: {}
  release:
    types: published

env:
  PORT: 5001
  IMAGE: ${{vars.IMAGE}}:${{github.sha}}
jobs:
  test:
    runs-on: ubuntu-latest
    name: Setup, test, and build project
    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Install dependencies
        run: npm ci

      - name: Test application
        run: npm test
  build:
    needs: test
    runs-on: ubuntu-latest
    name: Setup project, Authorize GitHub Actions to GCP and Docker Hub, and deploy
    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Authenticate for GCP
        id: gcp-auth
        uses: google-github-actions/auth@v0
        with:
          credentials_json: ${{ secrets.GCP_SERVICE_ACCOUNT }}

      - name: Set up Cloud SDK
        uses: google-github-actions/setup-gcloud@v0

      - name: Authenticate for Docker Hub
        id: docker-auth
        env:
          D_USER: ${{secrets.DOCKER_USER}}
          D_PASS: ${{secrets.DOCKER_PASSWORD}}
        run: |
          docker login -u $D_USER -p $D_PASS
      - name: Build and tag Image
        run: |
          docker build -t ${{env.IMAGE}} .
      - name: Push the image to Docker hub
        run: |
          docker push ${{env.IMAGE}}
      - name: Enable the Billing API
        run: |
          gcloud services enable cloudbilling.googleapis.com --project=${{secrets.GCP_PROJECT_ID}}
      - name: Deploy to GCP Run - Production environment (If a new release was published from the master branch)
        if: github.event_name == 'release' && github.event.action == 'published' && github.event.release.target_commitish == 'main'
        run: |
          gcloud run deploy ${{vars.GCR_PROJECT_NAME}} \
          --region ${{vars.GCR_REGION}} \
          --image ${{env.IMAGE}} \
          --platform "managed" \
          --allow-unauthenticated \
          --tag production \
      - name: Deploy to GCP Run - Staging environment
        if: github.ref != 'refs/heads/main'
        run: |
          echo "Deploying to staging environment"
          # Deploy service with to staging environment
          gcloud run deploy ${{vars.GCR_STAGING_PROJECT_NAME}} \
          --region ${{vars.GCR_REGION}} \
          --image ${{env.IMAGE}} \
          --platform "managed" \
          --allow-unauthenticated \
          --tag staging \

The CD pipeline configuration combines Continuous Delivery and Continuous Deployment workflows into a single file for simplicity. It builds on the concepts of CI/CD we discussed earlier, automating testing, building, and deploying the application to Google Cloud Run.

Explanation of the CD pipeline:

1. Workflow Triggers (on)

• push: Workflow triggers on pushes to the staging branch.

• workflow_dispatch: Enables manual execution of the workflow via the GitHub Actions interface.

• release: Triggers when a new release is published.
  Example: When a release is published from the main branch, the app deploys to the production environment.

2. Job 1 – Testing the Codebase: The first job in the pipeline, Test, ensures the codebase is functional and error-free before proceeding with delivery or deployment

3. Job 2 – Building and Deploying the Application: Aha! Moment ✨: These jobs run sequentially. 😃 The Build job begins only after the Test job is completed successfully. It prepares the application for deployment and manages the actual deployment process.

   Here's what happens:

   • Authorization for GCP and Docker Hub: The workflow authenticates with both Google Cloud Platform (GCP) and Docker Hub. For GCP, it uses the google-github-actions/auth@v0 action to handle service account credentials stored as secrets. Similarly, it logs into Docker Hub with stored credentials to enable image uploads.

   • Build and Push Docker Image: The application is built into a Docker image and tagged with a unique identifier (${{env.IMAGE}}). This image is then pushed to Docker Hub, making it accessible for deployment.

   • Deploy to Google Cloud Run: Based on the event that triggered the workflow, the application is deployed to either the staging or production environment in Google Cloud Run. A push to the staging branch deploys to the staging environment (Continuous Delivery), while a release from the main branch deploys to production (Continuous Deployment).

To ensure the security and flexibility of our pipeline, we rely on external variables and secrets rather than hardcoding sensitive information directly into the workflow file.

Why? Workflow configuration files are part of your repository and accessible to anyone with access to the codebase. If sensitive data, like API keys or passwords, is exposed here, it can be easily compromised. 😨

Instead, we use GitHub’s Secrets to securely store and access this information. Secrets allow us to define variables that are encrypted and only accessible by our workflows. For example:

• DockerHub Credentials: We’ll add a Docker username and access token to the repository’s secrets. These are essential for authenticating with DockerHub to upload the built Docker images.

• Google Cloud Service Account Key: This key will grant the pipeline the necessary permissions to deploy the application on Google Cloud Run securely.

We'll set up these variables and secrets incrementally as we proceed, ensuring each step is fully secure and functional. 🎯

Set Up a Docker Hub Repository for the Project's Image and Generate an Access Token for Publishing the Image 📦

Before we dive into the steps, let’s quickly go over what we’re about to do. In this section, you’ll learn how to create a Docker Hub repository, which acts like an online storage space for your application’s container image.

Think of a container image as a snapshot of your application, ready to be deployed anywhere. To ensure smooth and secure access, we’ll also generate a special access token, kind of like a revokable password that our CI/CD pipeline can use to upload your app’s image to Docker Hub. Let’s get started! 🚀

Step 1: Sign Up for Docker Hub

Here are the steps to follow to sign up for Docker Hub:

1. Go to the Docker Hub website: Open your web browser and visit Docker Hub - https://hub.docker.com/.

2. Create an account: On the Docker Hub homepage, you’ll see a button labelled "Sign Up" in the top-right corner. Click on it.

3. Fill in your details: You'll be asked to provide a few details like your username, email address, and password. Choose a strong password that you can remember.

4. Agree to the terms: You’ll need to check a box to agree to Docker’s terms of service. After that, click “Sign Up” to create your account.

5. Verify your email: Docker Hub will send you an email to verify your account. Open that email and click on the verification link to complete your account creation.

Step 2: Sign In to Docker Hub

After verifying your email, go back to Docker Hub, and click on "Sign In" at the top right. Then you can use the credentials you just created to log in.

Step 3: Generate an Access Token (for the CI/CD pipeline)

Now that you have an account, you can create an access token. This token will allow your GitHub Actions workflow to securely sign into Docker Hub and upload Docker images.

Once you’re logged into Docker Hub, click on your profile picture (or avatar) in the top right corner. This will open a menu. From the menu, click “Account Settings”.

Then in the left-hand menu of your account settings, scroll to the "Security" tab. This section is where you manage your tokens and passwords.

Now you’ll need to create a new access token. In the Security tab, you’ll see a link labelled “Personal access tokens” – click on it. Click the button labelled “Generate new token”.

You’ll be asked to give your token a description. You can name it something like "GitHub Actions CI/CD" so that you know what it's for.

After giving it a description, click on the “Access permissions dropdown“ and select “Read & Write“, or “Read, Write, Delete“. Click “Generate“

Now, you need to copy the credentials. After clicking the generate button, Docker Hub will create an access token. Immediately copy this token along with your username and save it somewhere safe, like in a file (don’t worry, we’ll add it to our GitHub secrets). You won’t be able to see this token again, so make sure you save it!

Step 4: Add the Token to GitHub as a Secret

To do this, open your GitHub repository where the codebase is hosted. In the GitHub repo, click on the Settings tab (located near the top of your repo page).

Then on the left sidebar, scroll down and click on “Secrets and Variables”, then choose “Actions”.

Here are the steps to create and manage your new secret:

1. Add a new secret: Click on the “New repository secret” button.

2. Set up the secret:

   • In the Name field, type DOCKER_PASSWORD.

   • In the Value field, paste the access token you copied earlier.

3. Save the secret: Finally, click Add secret to save your Docker access token securely in GitHub.

Then you’ll repeat the process for your Docker username. Create a new secret called DOCKER_USER and add your Docker username that you copied earlier.

And that’s it! Now your CI/CD pipeline can use this token to securely log in to Docker Hub and upload images automatically when triggered. 🎉

Step 5: Creating the Dockerfile for the Project

Before you can build and publish the Docker image to Docker Hub, you need to create a Dockerfile that contains the necessary instructions to build your application.

Follow the steps below to create the Dockerfile in the root folder of your project:

1. Navigate to your project’s root folder.

2. Create a new file named Dockerfile.

3. Open the Dockerfile in a text editor and paste the following content into it:

FROM node:18-slim

WORKDIR /app

COPY package.json .

RUN npm install -f

COPY . .

# EXPOSE 5001
EXPOSE 5001

CMD ["npm", "start"]

Explanation of the Dockerfile:

• FROM node:18-slim: This sets the base image for the Docker container, which is a slim version of the official Node.js image based on version 18.

• WORKDIR /app: Sets the working directory for the application inside the container to /app.

• COPY package.json .: Copies the package.json file into the working directory.

• RUN npm install -f: Installs the project dependencies using npm.

• COPY . .: Copies the rest of the project files into the container.

• EXPOSE 5001: This tells Docker to expose port 5001, which is the port our app will run on inside the container.

• CMD ["npm", "start"]: This sets the default command to start the application when the container is run, using npm start.

Create a Google Cloud Account, Project, and Billing Account ☁️

In this section, we’re laying the foundation for deploying our application to Google Cloud. First, we’ll set up a Google Cloud account (don’t worry, it’s free to get started!). Then, we’ll create a new project where all the resources for your app will live.

Finally, we’ll enable billing so you can unlock the cloud services needed for deployment. Think of this as setting up your workspace in the cloud—organized, ready, and secure! Let’s dive in! ☁️

Step 1: Create or Sign in to a Google Cloud Account 🌐

First, go to Google Cloud Console. If you don’t have a Google Cloud account, you’ll need to create one.

To do this, click on Get Started for Free and follow the steps to set up your account (you’ll need to provide payment information, but Google offers $300 in free credits to get started). If you already have a Google account, simply sign in using your credentials.

Once you’ve signed in, you’ll be taken to your Google Cloud dashboard. This is where you can manage all your cloud projects and resources.

Step 2: Create a New Google Cloud Project 🏗️

At the top left of the Google Cloud Console, you’ll see a drop-down menu beside the Google Cloud logo. Click on this drop-down to display your current projects.

Now it’s time to create a new project. In the top-left corner of the pop-up modal, click on the New Project button.

You’ll be redirected to a page where you’ll need to provide some basic details for your new project. So now enter the following information:

• Project Name: Enter a name of your choice for the project (for example, gcr-ci-cd-project).

• Location: Select a location for your project. You can leave it as the default "No organization" if you're just getting started.

Once you've entered the project name, click the Create button. Google Cloud will now start creating your new project. It may take a few seconds.

Step 3: Access Your New Project 🛠️

After a few seconds, you’ll be redirected to your Google Cloud dashboard.

Click on the drop-down menu beside the Google Cloud logo again, and you should now see your newly created project listed in the modal where you can select it.

Then click on the project name (for example, gcr-ci-cd-project) to enter your project’s dashboard.

Step 4: Link A Billing Account To Your Project 💳

To access the billing page, in the Google Cloud Console, find the Navigation Menu (the three horizontal lines) at the top left of the screen. Click on it to open a list of options. Scroll down and click on Billing. This will take you to the billing section of your Google Cloud account.

If you haven't set up a billing account yet, you'll be prompted to do so. Click on the "Link a billing account" button to start the process.

Now you can create a new billing account (if you don’t have one). You’ll be redirected to a page where you can either select an existing billing account or create a new one. If you don't already have a billing account, click on "Create a billing account".

Provide the necessary details, including:

• Account name (for example, "Personal Billing Account" or your business name).

• Country: Choose the country where your business or account is based.

• Currency: Choose the currency in which you want to be billed.

  

Next, enter your payment information (credit card or bank account details). Google Cloud will verify your payment method, so make sure the information is correct.

Read and agree to the Google Cloud Terms of Service and Billing Account Terms. Once you’ve done this, click "Start billing" to finish setting up your billing account

After setting up your billing account, you’ll be taken to a page that asks you to link it to your project. Select the billing account you just created or an existing billing account you want to use. Click Set Account to link the billing account to your project.

After you’ve linked your billing account to your project, you should see a confirmation message indicating that billing has been successfully enabled for your project.

You can always verify this by returning to the Billing section in the Google Cloud Console, where you’ll see your billing account listed.

Create a Google Cloud Service Account to Enable Deployment of the Node.js Application to Google Cloud Run via the CD Pipeline 🚀

Why Do We Need a Service Account and Key? 🤔

A service account allows our CI/CD pipeline to authenticate and interact with Google Cloud services programmatically. By assigning specific roles (permissions), we ensure the service account can only perform tasks related to deployment, such as managing Google Cloud Run.

The service account key is a JSON file containing the credentials used for authentication. We securely store this key as a GitHub secret to protect sensitive information.

Step 1: Open the Service Accounts Page

Here are the steps you can follow to set up your service account and get your key:

First, visit the Google Cloud Console at https://console.cloud.google.com/. Ensure you’ve selected the correct project (e.g. gcr-ci-cd-project). To change projects, click the drop-down menu next to the Google Cloud logo at the top-left corner and select your project.

Then navigate to the Navigation Menu (three horizontal lines in the top-left corner) and click on IAM & Admin > Service Accounts.

Step 2: Create a New Service Account

Click on the "Create Service Account" button. This will open a form where you’ll define your service account details.

Next, enter the Service Account details:

• Name: Enter a descriptive name (for example, ci-cd-sa).

• ID: This will auto-fill based on the name.

• Description: Add a description to help identify its purpose, such as “Used for deploying Node.js app to Cloud Run.”

• Click Create and Continue to proceed.

Step 3: Assign Necessary Roles (Permissions)

On the next screen, you’ll assign roles to the service account. Add the following roles one by one:

• Cloud Run Admin: Allows management of Cloud Run services.

• Service Account User: Grants the ability to use service accounts.

• Service Usage Admin: Enables control over enabling APIs.

• Viewer: Provides read-only access to view resources.

To add a role:

• Click on "Select a Role".

• Use the search bar to type the role name (for example, "Cloud Run Admin") and select it.

• Repeat for all four roles.

Your screen should look similar to this:

After assigning the roles, click Continue.

Step 4: Skip Granting Users Access to the Service Account

On the next screen, you’ll see an option to grant additional users access to this service account. Click Done to complete the creation process.

Step 5: Generate a Service Account Key 🔑

You should now see your newly created service account in the list. Find the row for your service account (for example, ci-cd-sa) and click the three vertical dots under the “Actions” column. Select "Manage Keys" from the drop-down menu.

To add a new key:

• Click on "Add Key" > "Create New Key".

• In the pop-up dialog, select JSON as the key type.

• Click Create.

  

Now, download the key file. A JSON file will automatically be downloaded to your computer. This file contains the credentials needed to authenticate with Google Cloud.

Make sure you keep the key secure and store it in a safe location. Don’t share it – treat it as sensitive information.

Step 6: Add the Service Account Key to GitHub Secrets 🔒

Start by opening the downloaded JSON file using a text editor (like Notepad or VS Code). Then select and copy the entire contents of the file.

Then navigate to the repository you created for this project on GitHub. Click on the Settings tab at the top of the repository. Scroll down and find the Secrets and variables > Actions section.

Now you need to add a new secret. Click the "New repository secret" button. In the Name field, enter GCP_SERVICE_ACCOUNT. In the Value field, paste the JSON content you copied earlier. Click Add secret to save it.

Do the same for the GCP_PROJECT_ID secret, but now add your Google Project ID as the value. To get your project ID, follow these steps:

1. Navigate to the Google Cloud Console: Open Google Cloud Console at https://console.cloud.google.com/.

2. Locate the Project Dropdown: At the top-left of the screen, next to the Google Cloud logo, you will see a drop-down that shows the name of your current project.

3. View the Project ID: Click the drop-down, and you'll see a list of all your projects. Your Project ID will be displayed next to the project name. It is a unique identifier used by Google Cloud.

4. Copy the Project ID: Copy the Project ID that is displayed, and add it as the value of the GCP_PROJECT_ID secret.

Step 7: Adding External Variables to the GitHub Repository 🔧

Before proceeding with deployment, we need to define some external variables that were referenced in the CD workflow. These variables ensure that the pipeline knows critical details about your Google Cloud Run services and Docker container registry.

Here are the steps you’ll need to follow to do this:

1. First, go to your repository on GitHub.

2. Click the Settings tab at the top of the repository. Scroll down to Secrets and variables > Actions.

3. Click on the Variables tab next to Secrets. Click "New repository variable" for each variable. Then you’ll need to define these variables:

   • GCR_PROJECT_NAME: Set this to the name of your Cloud Run service for the production/live environment. For example, gcr-ci-cd-app.

   • GCR_STAGING_PROJECT_NAME: Set this to the name of your Cloud Run service for the staging/test environment. For example, gcr-ci-cd-staging.

   • GCR_REGION: Enter the region where you’d like to deploy the services. For this tutorial, set it to us-central1.

   • IMAGE: Specify the name of the Docker image/container registry where the published image will be uploaded. For example, <dockerhub-username>/ci-cd-tutorial-app.

4. After entering each variable name and value, click Add variable.

Enabling the Service Usage API on the Google Cloud Project 🌐

To deploy your application, the Service Usage API must be enabled in your Google Cloud project. This API allows you to manage Google Cloud services programmatically, including enabling/disabling APIs and monitoring their usage.

Follow these steps to enable it:

1. First, visit the Google Cloud Console at https://console.cloud.google.com/.

2. Then make sure you’re in the correct project. Click the project drop-down menu near the Google Cloud logo at the top-left corner. Select gcr-ci-cd-project , or the name you gave your project from the list of projects.

3. Next you’ll need to access the API library. Open the Navigation Menu (three horizontal lines in the top-left corner). Select APIs & Services > Library from the menu.

4. In the API Library, use the search bar to search for "Service Usage API".

5. Click on the Service Usage API from the search results. On the API’s details page, click Enable.

6. To verify, go to APIs & Services > Enabled APIs & Services in the Google Cloud Console. Confirm that the Service Usage API appears in the list of enabled APIs.

   

Create the Staging Branch and Merge the Feature Branch into It (Continuous Integration and Continuous Delivery) 🌟

When changes from the feature/ci-cd-pipeline branch are merged into the staging branch, we complete the Continuous Integration (CI) process, and the workflow ci-pipeline.yml will run. This ensures that the changes made in the feature branch are tested and integrated into a shared branch.

Once the pull request (PR) is merged into staging, the Continuous Delivery (CD) pipeline automatically triggers, deploying the application to the staging environment. This simulates how updates are tested in a safe environment before being pushed to production.

Create the staging Branch on the Remote Repository

To enable the CI/CD pipeline, we’ll first create a staging branch on the remote GitHub repository. This branch will serve as the test environment where changes are deployed before they reach the production environment.

To create the staging branch directly on GitHub, follow these steps:

1. First, navigate to your repository on GitHub. Open your web browser and go to the GitHub repository where you want to create the new staging branch.

2. Then, switch to the main branch. On the top of the repository page, locate the Branch dropdown (usually labelled as main or the current branch name). Click on the dropdown and make sure you are on the main branch.

3. Next, create the staging branch. In the same dropdown where you see the main branch, type staging into the text box. Once you start typing, GitHub will offer you the option to create a new branch called staging. Select the Create branch: staging option from the dropdown.

4. Finally, verify the branch**.** After creating the staging branch, GitHub will automatically switch to it. You should now see staging in the branch dropdown, confirming the new branch was created.

   

Merge Your Feature Branch into the Staging Branch via a Pull Request (PR)

This process combines both Continuous Integration (CI) and Continuous Delivery (CD). You will commit changes from your feature branch, push them to the remote feature branch, and then open a PR to merge those changes into the staging branch. Here's how to do it:

Step 1: Commit Local Changes on Your Feature Branch

First, you’ll want to make sure that you are on the correct branch (the feature branch) by running:

git status

If you are not on the feature/ci-cd-pipeline branch, switch to it by running:

git checkout feature/ci-cd-pipeline

Now, it’s time to add your changes you made for the commit:

git add .

This stages all changes, including new files, modified files, and deleted files.

Next, commit your changes with a clear and descriptive message:

git commit -m "Set up CI/CD pipelines for the project"

Then you can verify your commit by running:

git log

This will display your most recent commits, and you should see the commit message you just added.

Step 2: Push Your Feature Branch Changes to the Remote Repository

After committing your changes, push them to the remote repository:

git push origin feature/ci-cd-pipeline

This pushes your local changes on the feature/ci-cd-pipeline branch to the remote GitHub repository.

Once the push is successful, visit your GitHub repository in a web browser, and confirm that the feature/ci-cd-pipeline branch is updated with your new commit.

Step 3: Create a Pull Request to Merge the Feature Branch into Staging

Go to your repository on GitHub and ensure that you are on the main page of the repository.

You should see an alert at the top of the page suggesting you create a pull request for the recently pushed branch (feature/ci-cd-pipeline). Click the Compare & Pull Request button next to the alert.

Now, it’s time to choose the base and compare branches. On the PR creation page, make sure the base branch is set to staging (this is the branch you want to merge your changes into). The compare branch should already be set to feature/ci-cd-pipeline (the branch you just pushed). If they’re not selected correctly, use the dropdowns to change them.

You’ll want to come up with a good PR description for this. Write a clear title and description for the pull request, explaining what changes you're merging and why. For example:

• Title: "Merge CI/CD setup changes from feature branch"

• Description: "This pull request adds the CI/CD pipelines for GitHub Actions and Docker Hub integration to the project. It includes the configurations for both CI and CD workflows."

Now GitHub will show a list of all the changes that will be merged. Take a moment to review them and ensure everything looks correct.

If all looks good after reviewing, click on the Create pull request button. This will create the PR and notify team members (if any) that changes are ready to be reviewed and merged.

Wait a few seconds, and you should see a message indicating that all the checks have passed. Click on the link with the description "CI Pipeline to staging/production environment...". This should direct you to the Continuous Integration workflow, where you can view the steps that ran

The Continuous Integration (CI) Process

The CI process begins when a Pull Request is made to the staging branch. It triggers the GitHub Actions workflow defined in the .github/workflows/ci-pipeline.yml file. The workflow runs the necessary steps to set up the environment, install dependencies, and build the Node.js application.

It then runs automated tests (using npm test) to ensure that the changes do not break any functionality in the codebase. If all these steps are completed successfully, the CI pipeline confirms that the feature branch is stable and ready to be merged into the staging branch for further testing and deployment.

Step 4: Merge the Pull Request

If your team or collaborators are part of the project, they may review your PR. This step may involve discussing any changes or improvements. If everything looks good, a reviewer will merge the PR.

Once the PR has been reviewed and approved, you can merge the PR. To do this, just click on the Merge pull request button. Choose Confirm merge when prompted.

After merging, you can go to the staging branch to verify that the changes were successfully merged.

Navigating to the Actions Page After Merging the PR

Once you have successfully merged your pull request from the feature/ci-cd-pipeline branch into the staging branch, the Continuous Delivery (CD) pipeline will be triggered. To view the progress of the CD pipeline, navigate to the Actions tab in your GitHub repository. Here's how to do it:

1. Go to your GitHub repository.

2. At the top of the page, you will see the Actions tab next to the Code tab. Click on it.

3. On the Actions page, you will see a list of workflows that have been triggered. Look for the one labelled CD Pipeline to Google Cloud Run (staging and production). It should appear as a new run after the PR merge.

4. Click on the workflow run to view its progress and see the detailed logs for each step.

This will allow you to monitor the status of the CD pipeline and check if there are any issues during deployment.

If you look at the CD steps and workflow, you'll see that the step to deploy the application to the production environment was skipped, while the step to deploy to the staging environment was executed.

Continuous Delivery (CD) pipeline – what’s going on:

The Continuous Delivery (CD) Pipeline automates the process of deploying the application to Google Cloud Run (testing environment). This workflow is triggered by a push to the staging branch, which happens after the changes from the feature branch are merged into staging. It can also be manually triggered via workflow_dispatch or upon a new release being published.

The pipeline consists of multiple stages:

1. Test Job: The pipeline begins by setting up the environment and running tests using the npm test command. If the tests pass, the process moves forward.

2. Build Job: The next step builds the Docker image of the Node.js application, tags it, and then pushes it to Docker Hub.

3. Deployment to GCP: After the image is pushed, the workflow authenticates to Google Cloud and deploys the application. If the event is a release (that is, a push to the main branch), the application is deployed to the production environment. If the event is a push to staging, the app is deployed to the staging environment.

The CD process ensures that any changes made to the staging branch are automatically tested, built, and deployed to the staging environment, ready for further validation. When a release is published, it will trigger deployment to production, ensuring your app is always up to date.

Accessing the Deployed Application in the Staging Environment on Google Cloud Run 🌐

Once the deployment to Google Cloud Run is successfully completed, you'll want to access your application running in the staging environment. Follow these steps to find and visit your deployed application:

1. Navigate to the Google Cloud Console

Open the Google Cloud Console in your browser by visiting https://console.cloud.google.com. If you're not already signed in, make sure you log in with your Google account.

2. Go to the Cloud Run Dashboard

In the Google Cloud Console, use the Search bar at the top or navigate through the left-hand menu: Go to Cloud Run (you can type this into the search bar, or find it under Products & services > Compute > Cloud Run). Click on Cloud Run to open the Cloud Run dashboard.

3. Select Your Staging Service

In the Cloud Run dashboard, you should see a list of all your services deployed across various environments. Find the service associated with the staging environment. The name should be similar to what you defined in your workflow (for example, gcr-ci-cd-staging).

4. Access the Service URL

Once you've selected your staging service, you’ll be taken to the Service details page. This page provides all the important information about your deployed service.
On this page, look for the URL section under the Service URL heading. The URL will look something like: https://gcr-ci-cd-staging-<unique-id>.run.app.

5. Visit the Application

Click on the Service URL, and it will open your staging environment in a new tab in your browser. You can now interact with your application as if it were live, but in the staging environment.

Merge the Staging Branch into the Main Branch (Continuous Integration and Continuous Deployment) 🌐

In this section, we'll take the updates in the staging branch, merge them into the main branch, and trigger the CI/CD pipeline. This process not only ensures your changes are production-ready but also deploys them to the production/live environment. 🚀

Step 1: Push Local Changes and Open a Pull Request

Why? The first step involves merging the staging branch into the main branch. Just like in the previous Continuous Delivery process, this ensures the integration of thoroughly tested updates.

Here’s how to do it:

First, visit the GitHub repository where your project is hosted.

Then go to the Pull Requests tab. Click New Pull Request. Choose staging as the source branch (base branch) and main as the target branch. Add a clear title and description for the Pull Request, explaining why these updates are ready for production deployment.

Step 2: Continuous Integration (CI) Pipeline Execution

After merging the pull request, the Continuous Integration (CI) pipeline will automatically execute to validate that the changes are still stable when integrated into the main branch.

Pipeline Steps:

• Code Checkout: The workflow fetches the latest code from the main branch.

• Dependency Installation: The pipeline installs all required dependencies.

• Testing: Automated tests are run to validate the application's stability.

Step 3: Create a New Release

The Continuous Deployment (CD) workflow to deploy to the production environment is triggered by the creation of a new release from the main branch.

Let’s walk through the steps to create a release.

On your GitHub repository page, click on the Releases section (located under the Code tab).

Next, click Draft a new release. Set the Target branch to main. Enter a Tag version (for example, v1.0.0) following semantic versioning. Add a Release title and an optional description of the changes.

Then, click Publish Release to finalize.

Why run the Continuous Deployment pipeline on release instead of on push? 🤔

In our setup, we decided not to trigger the Continuous Deployment (CD) pipeline every time changes are pushed to the main branch. Instead, we trigger it only when a new release is created. This gives the team more control over when updates are deployed to the production environment.

Imagine a scenario where developers are working on new features—they may push changes to the main branch as part of their regular workflow, but these features might not be complete or ready for users yet. Automatically deploying every push could accidentally expose unfinished features to your users, which can be confusing or disruptive.

By requiring a release to trigger the deployment, the team gets a chance to finalize and polish all changes before they go live.

For example, developers can test new features in the staging environment, fix any issues, and merge those changes into the main branch without worrying about them immediately appearing in production. This workflow ensures that only well-tested and complete features make their way to your end users.

Ultimately, this approach helps maintain a smooth user experience. Instead of seeing half-built features or unexpected changes, users only see updates that are ready and functional. It also gives the team the flexibility to push changes to the main branch frequently—preventing merge conflicts and making collaboration easier—while keeping control over what gets deployed live. 🚀

Step 4: Navigate to the Actions Page

After the release is published, the CD pipeline for the production environment is triggered. To monitor this repeat the process taken for the Continuous Delivery workflow, follow these steps:

1. Go to the GitHub Actions tab: In your GitHub repository, click on the Actions tab.

2. Locate the deployment workflow: Look for the CD Pipeline to Google Cloud Run (staging and production) workflow. You’ll notice that the workflow has been triggered on the main branch due to the push event.

3. Open the workflow details: Click on the workflow to view detailed steps, logs, and statuses for each part of the deployment process.

This time, the Continuous delivery workflow deploys the application to the production/live environment.

Step 5: Access the Live Application

Once the deployment is complete, go to Google Cloud Console at https://console.cloud.google.com.

Navigate to Cloud Run from the menu. Select the service corresponding to the production environment (for example, gcr-ci-cd-app).

Locate the Service URL in the service details page. Open the URL in your browser to access the live application.

And now, congratulations – you’re done!

Conclusion 🌟

In this article, we explored how to build and automate a CI/CD pipeline for a Node.js application, using GitHub Actions, Docker Hub, and Google Cloud Run.

We set up workflows to handle Continuous Integration by testing and integrating code changes and Continuous Delivery to deploy those changes to a staging environment. We also containerized our app using Docker and deployed it seamlessly to Google Cloud Run.

Finally, we implemented Continuous Deployment, ensuring updates to the production environment happen only when a release is created from the main branch.

This approach gives teams the flexibility to push and test incomplete features without impacting end users. By following these steps, you've built a robust pipeline that makes deploying your application smoother, faster, and more reliable.

Study Further 📚

If you would like to learn more about Continuous Integration, Delivery, and Deployment you can check out the courses below:

• Continuous Integration and Continuous Delivery (CI/CD) (from IBM Coursera)

• GitHub Actions - The Complete Guide (from Udemy)

• Learn CI/CD by buliding a project (freeCodeCamp tutorial)

About the Author 👨‍💻

Hi, I’m Prince! I’m a software engineer passionate about building scalable applications and sharing knowledge with the tech community.

If you enjoyed this article, you can learn more about me by exploring more of my blogs and projects on my LinkedIn profile. You can find my LinkedIn articles here. And you can visit my website to read more of my articles as well. Let’s connect and grow together! 😊

In this tutorial, you'll learn what CI/CD is, and I'll help you set up a CI/CD pipeline using Husky and GitHub Actions in a Next.js application.

This tutorial assumes that you already have knowledge of React and Next.js or other modern JavaScript frameworks. You will need also a GitHub account, and basic knowledge of Git will be strongly beneficial.

If you already have a working web app that is not built with Next.js, you might still find this article useful. All the concepts and most of the configurations will work with little adaptation in apps created with other frameworks.

Here's What We'll Cover:

1. What is CI/CD?
   – What is CI?
   – What is CD?
   – What is a CI/CD pipeline and what are its benefits?
2. How to Set Up a CI/CD Pipeline
   – Step 1: Set Up a Next.js App with Vitest
   – Step 2: Set a Git Hook
   – Step 3: Create a GitHub Actions Workflow
   – Step 4: Deploy the Project
3. Conclusion

What is CI/CD?

Continuous Integration/Continuous Delivery or Continuous Deployment (CI/CD) is a practice that involves automating the process of building, testing, and deploying software.

Its main benefit is speeding up the entire development process. It also increases productivity by ensuring smooth code integration, standards, and security best practices adoption. It also helps produce a shorter feedback cycle with early issue detection, among other advantages explained below.

CI/CD is an essential tool in today’s software development practices, enabling teams to deliver high-quality software quickly, efficiently, and reliably.

Let’s learn more about it in detail.

What is CI?

Continuous Integration is a software practice that means that developers in a team merge code changes into a central repository multiple times a day.

Instead of having independent dev environments and merging at a specific time, developers frequently integrate their changes to an application into a shared branch or “trunk”.

What is CD?

The CD in CI/CD usually refers to Continuous Delivery. It's a practice that, on top of CI, automates the software integration, testing, and release process. The automation stops just before deploying to production, where a human-controlled step is needed.

But CD can also refer to Continuous Deployment, which adds automation to the step of releasing software to a production environment.

Even though CD usually refers to Continuous Delivery, both terms are sometimes used interchangeably. The difference between them is the amount of automation implemented in a project.

What is a CI/CD pipeline and what are its benefits?

When put together, these two practices create a CI/CD pipeline. Adding CI/CD to your project brings the following benefits:

• Faster development: reduces the time required to deliver new features thanks to automating the build, test and deploy.
• Enhanced Collaboration: encourages frequent code integrations and reduces integration conflicts.
• Improved Code Quality: enforces the adoption of coding standards and best practices throughout the codebase.
• Early Detection of Issues: makes the feedback cycle smaller, as issues can be caught in advance.
• Increased Productivity: prevents developers from needing to work on repetitive tasks.

These are some of the reasons why CI/CD is a core practice in modern software development and why it is such an important topic to learn about. The following steps will guide you through the process of setting up a CI/CD pipeline for your project.

How to Set Up a CI/CD Pipeline

Step 1: Set Up a Next.js App

If you already have a working web app, you can skip this and go directly to the first step.

Otherwise, let's set up a basic Next.js app with the default ESLint configuration and Vitest, and push it to a GitHub repo.

Create a Next.js app

Navigate into the directory where you want to create the new project folder, then run the following command in your terminal:

npx create-next-app@latest

When prompted with the installation options, make sure you choose to use ESLint in your project. This will ensure that ESLint is properly installed and a lint script is created in the package.json.

Wait for create-next-app to create the folder and install the project dependencies. Once it's done, navigate into the new folder and start the dev server:

cd <your-project-name>
npm run dev

Set up Vitest

Let's add Vitest to the project and add some automated tests to run in the CI/CD pipeline.

First, install vitest and the dev dependencies needed:

npm install -D vitest @vitejs/plugin-react jsdom @testing-library/react

Create a vitest.config.js file (or vitest.config.ts if using TypeScript) with the following content:

import { defineConfig } from 'vitest/config'
import react from '@vitejs/plugin-react'

export default defineConfig({
  plugins: [react()],
  test: {
    environment: 'jsdom',
  },
})

And finally, add the test script to the package.json:

 "test": "vitest --no-watch"

Note that I added the no-watch option to the test script. This prevents Vitest from starting in the default watch mode in dev environment.

Now, you can add tests for your project. If you don't know how to start, you can check out this guide for some examples.

Push the project to GitHub

Log in into your GitHub account and create new repository. Once your are done, you can connect the local repo with the one you just created, adding this repo as the remote. Then push the changes:

git add .
git commit -m "first commit"
git remote add origin git@github.com:<your-user-name>/<your-repo-name>.git
git push origin main

You should be now ready to continue to the interesting part of this tutorial. :)

Step 2: Set a Git Hook

A Git hook is a script that allows you to run some event within the Git lifecycle. In this case we will be using Husky.

Husky is a pre-commit hook for Git that allows you maintain code quality by executing some task upon committing or pushing. You can run various checks before making a commit with new changes, such as linting the code and running automated tests.

By implementing these checks, you can avoid wasting time and resources by catching issues in advance before triggering the GitHub Actions workflow.

Let’s start by adding Husky to the project with the following command:

npm install --save-dev husky

Next, let’s set up the project using the Husky init command:

npx husky init

After running this command, you will notice that a pre-commit file was created under ./husky. Also, a “prepare” script was added in the package.json.

If you open the pre-commit file inside ./husky, you will find the following content:

npm test

As its name suggests, this file contains the code that executes before completing a commit. With everything set up as described, tests will run each time you attempt to create a new commit and new commits will be added only if all tests pass.

Adding more git hooks

Now, let’s change the content in the pre-commit file so the code linter also executes before creating a new commit.

You can open your preferred code editor and add npm run lint (or the corresponding ESLint script if you’re not using Next.js) in a new line in the pre-commit file. Alternatively, you can simply run the following command from the root folder of your project:

echo "npm run lint" >> ./.husky/pre-commit

Now, each time you attempt to make a new commit, the tests and the linter will run, and the commit will be created – only if all tests are passing and no errors are found in the code.

Setting up lint-staged

You can go one step further and include a tool called lint-staged. This tool will be especially useful if your project is large, because it allows you to run the Git hooks only for staged files. In this case, it will lint only the files that will be committed, avoiding wasting time by linting the entire project.

To start using lint-staged, let's add it as a dev dependency to the project:

npm install --save-dev lint-staged

There are different ways to configure lint-staged and you can choose the one that best suits your needs. I will add a lint-staged script and object to the package.json of my project with the following content:

  "scripts": {
    "dev": "next dev",
    "build": "next build",
    "start": "next start",
    "lint": "next lint",
    "test": "vitest --no-watch",
    "prepare": "husky"
  },
  "lint-staged": {
    "*.{js, jsx,ts,tsx}": [
        "eslint --fix"
        ]
    },

Now, I can replace npm run lint with npm run lint-staged in the pre-commit file.

Each time I make a new commit, any js, jsx, ts, or tsx staged files will be linted and, if there are fixable issues, they will be automatically fixed.

Let's test that the pre-commit hook is working as expected by:

1. Running git add .
2. Running git commit
3. Waiting for the linter to run and entering a commit message when prompted
4. Running git log to confirm that the commit was properly created

If you want, you can add more checks to your pre-commit file to fit your project's needs. For example, you could run a tool like Prettier to automatically format your code, or commitlint to lint your commit messages.

Now, let’s move on to setting up a GitHub Actions workflow for the project.

Step 3: Create a GitHub Actions Workflow

With the first part complete, we can move on to the next step. Here, you will add a GitHub Actions workflow to ensure the smooth integration of changes into the entire project.

GitHub Actions Basics

GitHub Actions is a CI/CD platform that allows you to automate the building, testing, and deployment of your project. It also lets you perform actions when certain activities happen in your repository, such as opening a pull request or creating an issue.

GitHub Actions are configured through workflows defined in YAML files. These workflows typically run when triggered by an event in the repository, but they can also be scheduled or run manually.

Workflows are located in the .github/workflows folder and run different jobs. Each job includes a set of steps that run in order on the same runner or server. A step can be either a shell script or an action (a reusable piece of code that helps reduce repetitive code in your workflows).

Let's put all this together by creating the first workflow.

Creating a workflow to execute when you push to main branch

First create a .github/workflows/ under your project root. Then create a run-test.yml file. You will be adding content to this file to create a CI workflow.

The first line is optional and includes a name for the workflow. It will appear at the "Actions" tab in the GitHub repo:

name: Run linter and tests on push

Then, you will use the on key to define the event or events that will trigger the workflow run. This can be an event in your repo or a time schedule. In this case, let's set it to run each time a push to the repo happens:

on:
  push

You can also set options below the on keyword to limit the execution of a workflow to some branch or files – for example to run only on push to main branch:

on:
  push:
    branches:
      - main

Below this, you will add the jobs key. It groups all the jobs in the workflow, followed by the name of the first job, in this case run-linter-and-tests.

The lines below that define workflow properties, configuring it to run on the latest version of an Ubuntu Linux runner and grouping all the steps that run on this job.

jobs:
  run-linter-and-tests:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Install dependencies
        run: npm i

      - name: Lint code
        run: npm run lint

      - name: Run tests
        run: npm test

As mentioned before, each step can be either a shell script or an action. You can see the difference between the first and the second step in the previous code.

The first one specifies with the uses keyword that will run the actions/checkout. This action is used to checkout the repository onto the runner so the workflow can use the repository code. The second step Install dependencies uses the run keyword to tell the job to execute the npm i command on the runner.

This is the complete resulting file:

name: CI workflow
on:
  push

jobs:
  run-linter-and-tests:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: npm install
        run: npm i

      - name: Lint code
        run: npm run lint

      - name: Run tests
        run: npm test

Let's commit the changes and push them to the GitHub repository.

Now each time you push to your repository, the workflow will trigger. If you click on the "Actions" tab in your GitHub repository navigation bar, you will find a list of all the runs from all your workflows and its complete logs.

"Actions" tab in a GitHub repository navigation bar

Also, you will see that in the GitHub repository's "Code" tab, a green checkmark appears next to the last commit message. This means that workflows ran and finished successfully.

When jobs are still running, you'll see a brown dot, and a red cross when a workflow finished with an error.

Adding a second workflow to run when a PR is created

Each repository can have one or more workflows, so let's add a second workflow to run each time a PR is created. Let's run the code coverage report each time a PR is opened against the main branch of the repo.

First, create and checkout a new add-wf branch:

git checkout -b add-wf

Then, create a new YAML file under the .github/workflows directory and start adding some content on it.

First, let's add the name and when to run the workflow with the on keyword:

name: Run Coverage on PR
on: pull_request

After that, you will use the jobs keyword to describe the jobs to run. Let's define the first one as build-and-run-coverage to run in ubuntu-latest runner:

jobs:
  build-and-run-coverage:
    runs-on: ubuntu-latest

Now, let's add steps for this job:

  steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Install dependencies
        run: npm i

      - name: Build code
        run: npm run build

      - name: Run tests and coverage
        run: npm run coverage

Following is the complete resulting code:

name: Run Coverage on PR
on: pull_request

jobs:
  build-and-run-coverage:
    runs-on: ubuntu-latest

      steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Install dependencies
        run: npm i

      - name: Build code
        run: npm run build

      - name: Run tests and coverage
        run: npm run coverage

Now, you can push the change to your GitHub repo:

git add .
git commit -m 'add a wf to run on opened PR'
git push origin add-wf

Now you can open a PR against your main branch and and wait for the workflow to complete.

Comment coverage report in the PR

As mentioned earlier in this article, actions are reusable pieces of code that avoid repetitive code in the workflow. One cool thing about them is that there are many already written by the community that you can use in your workflows, saving lots of time.

To complete the workflow we created, let's add a new step that uses an action to report coverage results as a comment on the pull request.

First, let's modify the permissions keyword to ensure the workflow has the right access to content and to create comments:

 permissions:
      contents: read
      pull-requests: write

Then, let's use the Vitest Coverage Report action by adding a step into the build-and-run-coverage job:

- name: Report Coverage
        uses:  davelosert/vitest-coverage-report-action@v2

The final yaml file will look like this:

name: Run Coverage on PR
on: pull_request

jobs:
  build-and-run-coverage:
    runs-on: ubuntu-latest

    permissions:
      contents: read
      pull-requests: write

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Install dependencies
        run: npm i

      - name: Build
        run: npm run build

      - name: Run test and coverage
        run: npm run coverage

      - name: Report Coverage
        uses:  davelosert/vitest-coverage-report-action@v2

There is one more step to ensure all works as expected. You must add the json-summary reporter in the Vitest configuration:

import { defineConfig } from "vitest/config";
import react from "@vitejs/plugin-react";

export default defineConfig({
  plugins: [react()],
  test: {
    environment: "jsdom",
    coverage: {
      provider: "v8",
      extension: [".tsx"],
      reporter: ['text', 'json-summary', 'json'],
    },
  },
});

Now, make some changes in your project and add corresponding tests to check if the workflow is working as expected.

Once you push your changes to the GitHub repo, open a PR against the main branch of your project. After the workflows finish running, you should see a comment showing the coverage result:

Coverage Report in a pull request comment

Step 4: Deploy the Project

As a last step in this tutorial, let's deploy the project on Vercel. You will set up an automatic deployment through Git that will trigger a redeploy each time new changes are pushed or merged into the main branch.

First, log in to your Vercel account, or create one if you don't already have one. Then, in your dashboard, click on "Add New Project" and click on the "Import" button next to your repository name in the "Import Git Repository" section.

If you don't see your repository listed, it may be due to your GitHub app permissions configuration. You can manage them in your settings section in your GitHub account.

Finally, choose a name for the project in the "Configure Project" section and click on the "Deploy" button. You can now see the deploy details by clicking on the "Deployment" link.

Vercel automatic deployments ensure that the deployed project is always updated with the latest changes. They also have the benefit of Preview Deployments, a preview URL that lets you test new features in advance of merging changes into production.

If you have followed along with the tutorial, with this step completed, you'll have completed the CD part of the CI/CD pipeline for your project. Now, you can be sure any code that is pushed to the main branch is linted and tested, and once all checks pass, it is automatically pushed to production.

Conclusion

In this guide, you learned about the importance of CI/CD in today’s software development ecosystem and its main benefits. You also took your first steps in this area by creating your own CI/CD pipeline for your project, learning how to use Husky and GitHub Actions.

Now, you can keep learning more about these tools and improve your CI/CD pipeline by customizing it to better fit your project's needs.

I hope you were able to gain some new knowledge and enjoyed following along. Thanks for reading!

In this article, we'll learn how to set up a CI/CD pipeline with GitHub Actions and AWS. I've divided the guide into three parts to help you work through it:

First, we'll cover some important terminology so you're not lost in a bunch of big buzzwords.

Second, we'll set up continuous integration so we can automatically run builds and tests.

And finally, we'll set up continuous delivery so we can automatically deploy our code to AWS.

Alright, that was a lot. Let's start by diving into each of these terms so you understand exactly what we're doing here.

Part One: Demystifying the Hefty Buzzwords

The key to making sense of the title of this piece lies in understanding the terms CI/CD Pipeline, GitHub Actions, and AWS.

If you already have a strong grasp of what these terms are, you can just skip to down to Part 2.

What is a CI/CD Pipeline?

A CI/CD Pipeline is simply a development practice. It tries to answer this one question: How can we ship quality features to our production environment faster? In other words, how can we hasten the feature release process without compromising on quality?

How does the CI/CD Pipeline help us hasten the feature release process, you might ask?

The diagram below depicts a typical feature delivery cycle with or without the CI/CD pipeline.

the feature release process. Source: Author

Without the CI/CD Pipeline, each step in the diagram above will be performed manually by the developer. In essence, to build the source code, someone on your team has to manually run the command to initiate the build process. Same thing with running tests and deployment.

The CI/CD approach is a radical shift from the manual way of doing things. It is entirely based on the premise that we can speed up the feature release process reasonably, if we automate steps 3-6 in the diagram above.

With the CI/CD Pipeline, we set up a mechanism that automatically starts the build process, runs the tests, deploys to the User Acceptance Testing (UAT) environment, and finally to the production environment each time a member of the team pushes their change to the shared repo, for example.

Continuous Integration happens each time the build process is initiated, and tests run on a new change.

Continuous Delivery happens when a newly integrated change is automatically deployed to the UAT environment and then manually deployed to the production environment from there.

Continuous Deployment happens when an update in the UAT environment is automatically deployed to the production environment as an official release.

Continuous Integration vs Continuous Deployment vs Continuous Delivery. Source: Author

Note: If the deployment from the UAT environment to the production environment is initiated manually, then it is a Continuous Integration/Continuous Delivery setup. Otherwise, it is a Continuous Integration/Continuous Deployment set up.

However, we can't help but ask, what is that entity that automates the different phases of the CI/CD Pipeline?

There are variety of tools we can use to automate the build, tests, and deployment steps in the CI/CD Pipeline – for example, CI Circle, Travis CI, Jenkins, GitHub Actions, and so on. In this article we will be focusing on GitHub Actions.

What are GitHub Actions?

For want of a better way of making the GitHub Actions term super comprehensible, I'm going to oversimplify this.

In the CI/CD Pipeline, GitHub Actions is the entity that automates the boring stuff. Think of it as some plugin that comes bundled with every GitHub repository you create.

The plugin exists on your repo to execute whatever task you tell it to. Usually, you'd specify what tasks the plugin should execute through a YAML configuration file. Whatever command you add to the configuration file, will translate to something like this in plain English:

"hey GitHub Actions, each time a PR is opened on X branch, automatically build and test the new change. And each time a new change is merged into or pushed to X branch, deploy that change to Y server."

At the core of GitHub Actions lies five concepts: jobs, workflows, events, actions, and runners.

Jobs are the tasks you command GitHub Actions to execute through the YAML config file. A job could be something like telling GitHub actions to build your source code, run tests, or deploy the code that has been built to some remote server.

Workflows are essentially automated processes that contain one or more logically related jobs. For example, you could put the build and run tests jobs into the same workflow, and the deployment job into a different workflow.

Remember, we mentioned that you tell GitHub Actions what job(s) to execute through a configuration file right? GitHub Actions considers each configuration file that you put in some folder in your repo a workflow. We will talk more about this folder in the next part.

So, to create a separate workflow for the deployment job and then a different workflow that combines the build and tests jobs, you'd have to add two config files to your repo. But if you are merging all the three jobs into a single workflow, then you'd need to add just one config file.

Events are literally the events that trigger the execution of a job by GitHub Actions. Recall we mentioned passing jobs to be executed through a config file? In that config file you'd also have to specify when a job should be executed.

For example, is it on-PR to main? Is it on-push to main? is it on-merge to main? A job can only be executed by a GitHub Action when some event happens.

Okay, let me quickly correct myself. It's not always the case that some event has to happen before a job could be executed. You could schedule jobs too.

For example, in your config file, instead of specifying that the event that should trigger the execution of, let's say, the build-and-test job, you could schedule it to happen a 2am everyday. In fact, you could both schedule a job and specify an event for that same job.

Actions are the reusable commands that you can reuse in your config file. You can write your custom actions or use existing ones.

A runner is the remote computer that GitHub Actions uses to execute the jobs you tell it to.

For example, when the build-and-test job is triggered based on some event, GitHub Actions will pull your code to that computer and execute the job.

The same thing happens in the case of the deployment job. The runner triggers the deployment of the built code to some remote server you specify. In our case, we will be using a service called AWS.

What is AWS?

AWS stands for Amazon Web Services. It is a platform owned by Amazon, and this platform allows you access to a broad range of cloud computing services.

Cloud computing – I thought you said no big words? Most times businesses and even individual developers build applications just so other people can use them. For that reason, these applications have to be available on the interwebs.

Making an application accessible to some target users, ideally, entails uploading that application to a special computer that runs 24/7 and is super fast.

Imagine if it were the case that, before you could make your applications available to other internet users, you'd have to own and set up such a computer. It is quite doable, but it comes with a lot of hurdles.

For example, what if you just want to test out the application? You'd go through all the stress of setting up a hardware infrastructure just for testing? Furthermore, what if you want to upload 1000 different applications or your application is beginning to handle 1billion concurrent requests? Things start to get complicated.

Cloud computing platforms like AWS exist to save you all that stress. These platforms already have numerous of these special computers setup and kept in buildings called data centers.

Instead of having to setup your own hardware infrastructure from scratch, they allow you upload your application to one of their pre-configured computers over the internet. In return, you pay some certain amount to them.

In fact, some of these platforms have free plans that allows you test out small applications. In addition to uploading your application's code, some of these platforms also allow you host your database and store your media files, amongst other features.

In its most simplistic form, Cloud Computing is primarily about storing or executing (sometimes both) certain things on someone else's computer, usually, over a network.

So when I said AWS gives access to a wide range of cloud services, I was just saying it provides businesses and individuals with some special computer where they could upload their code, databases, and media files as a service.

Okay, now that we fully understand the different parts of our title, we will now restate it in the form of objectives. These objectives will then dictate what the remaining two parts in this article will contain.

Objective 1: How to automatically build and run unit tests on push or on PR to the main branch with GitHub Actions.

Objective 2: How to automatically deploy to AWS on push or on PR to the main branch with GitHub Actions.

Part 2: Continuous Integration – How to Automatically Run Builds and Tests

In this section, we will be seeing how we can configure GitHub Actions to automatically run builds and tests on push or pull request to the main branch of a repo.

Prerequisites

• A Django project setup locally with at least one view that returns some response defined.
• A testcase written for the view(s) you've defined.

I have created a demo Django project which you can grab from this repository:

git clone [https://github.com/Nyior/django-github-actions-aws](https://github.com/Nyior/django-github-actions-aws)

After you download the code, create a virtualenv and install the requirements via pip:

pip install -r requirements.txt

Note: The project already has all the files we will be adding incrementally as we proceed. Maybe you could still download and try to make sense of the content of the files as we proceed. The project is certainly not interesting. It's just created for demo purposes.

Now that you have a Django project setup locally, let's configure GitHub Actions.

How to Configure GitHub Actions

Okay, so we have our project setup. We also have a testcase written for the view that we have defined, but most importantly we've pushed our shiny change to GitHub.

The goal is to have GitHub trigger a build and run our tests each time we push or open a pull request on main/master. We just pushed our change to main, but GitHub Actions didn't trigger the build or run our tests.

Why not? Because we haven't defined a workflow yet. Remember, a workflow is where we specify the jobs we want GitHub Actions to execute.

In fact, Nyior, how did you even know that no build was triggered and by extension no workflow defined? Every GitHub repo has an Action tab. If you navigate to that tab, you'll know if a repo has a workflow defined on it or not.

How? See the images below.

A GitHub Repo With a Workflow Defined on it

A GitHub Repo With No Workflow Defined on it

The first repo in the first image has a workflow defined on it named 'Lint and Test'. The second repo in the second image has no workflow – it's why you don't see a list with the heading 'All Workflows' as is the case with the first repo.

Oh okay, now I get it. So how do I define a workflow on my repo?

• Create a folder named .github in the root of your project directory.
• Create a folder named workflows in the .github directory: This is where you'll create all your YAML files.
• Let's create our first workflow that will contain our build and test jobs. We do that by creating a file with a .yml extension. Let's name this file build_and_test.yml
• Add the content below in the yaml file you just created:

name: Build and Test

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest

    steps:
    - name: Checkout code
      uses: actions/checkout@v2
    - name: Set up Python Environment
      uses: actions/setup-python@v2
      with:
        python-version: '3.x'
    - name: Install Dependencies
      run: |
        python -m pip install --upgrade pip
        pip install -r requirements.txt

    - name: Run Tests
      run: |
        python manage.py test

Let's make sense of each line in the file above.

• name: Build and Test This is the name of our workflow. When you navigate to the actions tab, each workflow you define will be identified by the name you give it here on that list.
• on: This is where you specify the events that should trigger the execution of our workflow. In our config file we passed it two events. We specified the main branch as the target branch.
• jobs: Remember, a workflow is just a collection of jobs.
• test: This is the name of the job we've defined in this workflow. You could name it anything really. Notice it's the only job and the build job isn't there? Well, it's Python code so no build step is required. This is why we didn't define the build job.
• runs-on GitHub provides Ubuntu Linux, Microsoft Windows, and macOS runners to run your workflows. This where you specify the type of runner you want to use. In our case, we are using the Ubuntu Linux runner.
• A job is made up of a series of steps that are usually executed sequentially on the same runner. In our file above, each step is marked by a hyphen. name represents the name of the step. Each step could either be a shell script that is being executed or an action . You define a step with uses if it's executing an action or you define a step with run if it's executing a shell script.

Now that you've defined a workflow by adding the config file in the designated folder, you can commit and push your change to your remote repo.

If you navigate to the Actions tab of your remote repo, you should see a workflow with the name Build and Test (the name which we've given it) listed there.

Part 3: Continuous Delivery – How to Automatically Deploy Our Code to AWS

In this section, we will see how we can have GitHub Actions automatically deploy our code to AWS on push or pull request to the main branch. AWS offers a broad range of services. For this tutorial, we will be using a compute service called Elastic Beanstalk.

Compute Service? Elastic Beanstalk? Come onnn :(

No worries, relax, you'll get it. Remember we mentioned that cloud computing is all about storing and executing certain things on someone else's computer via the internet right – certain things?

Yes. For example, we can store and execute source code or we can just store media files. Amazon knows this, and as a result, their cloud infrastructure encompasses a plethora of service categories. Each service category allows us do a certain thing out of the certain things that we can do.

For example, there is a service category that allows the upload and execution of source codes that power our applications (Compute Service). There is the service category that allows us persist our media files (Storage Service). Then there is the service category that allows us manage our databases (Database Service).

Each service category is made up of one or more services. Each service in a category just presents us with a different way of solving the problem that the category it belongs to addresses.

For example, each service in the compute category provides us with a different approach to deploying and executing our application code on the cloud – one problem, different approaches. Elastic Beanstalk is one of the services in the compute category. Others are, but not limited to, EC2 and Lambda.

Amazon S3 is one of the services in the storage category. And Amazon RDS is one of the services in the Database category.

Hopefully, you now understand what I mean by "we will be using a compute service called Elastic Beanstalk." Of all the compute services, why Elastic Beanstalk? Well, because it's one of the easiest to work with.

That Being Said, Let's Configure Stuff <3

For brevity's sake we are going with the Continuous Delivery setup. In addition, we are going to have just one one deployment environment that will serve as our UAT environment.

To help you get the big picture right, in summary, this is how our deployment setup is going to work: on push or pull request to main, GitHub Actions will test and upload our source code to Amazon S3. The code is then pulled from Amazon S3 to our Elastic Beanstalk environment. Picture the flow this way:

GitHub -> Amazon S3 -> Elastic Beanstalk

Why aren't we pushing directly to Elastic Beanstalk, you might ask?

The only other way we could upload code directly to an Elastic Beanstalk instance with our current setup, is if we were using the AWS Elastic Beanstalk CLI (EB CLI).

Using the EB CLI requires running some shell command that would then require that we respond with some input.

Now, if we are deploying from our local machine to Elastic Beanstalk, when we run the EB CLI commands, we'd be there to type in the required responses. But with our current setup, those commands would be executed on GitHub Runners. We wouldn't be there to provide the required responses. The EB CLI isn't the easiest deployment tool for our use case.

With the approach we've picked, we'd run a shell command that uploads our code to S3 and then another command that pulls the uploaded code to our Elastic Beanstalk instance. These commands, when run, do not require that we submit some responses. Having the Amazon s3 step is the easiest way to go about this.

To implement our approach and have our code deployed to Elastic Beanstalk, follow the steps below:

Step 1: Setup an AWS Account

Create an IAM. To keep things simple, when adding permissions, just add "Administrator Access" to the user (this has some security pitfalls, though). To accomplish this, follow the steps in modules 1 and 2 of this guide.

In the end, make sure to grab and keep your AWS secret and access keys. We will be needing them in the subsequent sections.

Now that we have an AWS account properly setup, it's time to set up our Elastic Beanstalk environment.

Step 2: Setup your Elastic Beanstalk Environment

Once logged into your AWS account, take the following steps to set up your Elastic Beanstalk environment.

First, search for "elastic beanstalk" in the search field as shown in the image below. Then click on the Elastic Beanstalk service.

searching for elastic beanstalk.

Once you click on the Elastic Beanstalk service in the previous step, you'll be taken to the page shown in the image below. On that page, click on the "Create a New Environment" prompt. Make sure to select "Web server environment" in the next step.

creating an environment

After selecting the "Web server environment" in the previous page you'll be taken to the page shown in the images below.

On that page, submit an application name, an environment name, and also select a platform. For this tutorial, we are going with the Python platform.

an application and an environment names

selecting a platform

Once you submit the form filled out in the previous step, after a while your application and its associated environment will be created. You should see the names you submitted displayed on the left side bar.

Grab the application name and the environment name. We will be needing them in the subsequent steps.

Now that we have our Elastic Beanstalk environment fully setup, it's time to configure GitHub Actions to trigger automatic deployment to AWS on push or pull request to main.

Step 3: Configure your Project for Elastic Beanstalk

By default, Elastic Beanstalk looks for a file named application.py in our project. It uses that file to run our application, but we don't have that file in our project. Do we? We need to tell Elastic Beanstalk to use the wsgi.py file in our project to run our application instead. To do that, take the following step:

Create a folder named .ebextensions in your project directory. In that folder create a config file. You can name it anything you want. I named mine eb.config. Add the content below to your config file:

option_settings:
  aws:elasticbeanstalk:container:python:
    WSGIPath: django_github_actions_aws.wsgi:application

After completing the step above, your project directory should now look similar to the one in the image below:

demo project structure

One last thing you need to do in this section is to go to your settings.py file and update the ALLOWED_HOSTS setting to all:

ALLOWED_HOSTS = ['*']

Note that using the wildcard has major security drawbacks. We are only using it here for demo purposes.

Now that we are done configuring our project for Elastic Beanstalk, it's time to update our workflow file.

Step 4: Update your Workflow File

There are five important pieces of information we need to complete this step: application name, environment name, access key id, secret access key, and the server region (after login, you can grab the region from the right-most section of the navbar).

Because the access key id and the secret access key are sensitive data, we'll hide them somewhere in our repository and access them in our workflow file.

To do that, head over to the settings tab of your repo, and then click on secrets as shown in the image below. There, you can create your secrets as key-value pairs.

embedding secret data in your repo

Next, add the deployment job to the end of your existing workflow file:

deploy:
    needs: [test]
    runs-on: ubuntu-latest

    steps:
    - name: Checkout source code
      uses: actions/checkout@v2

    - name: Generate deployment package
      run: zip -r deploy.zip . -x '*.git*'

    - name: Deploy to EB
      uses: einaregilsson/beanstalk-deploy@v20
      with:

          // Remember the secrets we embedded? this is how we access them
        aws_access_key: ${{ secrets.AWS_ACCESS_KEY_ID }}
        aws_secret_key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

        // Replace the values here with your names you submitted in one of 
        // The previous sections
        application_name: django-github-actions-aws
        environment_name: django-github-actions-aws

        // The version number could be anything. You can find a dynamic way 
        // Of doing this.
        version_label: 12348
        region: "us-east-2"
        deployment_package: deploy.zip

needs simply tells GitHub Actions to only start executing the deployment job after the test job has been completed with a passing status.

The step Deploy to EB uses an existing action, einaregilsson/beanstalk-deploy@v20. Remember how we said actions are some reusable applications that takes care of some frequently repeated tasks for us? einaregilsson/beanstalk-deploy@v20 is one of those actions.

To reinforce the above, remember that our deployment was suppose to go through the following steps: GitHub -> Amazon S3 -> Elastic Beanstalk.

However, throughout this tutorial, we didn't do any Amazon s3 set up. Furthermore, in our workflow file we didn't upload to an s3 bucket nor did we pull from an s3 bucket to our Elastic Beanstalk environment.

Normally, we are supposed to do all that, but we didn't here – because under the hood, the einaregilsson/beanstalk-deploy@v20 action does all the heavy lifting for us. You can also create your own action that takes care of some repetitive tasks and make it available to other developers through the GitHub Marketplace.

Now that you've updated your workflow file locally, you can then commit and push this change to your remote. Your jobs will run and your code will be deployed to the Elastic Beanstalk instance you created. And that's it. We're done >>>

Wrapping Up

Wow! This was a really long one, wasn't it? In summary I explained what the terms GitHub Actions, CI/CD Pipeline, and AWS mean. In addition, we saw how we could configure GitHub Actions to automatically deploy our code to an Elastic Beanstalk instance on AWS.

If you love this work, and would like to stay up to date on future articles I will be putting out, let's connect on Twitter, Linkedin, or GitHub. I use those channels to share what I work on, immediately after I put them out.

Credits:

Cover image: www.freepik.com

These days, applications have to evolve as the needs of their target users grow and change. This is why engineering teams often adopt Agile software development principles (or any iterative variation).

Agile principles involve continuous integration and continuous delivery (CI/CD). This means that developers will frequently make code updates for new features to the existing codebase of the application.

So then how can you verify that a recent code addition doesn't break a part of the application? The answer is continuous testing.

What is Continuous Testing?

Continuous testing is a critical part of the CI/CD pipeline. It helps development teams discover if a particular code commit will break the application build and if it should be integrated or not.

In other words, continuous testing is the practice of integrating automated tests into a software delivery pipeline to determine the risks associated with each code release or addition. These automated tests are usually triggered during or after builds and are carried out using automation testing frameworks or tools.

Let me now introduce you to four recommended automation tools you can use for continuous testing.

Tools for Continuous Testing

1. TestSigma

TestSigma is a cloud-based automation testing tool for continuous testing. It has a low learning curve, as automated tests can be written in plain English, and requires no coding skills. Tests can also be extended with Selenium and JS-based custom functions for more advanced use cases.

TestSigma can be used for web applications, native mobile apps, regression, cross-browser and data-driven testing. It also features inbuilt seamless integrations with test management, bug reporting, CI/CD, and communication tools such as GitHub, Slack, Jira, BrowserStack, Jenkins, AWS, Bamboo, Azure DevOps, Circle CI, and so on.

TestSigma also uses AI to reduce maintenance effort and increase productivity by identifying affected tests and potential failures upfront to save execution time & cost, alongside other features.

The platform has a free tier, but to use all the features mentioned above, you’ll need to commit to a paid plan.

2. Tricentis Tosca

Tosca is another no-code continuous testing tool which makes it easy to learn. QA engineers with zero scripting knowledge can easily set up automated tests using a GUI.

Tosca is suitable for enterprise-level applications and is versatile because it supports and integrates seamlessly with over 160 technologies/languages. With Tosca, you can run tests on the web, mobile, and desktops with Windows OS (Mac and Linux are only possible with virtualization tools).

Tosca automatically creates and provisions on-demand test data to reduce the time it takes to provision and make reliable test data for test automation.

The platform offers free trials for a limited amount of time and custom pricing, which the sales team decides on based on your specific needs.

3. Katalon Studio

Katalon is another comprehensive continuous testing tool built on top of the popular open-source Selenium and Appium. It can be used for testing web, API, mobile, and desktop applications across Windows, macOS, and Linux operating systems.

In fact, with Katalon, you can execute tests on all OSs, browsers, and devices, as well as on cloud, on-premise, and hybrid environments.

Katalon also provides other useful features like recording test steps, executing test cases, providing infrastructure, analytics reporting, and CI/CD integration with the most popular CI tools (like Jenkins, Bamboo, Azure, and CircleCI).

Katalon Studio is easy to get started with because it offers codeless test creation for beginners. For advanced usage, experts can extend automation capabilities using the plugins in the Katalon Store.

It also has extensive documentation featuring a well-organized library of tutorials alongside images and videos to help you out if you ever get stuck on something. It has a robust free tier and an enterprise tier for advanced usage.

4. Watir

Watir is another continuous testing automation tool powered by the Selenium framework, and it is open-source. Watir can only run tests for web applications on Windows and can only execute simple and easily maintainable tests.

It is not codeless, as scripts have to be written in Ruby, Java, .NET or Python using its sister software: Watij, WatiN, and Nerodia. Regardless, it's easy to get started with if you're already familiar with Ruby because it features extensive documentation.

Watir can also be integrated with a couple of CI tools such as Jenkins and GitHub.

Even though Watir seems limited, most teams find its simplicity appealing. It is prevalent within the Ruby community and is even used by large companies like Slack and Oracle.

How to choose a continuous testing tool

There are other excellent continuous testing tools available aside from the four that I have mentioned above. I favour no-code testing tools because it lets teams set up and maintain automated tests much faster.

Regardless, here are a few things to consider before choosing a continuous testing tool:

1. Application types supported: Does the tool support your intended application type (for example, mobile, web, desktop)?
2. Learning curve: How easy/difficult is it to use? Will you need to learn a new scripting language? Ideally, you should go for something with a low learning curve that you and your team can get started with in the shortest amount of time.
3. Costs: Is the cost of the tool a feasible addition to your budget in the long run?
4. Integration capabilities: Can it integrate seamlessly with your existing CI/CD pipeline?
5. Scalability and reusability: Does the tool support scalability and reusability of test cases across multiple projects?
6. Documentation and Community: How concise and rich is the documentation for the tool? You’re going to run into some mental blocks in the future, and you may not be able to get through without proper documentation and community support.

Conclusion

With the right tools, continuous testing eliminates the risks associated with frequent code releases by ensuring that only quality code is delivered to the end-user.

As I previously mentioned, the tools I have listed above are not an exhaustive list of all the continuous testing tool options. They're just the ones I recommend, and they may or may not be the right choice for you.

Do some further research, check out different tools, and settle for one that will integrate seamlessly into your current setup and meet your needs.

But automation can be tough and can sometimes prove to be costly. How can Github Actions help harden our code and give us more time to work on features instead of bugs?

• What are Github Actions?
• What is CI/CD?
• What are we going to build?
• Part 0: Setting up a project
• Part 1: Automating tests
• Part 2: Post new pull requests to Slack

What are Github Actions?

Actions are a relatively new feature to Github that allow you to set up CI/CD workflows using a configuration file right in your Github repo.

Previously, if you wanted to set up any kind of automation with tests, builds, or deployments, you would have to look to services like Circle CI and Travis or write your own scripts. But with Actions, you have first class support to powerful tooling to automate your workflow.

What is CI/CD?

CD/CD stands for Continuous Integration and Continuous Deployment (or can be Continuous Delivery). They're both practices in software development that allow teams to build projects together quickly, efficiently, and ideally with less errors.

Continuous Integration is the idea that as different members of the team work on code on different git branches, the code is merged to a single working branch which is then built and tested with automated workflows. This helps to constantly make sure everyone's code is working properly together and is well-tested.

Continuous Deployment takes this a step further and takes this automation to the deployment level. Where with the CI process, you automate the testing and the building, Continuous Deployment will automate deploying the project to an environment.

The idea is that the code, once through any building and testing processes, is in a deployable state, so it should be able to be deployed.

What are we going to build?

We're going to tackle two different workflows.

The first will be to simply run some automated tests that will prevent a pull request from being merged if it is failing. We won't walk through building the tests, but we'll walk through running tests that already exist.

In the second part, we'll set up a workflow that sends a message to slack with a link to a pull request whenever a new one is created. This can be super helpful when working on open source projects with a team and you need a way to keep track of requests.

Part 0: Setting up a project

For this guide, you can really work through any node-based project as long as it has tests you can run for Part 1.

If you'd like to follow along with a simpler example that I'll be using, I've set up a new project that you can clone with a single function that has two tests that are able to run and pass.

A function with two tests

If you'd like to check out this code to get started, you can run:

git clone --single-branch --branch start git@github.com:colbyfayock/my-github-actions.git

Once you have that cloned locally and have installed the dependencies, you should be able to run the tests and see them pass!

Passing tests

It should also be noted that you'll be required to have this project added as a new repository on Github in order to follow along.

Follow along with the commit!

Part 1: Automating tests

Tests are an important part of any project that allow us to make sure we're not breaking existing code while we work. While they're important, they're also easy to forget about.

We can remove the human nature out of the equation and automate running our tests to make sure we can't proceed without fixing what we broke.

Step 1: Creating a new action

The good news, is Github actually makes it really easy to get this workflow started as it comes as one of their pre-baked options.

We'll start by navigating to the Actions tab on our repository page.

Github Actions starting page

Once there, we'll immediately see some starter workflows that Github provides for us to dive in with. Since we're using a node project, we can go ahead and click Set up this workflow under the Node.js workflow.

Setting up a Node.js Github Action workflow

After the page loads, Github will land you on a new file editor that already has a bunch of configuration options added.

We're actually going to leave this "as is" for our first step. Optionally, you can change the name of the file to tests.yml or something you'll remember.

Adding a new Github Action workflow file

You can go ahead and click Start commit then either commit it directory to the master branch or add the change to a new branch. For this walkthrough, I'll be committing straight to master.

To see our new action run, we can again click on the Actions tab which will navigate us back to our new Actions dashboard.

Viewing Github Action workflow events

From there, you can click on Node.js CI and select the commit that you just made above and you'll land on our new action dashboard. You can then click one of the node versions in the sidebar via build (#.x), click the Run npm test dropdown, and we'll be able to see the output of our tests being run (which if you're following along with me, should pass!).

Viewing logs of a Github Action workflow

Follow along with the commit!

Step 2: Configuring our new action

So what did we just do above? We'll walk through the configuration file and what we can customize.

Github Action Node.js workflow file

Starting from the top, we specify our name:

name: Node.js CI

This can really be whatever you want. Whatever you pick should help you remember what it is. I'm going to customize this to "Tests" so I know exactly what's going on.

on:
  push:
    branches: [ master ]
  pull_request:
    branches: [ master ]

The on key is how we specify what events trigger our action. This can be a variety of things like based on time with cron. But here, we're saying that we want this action to run any time someone pushes commits to master or someone creates a pull request targeting the master branch. We're not going to make a change here.

jobs:
  build:
    runs-on: ubuntu-latest

This next bit creates a new job called build. Here we're saying that we want to use the latest version of Ubuntu to run our tests on. Ubuntu is common, so you'll only want to customize this if you want to run it on a specific environment.

    strategy:
      matrix:
        node-version: [10.x, 12.x, 14.x]

Inside of our job we specify a strategy matrix. This allows us to run the same tests on a few different variations.

In this instance, we're running the tests on 3 different versions of node to make sure it works on all of them. This is definitely helpful to make sure your code is flexible and future proof, but if you're building and running your code on a specific node version, you're safe to change this to only that version.

    steps:
    - uses: actions/checkout@v2
    - name: Use Node.js ${{ matrix.node-version }}
      uses: actions/setup-node@v1
      with:
        node-version: ${{ matrix.node-version }}
    - run: npm ci
    - run: npm run build --if-present
    - run: npm test

Finally, we specify the steps we want our job to run. Breaking this down:

• uses: actions/checkout@v2: In order for us to run our code, we need to have it available. This checks out our code on our job environment so we can use it to run tests.
• uses: actions/setup-node@v1: Since we're using node with our project, we'll need it set up on our environment. We're using this action to do that setup for us for each version we've specified in the matrix we configured above.
• run: npm ci: If you're not familiar with npm ci, it's similar to running npm install but uses the package-lock.json file without performing any patch upgrades. So essentially, this installs our dependencies.
• run: npm run build --if-present: npm run build runs the build script in our project. The --if-present flag performs what it sounds like and only runs this command if the build script is present. It doesn't hurt anything to leave this in as it won't run without the script, but feel free to remove this as we're not building the project here.
• run: npm test: Finally, we run npm test to run our tests. This uses the test npm script set up in our package.json file.

And with that, we've made a few tweaks, but our tests should run after we've committed those changes and pass like before!

Logs of passing tests in Github Action workflow

Follow along with the commit!

Step 3: Testing that our tests fail and prevent merges

Now that our tests are set up to automatically run, let's try to break it to see it work.

At this point, you can really do whatever you want to intentionally break the tests, but here's what I did:

Code diff - https://github.com/colbyfayock/my-github-actions/pull/1

I'm intentionally returning different expected output so that my tests will fail. And they do!

Failing status checks on pull request

In my new pull request, my new branch breaks the tests, so it tells me my checks have failed. If you noticed though, it's still green to merge, so how can we prevent merges?

We can prevent pull requests from being merged by setting up a Protected Branch in our project settings.

First, navigate to Settings, then Branches, and click Add rule.

Github branch protection rules

We'll then want to set the branch name pattern to *, which means all branches, check the Require status checks to pass before merging option, then select all of our different status checks that we'd like to require to pass before merging.

Setting up a branch protection rule in Github

Finally, hit Create at the bottom of the page.

And once you navigate back to the pull request, you'll notice that the messaging is a bit different and states that we need our statuses to pass before we can merge.

Failing tests preventing merge in pull request

Note: as an administrator of a repository, you'll still be able to merge, so this technically only prevents non-administrators from merging. But will give you increased messaging if the tests fail.

And with that, we have a new Github Action that runs our tests and prevents pull requests from merging if they fail.

Follow along with the pull request!

Note: we won't be merging that pull request before continuing to Part 2.

Part 2: Post new pull requests to Slack

Now that we're preventing merge requests if they're failing, we want to post a message to our Slack workspace whenever a new pull request is opened up. This will help us keep tabs on our repos right in Slack.

For this part of the guide, you'll need a Slack workspace that you have permissions to create a new developer app with and the ability to create a new channel for the bot user that will be associated with that app.

Step 1: Setting up Slack

There are a few things we're going to walk through as we set up Slack for our workflow:

• Create a new app for our workspace
• Assign our bot permissions
• Install our bot to our workspace
• Invite our new bot to our channel

To get started, we'll create a new app. Head over to the Slack API Apps dashboard. If you already haven't, log in to your Slack account with the Workspace you'd like to set this up with.

Creating a new Slack app

Now, click Create New App where you'll be prompted to put in a name and select a workspace you want this app to be created for. I'm going to call my app "Gitbot" as the name, but you can choose whatever makes sense for you. Then click Create App.

Configuring a new Slack app

Once created, navigate to the App Home link in the left sidebar. In order to use our bot, we need to assign it OAuth scopes so it has permissions to work in our channel, so select Review Scopes to Add on that page.

Reviewing Slack app scopes

Scroll own and you'll see a Scopes section and under that a Bot Token section. Here, click Add an OAuth Scope. For our bot, we don't need a ton of permissions, so add the channels:join and chat:write scopes and we should be good to go.

Adding scopes for a Slack app Bot Token

Now that we have our scopes, let's add our bot to our workspace. Scroll up on that same page to the top and you'll see a button that says Install App to Workspace.

Installing Slack app to a workspace

Once you click this, you'll be redirected to an authorization page. Here, you can see the scopes we selected for our bot. Next, click Allow.

Allowing permission for Slack app to be installed to workspace

At this point, our Slack bot is ready to go. At the top of the OAuth & Permissions page, you'll see a Bot User OAuth Access Token. This is what we'll use when setting up our workflow, so either copy and save this token or remember this location so you know how to find it later.

Note: this token is private - don't give this out, show it in a screencast, or let anyone see it!

Copying OAuth Access Token for Slack bot user

Finally, we need to invite our Slack bot to our channel. If you open up your workspace, you can either use an existing channel or create a new channel for these notifications, but you'll want to enter the command /invite @[botname] which will invite our bot to our channel.

Inviting Slack bot user to channel

And once added, we're done with setting up Slack!

Slack bot was added to channel

Create a Github Action to notify Slack

Our next step will be somewhat similar to when we created our first Github Action. We'll create a workflow file which we'll configure to send our notifications.

While we can use our code editors to do this by creating a file in the .github directory, I'm going to use the Github UI.

First, let's navigate back to our Actions tab in our repository. Once there, select New workflow.

Setting up a new Github Action workflow

This time, we're going to start the workflow manually instead of using a pre-made Action. Select set up a workflow yourself at the top.

Setting up a Github Action workflow manually

Once the new page loads, you'll be dropped in to a new template where we can start working. Here's what our new workflow will look like:

name: Slack Notifications

on:
  pull_request:
    branches: [ master ]

jobs:
  notifySlack:

    runs-on: ubuntu-latest

    steps:
    - name: Notify slack
      env:
        SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }}
      uses: abinoda/slack-action@master
      with:
        args: '{\"channel\":\"[Channel ID]\",\"blocks\":[{\"type\":\"section\",\"text\":{\"type\":\"mrkdwn\",\"text\":\"*Pull Request:* ${{ github.event.pull_request.title }}\"}},{\"type\":\"section\",\"text\":{\"type\":\"mrkdwn\",\"text\":\"*Who?:* ${{ github.event.pull_request.user.login }}\n*Request State:* ${{ github.event.pull_request.state }}\"}},{\"type\":\"section\",\"text\":{\"type\":\"mrkdwn\",\"text\":\"<${{ github.event.pull_request.html_url }}|View Pull Request>\"}}]}'

So what's happening in the above?

• name: we're setting a friendly name for our workflow
• on: we want our workflow to trigger when there's a pull request is created that targets our master branch
• jobs: we're creating a new job called notifySlack
• jobs.notifySlack.runs-on: we want our job to run on a basic setup of the latest Unbuntu
• jobs.notifySlack.steps: we really only have one step here - we're using a pre-existing Github Action called Slack Action and we're configuring it to publish a notification to our Slack

There are two points here we'll need to pay attention to, the env.SLACK_BOT_TOKEN and the with.args.

In order for Github to communicate with Slack, we'll need a token. This is what we're setting in env.SLACK_BOT_TOKEN. We generated this token in the first step. Now that we'll be using this in our workflow configuration, we'll need to add it as a Git Secret in our project.

_Github secrets including SLACK_BOTTOKEN

The with.args property is what we use to configure the payload to the Slack API that includes the channel ID (channel) and our actual message (blocks).

The payload in the arguments is stringified and escaped. For example, when expanded it looks like this:

{
  "channel": "[Channel ID]",
  "blocks": [{
    "type": "section",
    "text": {
      "type": "mrkdwn",
      "text": "*Pull Request:* ${{ github.event.pull_request.title }}"
    }
  }, {
    "type": "section",
    "text": {
      "type": "mrkdwn",
      "text": "*Who?:*n${{ github.event.pull_request.user.login }}n*State:*n${{ github.event.pull_request.state }}"
    }
  }, {
    "type": "section",
    "text": {
      "type": "mrkdwn",
      "text": "<${{ github.event.pull_request._links.html.href }}|View Pull Request>"
    }
  }]
}

Note: this is just to show what the content looks like, we need to use the original file with the stringified and escaped argument.

Back to our configuration file, the first thing we set is our channel ID. To find our channel ID, you'll need to use the Slack web interface. Once you open Slack in your browser, you want to find your channel ID in the URL:

https://app.slack.com/client/[workspace ID]/[channel ID]

Channel ID in Slack web app URL

With that channel ID, you can modify our workflow configuration and replace [Channel ID] with that ID:

with:
  args: '{\"channel\":\"C014RMKG6H2\",...

The rest of the arguments property is how we set up our message. It includes variables from the Github event that we use to customize our message.

We won't go into tweaking that here, as what we already have will send a basic pull request message, but you can test out and build your own payload with Slack's Block Kit Builder.

Follow along with the commit!

Test out our Slack workflow

So now we have our workflow configured with our Slack app, finally we're ready to use our bot!

For this part, all we need to do is create a new pull request with any change we want. To test this out, I simply created a new branch where I added a sentence to the README.md file.

Code diff - https://github.com/colbyfayock/my-github-actions/pull/2

Once you create that pull request, similar to our tests workflow, Github will run our Slack workflow! You can see this running in the Actions tab just like before.

As long as you set everything up correctly, once the workflow runs, you should now have a new message in Slack from your new bot.

Slack bot automated message about new pull request

Note: we won't be merging that pull request in.

What else can we do?

Customize your Slack notifications

The message I put together is simple. It tells us who created the pull request and gives us a link to it.

To customize the formatting and messaging, you can use the Github Block Kit Builder to create your own.

If you'd like to include additional details like the variables I used for the pull request, you can make use of Github's available contexts. This lets you pull information about the environment and the job to customize your message.

I couldn't seem to find any sample payloads, so here's an example of a sample github context payload you would expect in the event.

Sample github context

More Github actions

With our ability to create new custom workflows, that's not a lot we can't automate. Github even has a marketplace where you can browse around for one.

If you're feeling like taking it a step further, you can even create your own! This lets you set up scripts to configure a workflow to perform whatever tasks you need for your project.

Join in the conversation!

What do you use Github actions for?

Share with me on Twitter!

There is plenty of content out there describing what Continuous Integration, Continuous Delivery, and Continuous Deployment are. But what purposes do these processes serve in the first place?

It is crucial to understand the problems CI and CD solve to use them properly. This will allow your team to improve your process and avoid putting effort into chasing fancy metrics that do not bring any value to your process.

Continuous Integration is a team problem

If you work in a team, chances are there are several developers working on the same repository. There is a main branch in the repository carrying the latest version of the code. Developers work on different things on different branches. Once someone is done with their change, they'll push or merge it to the main branch. Eventually the whole team will pull this change.

The scenario we want to avoid is that a faulty commit makes it to the main branch. Faulty means the code does not compile or the app won't start or is unusable. Why? Not because the app is broken or because all tests must always be green. That is not a problem–you can decide not to deploy that version and wait for a fix.

The problem is that your entire team is stuck. All the developers who pulled the faulty commit will spend 5 minutes wondering why it doesn't work. Several will probably try to find the faulty commit. Some will try to fix the issue by themselves in parallel of the faulty code author.

This is a waste of time for your team. The worst part is that repeated incidents fuel a mistrust of the main branch and encourage developers to work apart.

Continuous Integration is all about preventing the main branch from breaking so your team is not stuck. That's it. It is not about having all your tests green all the time and the main branch deployable to production at every commit.

The process of Continuous Integration is independent of any tool. You could manually verify that the merge of your branch and the main branch works locally, and then only actually push the merge to the repository. But that would be very inefficient. That's why Continuous Integration is implemented using automated checks.

The checks ensure that, at the bare minimum:

• The app should build and start
• Most critical features should be functional at all times (user signup/login journey and key business features)
• Common layers of the application that all the developers rely on should be stable. This means unit tests on those parts.

In practice, this means you need to pull any unit test framework that works for you and secure the common layers of the application. Sometimes it is not that much code and can be done fairly quickly. Also you need to add a "smoke test" verifying that the code compiles and that the application starts. This is especially important in technologies with crazy dependency injections like Java Spring or .NET core. In large projects it is so easy to miswire your dependencies that verifying that the app always starts is a must.

If you have hundreds or thousands of tests you don't need to run them all for each merge. It will take a lot of time and most tests probably verify "non team blocker" features.

We'll see in the next sections how the process of Continuous Delivery will make good use of these many tests.

It's not about tools

Tools and automated checks are all fine. But if your developers only merge giant branches they work on for weeks, they won't help you. The team will spend a good amount of time merging the branches and fixing the code incompatibilities that will arise eventually. It is as much a waste of time as being blocked by a faulty commit.

Continuous Integration is not about tools. It is about working in small chunks and integrating your new code to the main branch and pulling frequently.

Frequently means at least daily. Split the task you are working on into smaller tasks. Merge your code very often and pull very often. This way nobody works apart for more than a day or two and problems do not have time to become snowballs.

A large task does not need to be all in one branch. It should never be. Techniques to merge work in progress to the main branch are called "branching by abstraction" and "feature toggles". See the blog post How to get started with Continuous Integration for more details.

Key points for a good CI build

It's very simple. Keep it short. 3-7 minutes should be the max. It's not about CPU and resources. It is about developers' productivity. The first rule of productivity is focus. Do one thing, finish it, then move to the next thing.

Context switching is costly. Studies show it takes ~23 minutes to deeply refocus on something when you get disturbed.

Imagine you push your branch to merge it. You start another task. You spend 15-20 minutes getting into it. The minute after you are in the zone you receive a "build failed" notification from your 20 minutes long CI build for the previous task. You get back to fix it. You push it again. You easily lost more than 20 minutes moving back and forth.

Multiply 20 minutes once or twice a day by the number of developers in your team... That's a lot of precious time wasted.

Now imagine if the feedback came within 3 minutes. You probably wouldn't have started the new task at all. You would have proof read your code one more time or reviewed a PR while waiting. The failed notification would come and you would fix it. Then you could move on to the next task. That is the kind of focus your process should enable.

Keeping your CI build short makes it a trade off. Tests that run longer or provide little value in the context of CI should be moved to the CD step. And yes, failures there also need to be fixed. But since they are not preventing anybody from doing their thing, you can take the fixes as a "next task" when you finish what you are doing. Just turn off the notifications while working and check every now and then. Keep the context switching to a minimum.

Continuous Delivery and Deployment are engineering problems

Let’s settle on the definitions to get that out of the way.

Continuous Delivery is about being able to deploy any version of your code at all times. In practice it means the last or pre-last version of your code. You don’t deploy automatically, usually because you don’t have to or are limited by your project lifecycle. But as soon as someone feels like it, a deployment can be done in a minimal amount of time. That someone can be the test/QA team that wants to test things out on a staging or pre-production environment. Or it can actually be time to roll out the code to production.

The idea of Continuous Delivery is to prepare artifacts as close as possible from what you want to run in your environment. These can be .jar or .war files if you are working with Java, or executables if you are working with .NET. These can also be folders of transpiled JS code or even Docker containers, whatever makes deployment shorter (i.e. you have pre-built as much as you can in advance).

By preparing artifacts, I don't mean turning code into artifacts. This is usually a few scripts and minutes of execution. Preparing means:

Run all the tests you can to ensure that, once deployed, the code will actually work. Run unit tests, integration tests, end to end tests, and even performance tests if you can automate that.

This way you can filter which versions of your main branch are actually production ready and which are not. The ideal test suite:

• Ensures that the application's key functionalities work. Ideally all functionalities
• Ensures that no performance deal breaker has been introduced so when your new version hits your many users, it has a chance to last
• Dry run any database updates your code needs to avoid surprises

It does not need to be very fast. 30 minutes or 1 hour is acceptable.

Continuous Deployment is the next step. You deploy the most up to date and production ready version of your code to some environment. Ideally production if you trust your CD test suite enough.

Note that, depending on the context, this is not always possible or worth the effort. Continuous Delivery is often enough to be productive, especially if you are working in a close network and have limited environments you can deploy to. It can also be that the release cycle of your software prevents unplanned deploys.

Continuous Delivery and Continuous Deployment (let’s call them CD from now on) are not team problems. They are about finding the right balance between execution time, maintenance efforts and relevance of your tests suite to be able to say "This version works as it should."

And it is a balance. If your tests last 30 hours that is a problem. See this epic post about what the Oracle database test suite looks like. But if you spend so much time keeping your tests up to date with the latest code that it impedes the team's progress, that is not good either. Also if your test suite ensures pretty much nothing... it is basically useless.

In an ideal world we want one set of deployable artifacts per commit to the main branch. You can see we have a vertical scalability problem: the faster we move from code to artifacts, the more ready we are to deploy the newest version of the code.

What’s the big difference?

Continuous Integration is a horizontal scalability problem. You want developers to merge their code often so the checks must be fast. Ideally within minutes to avoid developers switching context all the time with highly async feedback from the CI builds.

The more developers you have, the more computing power you need to run simple checks (build and test) on all the active branches.

A good CI build:

Ensures no code that breaks basic stuff and prevents other team members to work is introduced to the main branch, and

Is fast enough to provide feedback to developers within minutes to prevent context switching between tasks.

Continuous Delivery and Deployment are vertical scalability problems. You have one rather complex operation to perform.

A good CD build:

Ensures that as many features as possible are working properly.

The faster the better, but it is not a matter of speed. A 30-60 minutes build is OK.

A common misconception is to see CD as a horizontal scalability problem like CI: the faster you can move from code to artifacts, the more commits you can actually process, and the closer to the ideal scenario you can be.

But we don't need that. Producing artifacts for every commit and as fast as possible is usually overkill. You can very well approach CD on a best effort basis: have a single CD build that will just pick the latest commit to verify once a given build is finished.

Make no mistake about CD. It is really hard. Getting to sufficient test confidence to say your software is ready to be deployed automatically usually works on low surface applications like APIs or simple UIs. It is very difficult to achieve on a complex UI or a large monolith system.

Conclusion

Tools and principles used to execute CI and CD are often very similar. The goals are very different though.

Continuous Integration is a trade off between speed of feedback loop to developers and relevance of the checks your perform (build and test). No code that would impede the team progress should make it to the main branch.

Continuous Delivery of Deployment is about running as thorough checks as you can to catch issues on your code. Completeness of the checks is the most important factor. It is usually measured in terms code coverage or functional coverage of your tests. Catching errors early on prevents broken code to get deployed to any environment and saves the precious time of your test team.

Craft your CI and CD builds to achieve these goals and keep your team productive. No workflow is perfect. Problems will arise every now and then. Use them as lessons learned to strengthen your workflow every time they do.

Published on 27 Nov 2019 on the Fire CI Blog.

Testing is a pain in general. Some don't see the point. Some see it but think of it as an extra step slowing them down. Sometimes tests are there but very long to run or unstable. In this article you'll see how you can engineer tests for yourself with Docker.

We want fast, meaningful and reliable tests written and maintained with minimal effort. It means tests that are useful to you as a developer on a day-to-day basis. They should boost your productivity and improve the quality of your software. Having tests because everybody says "you should have tests" is no good if it slows you down.

Let's see how to achieve this with not that much effort.

The example we are going to test

In this article we are going to test an API built with Node/express and use chai/mocha for testing. I've chosen a JS'y stack because the code is super short and easy to read. The principles applied are valid for any tech stack. Keep reading even if Javascript makes you sick.

The example will cover a simple set of CRUD endpoints for users. It's more than enough to grasp the concept and apply to the more complex business logic of your API.

We are going to use a pretty standard environment for the API:

• A Postgres database
• A Redis cluster
• Our API will use other external APIs to do its job

Your API might need a different environment. The principles applied in this article will remain the same. You'll use different Docker base images to run whatever component you might need.

Why Docker? And in fact Docker Compose

This section contains a lot of arguments in favour of using Docker for testing. You can skip it if you want to get to the technical part right away.

The painful alternatives

To test your API in a close to production environment you have two choices. You can mock the environment at code level or run the tests on a real server with the database etc. installed.

Mocking everything at code level clutters the code and configuration of our API. It is also often not very representative of how the API will behave in production. Running the thing in a real server is infrastructure heavy. It is a lot of setup and maintenance, and it does not scale. Having a shared database, you can run only 1 test at a time to ensure test runs do not interfere with each other.

Docker Compose allows us to get the best of both worlds. It creates "containerized" versions of all the external parts we use. It is mocking but on the outside of our code. Our API thinks it is in a real physical environment. Docker compose will also create an isolated network for all the containers for a given test run. This allows you to run several of them in parallel on your local computer or a CI host.

Overkill?

You might wonder if it isn't overkill to perform end to end tests at all with Docker compose. What about just running unit tests instead?

For the last 10 years, large monolith applications have been split into smaller services (trending towards the buzzy "microservices"). A given API component relies on more external parts (infrastructure or other APIs). As services get smaller, integration with the infrastructure becomes a bigger part of the job.

You should keep a small gap between your production and your development environments. Otherwise problems will arise when going for production deploy. By definition these problems appear at the worst possible moment. They will lead to rushed fixes, drops in quality, and frustration for the team. Nobody wants that.

You might wonder if end to end tests with Docker compose run longer than traditional unit tests. Not really. You'll see in the example below that we can easily keep the tests under 1 minute, and at great benefit: the tests reflect the application behaviour in the real world. This is more valuable than knowing if your class somewhere in the middle of the app works OK or not.

Also, if you don't have any tests right now, starting from end to end gives you great benefits for little effort. You'll know all stacks of the application work together for the most common scenarios. That's already something! From there you can always refine a strategy to unit test critical parts of your application.

Our first test

Let’s start with the easiest part: our API and the Postgres database. And let’s run a simple CRUD test. Once we have that framework in place, we can add more features both to our component and to the test.

Here is our minimal API with a GET/POST to create and list users:

const express = require('express');
const bodyParser = require('body-parser');
const cors = require('cors');

const config = require('./config');

const db = require('knex')({
  client: 'pg',
  connection: {
    host : config.db.host,
    user : config.db.user,
    password : config.db.password,
  },
});

const app = express();

app.use(bodyParser.urlencoded({ extended: false }));
app.use(bodyParser.json());
app.use(cors());

app.route('/api/users').post(async (req, res, next) => {
  try {
    const { email, firstname } = req.body;
    // ... validate inputs here ...
    const userData = { email, firstname };

    const result = await db('users').returning('id').insert(userData);
    const id = result[0];
    res.status(201).send({ id, ...userData });
  } catch (err) {
    console.log(`Error: Unable to create user: ${err.message}. ${err.stack}`);
    return next(err);
  }
});

app.route('/api/users').get((req, res, next) => {
  db('users')
  .select('id', 'email', 'firstname')
  .then(users => res.status(200).send(users))
  .catch(err => {
      console.log(`Unable to fetch users: ${err.message}. ${err.stack}`);
      return next(err);
  });
});

try {
  console.log("Starting web server...");

  const port = process.env.PORT || 8000;
  app.listen(port, () => console.log(`Server started on: ${port}`));
} catch(error) {
  console.error(error.stack);
}

Here are our tests written with chai. The tests create a new user and fetch it back. You can see that the tests are not coupled in any way with the code of our API. The SERVER_URL variable specifies the endpoint to test. It can be a local or a remote environment.

const chai = require("chai");
const chaiHttp = require("chai-http");
const should = chai.should();

const SERVER_URL = process.env.APP_URL || "http://localhost:8000";

chai.use(chaiHttp);

const TEST_USER = {
  email: "john@doe.com",
  firstname: "John"
};

let createdUserId;

describe("Users", () => {
  it("should create a new user", done => {
    chai
      .request(SERVER_URL)
      .post("/api/users")
      .send(TEST_USER)
      .end((err, res) => {
        if (err) done(err)
        res.should.have.status(201);
        res.should.be.json;
        res.body.should.be.a("object");
        res.body.should.have.property("id");
        done();
      });
  });

  it("should get the created user", done => {
    chai
      .request(SERVER_URL)
      .get("/api/users")
      .end((err, res) => {
        if (err) done(err)
        res.should.have.status(200);
        res.body.should.be.a("array");

        const user = res.body.pop();
        user.id.should.equal(createdUserId);
        user.email.should.equal(TEST_USER.email);
        user.firstname.should.equal(TEST_USER.firstname);
        done();
      });
  });
});

Good. Now to test our API let's define a Docker compose environment. A file called docker-compose.yml will describe the containers Docker needs to run.

version: '3.1'

services:
  db:
    image: postgres
    environment:
      POSTGRES_USER: john
      POSTGRES_PASSWORD: mysecretpassword
    expose:
      - 5432

  myapp:
    build: .
    image: myapp
    command: yarn start
    environment:
      APP_DB_HOST: db
      APP_DB_USER: john
      APP_DB_PASSWORD: mysecretpassword
    expose:
      - 8000
    depends_on:
      - db

  myapp-tests:
    image: myapp
    command: dockerize
        -wait tcp://db:5432 -wait tcp://myapp:8000 -timeout 10s
        bash -c "node db/init.js && yarn test"
    environment:
      APP_URL: http://myapp:8000
      APP_DB_HOST: db
      APP_DB_USER: john
      APP_DB_PASSWORD: mysecretpassword
    depends_on:
      - db
      - myapp

So what do we have here. There are 3 containers:

• db spins up a fresh instance of PostgreSQL. We use the public Postgres image from Docker Hub. We set the database username and password. We tell Docker to expose the port 5432 the database will listen to so other containers can connect
• myapp is the container that will run our API. The build command tells Docker to actually build the container image from our source. The rest is like the db container: environment variables and ports
• myapp-tests is the container that will execute our tests. It will use the same image as myapp because the code will already be there so there is no need to build it again. The command node db/init.js && yarn test run on the container will initialize the database (create tables etc.) and run the tests. We use dockerize to wait for all the required servers to be up and running. The depends_on options will ensure that containers start in a certain order. It does not ensure that the database inside the db container is actually ready to accept connections. Nor that our API server is already up.

The definition of the environment is like 20 lines of very easy to understand code. The only brainy part is the environment definition. User names, passwords and URLs must be consistent so containers can actually work together.

One thing to notice is that Docker compose will set the host of the containers it creates to the name of the container. So the database won't be available under localhost:5432 but db:5432. The same way our API will be served under myapp:8000. There is no localhost of any kind here.

This means that your API must support environment variables when it comes to environment definition. No hardcoded stuff. But that has nothing to do with Docker or this article. A configurable application is point 3 of the 12 factor app manifesto, so you should be doing it already.

The very last thing we need to tell Docker is how to actually build the container myapp. We use a Dockerfile like below. The content is specific to your tech stack but the idea is to bundle your API into a runnable server.

The example below for our Node API installs Dockerize, installs the API dependencies and copies the code of the API inside the container (the server is written in raw JS so no need to compile it).

FROM node AS base

# Dockerize is needed to sync containers startup
ENV DOCKERIZE_VERSION v0.6.0
RUN wget https://github.com/jwilder/dockerize/releases/download/$DOCKERIZE_VERSION/dockerize-alpine-linux-amd64-$DOCKERIZE_VERSION.tar.gz \
    && tar -C /usr/local/bin -xzvf dockerize-alpine-linux-amd64-$DOCKERIZE_VERSION.tar.gz \
    && rm dockerize-alpine-linux-amd64-$DOCKERIZE_VERSION.tar.gz

RUN mkdir -p ~/app

WORKDIR ~/app

COPY package.json .
COPY yarn.lock .

FROM base AS dependencies

RUN yarn

FROM dependencies AS runtime

COPY . .

Typically from the line WORKDIR ~/app and below you would run commands that would build your application.

And here is the command we use to run the tests:

docker-compose up --build --abort-on-container-exit

This command will tell Docker compose to spin up the components defined in our docker-compose.yml file. The --build flag will trigger the build of the myapp container by executing the content of the Dockerfile above. The --abort-on-container-exit will tell Docker compose to shutdown the environment as soon as one container exits.

That works well since the only component meant to exit is the test container myapp-tests after the tests are executed. Cherry on the cake, the docker-compose command will exit with the same exit code as the container that triggered the exit. This means that we can check if the tests succeeded or not from the command line. This is very useful for automated builds in a CI environment.

Isn't that the perfect test setup?

The full example is here on GitHub. You can clone the repository and run the docker compose command:

docker-compose up --build --abort-on-container-exit

Of course you need Docker installed. Docker has the troublesome tendency of forcing you to sign up for an account just to download the thing. But you actually don't have to. Go to the release notes (link for Windows and link for Mac) and download not the latest version but the one right before. This is a direct download link.

The very first run of the tests will be longer than usual. This is because Docker will have to download the base images for your containers and cache a few things. The next runs will be much faster.

Logs from the run will look as below. You can see that Docker is cool enough to put logs from all the components on the same timeline. This is very handy when looking for errors.

Creating tuto-api-e2e-testing_db_1    ... done
Creating tuto-api-e2e-testing_redis_1 ... done
Creating tuto-api-e2e-testing_myapp_1 ... done
Creating tuto-api-e2e-testing_myapp-tests_1 ... done
Attaching to tuto-api-e2e-testing_redis_1, tuto-api-e2e-testing_db_1, tuto-api-e2e-testing_myapp_1, tuto-api-e2e-testing_myapp-tests_1
db_1           | The files belonging to this database system will be owned by user "postgres".
redis_1        | 1:M 09 Nov 2019 21:57:22.161 * Running mode=standalone, port=6379.
myapp_1        | yarn run v1.19.0
redis_1        | 1:M 09 Nov 2019 21:57:22.162 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
redis_1        | 1:M 09 Nov 2019 21:57:22.162 # Server initialized
db_1           | This user must also own the server process.
db_1           | 
db_1           | The database cluster will be initialized with locale "en_US.utf8".
db_1           | The default database encoding has accordingly been set to "UTF8".
db_1           | The default text search configuration will be set to "english".
db_1           | 
db_1           | Data page checksums are disabled.
db_1           | 
db_1           | fixing permissions on existing directory /var/lib/postgresql/data ... ok
db_1           | creating subdirectories ... ok
db_1           | selecting dynamic shared memory implementation ... posix
myapp-tests_1  | 2019/11/09 21:57:25 Waiting for: tcp://db:5432
myapp-tests_1  | 2019/11/09 21:57:25 Waiting for: tcp://redis:6379
myapp-tests_1  | 2019/11/09 21:57:25 Waiting for: tcp://myapp:8000
myapp_1        | $ node server.js
redis_1        | 1:M 09 Nov 2019 21:57:22.163 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
db_1           | selecting default max_connections ... 100
myapp_1        | Starting web server...
myapp-tests_1  | 2019/11/09 21:57:25 Connected to tcp://myapp:8000
myapp-tests_1  | 2019/11/09 21:57:25 Connected to tcp://db:5432
redis_1        | 1:M 09 Nov 2019 21:57:22.164 * Ready to accept connections
myapp-tests_1  | 2019/11/09 21:57:25 Connected to tcp://redis:6379
myapp_1        | Server started on: 8000
db_1           | selecting default shared_buffers ... 128MB
db_1           | selecting default time zone ... Etc/UTC
db_1           | creating configuration files ... ok
db_1           | running bootstrap script ... ok
db_1           | performing post-bootstrap initialization ... ok
db_1           | syncing data to disk ... ok
db_1           | 
db_1           | 
db_1           | Success. You can now start the database server using:
db_1           | 
db_1           |     pg_ctl -D /var/lib/postgresql/data -l logfile start
db_1           | 
db_1           | initdb: warning: enabling "trust" authentication for local connections
db_1           | You can change this by editing pg_hba.conf or using the option -A, or
db_1           | --auth-local and --auth-host, the next time you run initdb.
db_1           | waiting for server to start....2019-11-09 21:57:24.328 UTC [41] LOG:  starting PostgreSQL 12.0 (Debian 12.0-2.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
db_1           | 2019-11-09 21:57:24.346 UTC [41] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
db_1           | 2019-11-09 21:57:24.373 UTC [42] LOG:  database system was shut down at 2019-11-09 21:57:23 UTC
db_1           | 2019-11-09 21:57:24.383 UTC [41] LOG:  database system is ready to accept connections
db_1           |  done
db_1           | server started
db_1           | CREATE DATABASE
db_1           | 
db_1           | 
db_1           | /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
db_1           | 
db_1           | waiting for server to shut down....2019-11-09 21:57:24.907 UTC [41] LOG:  received fast shutdown request
db_1           | 2019-11-09 21:57:24.909 UTC [41] LOG:  aborting any active transactions
db_1           | 2019-11-09 21:57:24.914 UTC [41] LOG:  background worker "logical replication launcher" (PID 48) exited with exit code 1
db_1           | 2019-11-09 21:57:24.914 UTC [43] LOG:  shutting down
db_1           | 2019-11-09 21:57:24.930 UTC [41] LOG:  database system is shut down
db_1           |  done
db_1           | server stopped
db_1           | 
db_1           | PostgreSQL init process complete; ready for start up.
db_1           | 
db_1           | 2019-11-09 21:57:25.038 UTC [1] LOG:  starting PostgreSQL 12.0 (Debian 12.0-2.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
db_1           | 2019-11-09 21:57:25.039 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
db_1           | 2019-11-09 21:57:25.039 UTC [1] LOG:  listening on IPv6 address "::", port 5432
db_1           | 2019-11-09 21:57:25.052 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
db_1           | 2019-11-09 21:57:25.071 UTC [59] LOG:  database system was shut down at 2019-11-09 21:57:24 UTC
db_1           | 2019-11-09 21:57:25.077 UTC [1] LOG:  database system is ready to accept connections
myapp-tests_1  | Creating tables ...
myapp-tests_1  | Creating table 'users'
myapp-tests_1  | Tables created succesfully
myapp-tests_1  | yarn run v1.19.0
myapp-tests_1  | $ mocha --timeout 10000 --bail
myapp-tests_1  | 
myapp-tests_1  | 
myapp-tests_1  |   Users
myapp-tests_1  | Mock server started on port: 8002
myapp-tests_1  |     ✓ should create a new user (151ms)
myapp-tests_1  |     ✓ should get the created user
myapp-tests_1  |     ✓ should not create user if mail is spammy
myapp-tests_1  |     ✓ should not create user if spammy mail API is down
myapp-tests_1  | 
myapp-tests_1  | 
myapp-tests_1  |   4 passing (234ms)
myapp-tests_1  | 
myapp-tests_1  | Done in 0.88s.
myapp-tests_1  | 2019/11/09 21:57:26 Command finished successfully.
tuto-api-e2e-testing_myapp-tests_1 exited with code 0

We can see that db is the container that initializes the longest. Makes sense. Once it's done the tests start. The total runtime on my laptop is 16 seconds. Compared to the 880ms used to actually execute the tests, it is a lot. In practice, tests that run under 1 minute are gold as it is almost immediate feedback. The 15'ish seconds overhead are a buy in time that will be constant as you add more tests. You could add hundreds of tests and still keep execution time under 1 minute.

Voilà! We have our test framework up and running. In a real world project the next steps would be to enhance functional coverage of your API with more tests. Let's consider CRUD operations covered. It's time to add more elements to our test environment.

Adding a Redis cluster

Let's add another element to our API environment to understand what it takes. Spoiler alert: it's not much.

Let us imagine that our API keeps user sessions in a Redis cluster. If you wonder why we would do that, imagine 100 instances of your API in production. Users hit one or another server based on round robin load balancing. Every request needs to be authenticated.

This requires user profile data to check for privileges and other application specific business logic. One way to go is to make a round trip to the database to fetch the data every time you need it, but that is not very efficient. Using an in memory database cluster makes the data available across all servers for the cost of a local variable read.

This is how you enhance your Docker compose test environment with an additional service. Let’s add a Redis cluster from the official Docker image (I've only kept the new parts of the file):

services:
  db:
    ...

  redis:
    image: "redis:alpine"
    expose:
      - 6379

  myapp:
    environment:
      APP_REDIS_HOST: redis
      APP_REDIS_PORT: 6379
    ...
  myapp-tests:
    command: dockerize ... -wait tcp://redis:6379 ...
    environment:
      APP_REDIS_HOST: redis
      APP_REDIS_PORT: 6379
      ...
    ...

You can see it's not much. We added a new container called redis. It uses the official minimal redis image called redis:alpine. We added Redis host and port configuration to our API container. And we've made tests wait for it as well as the other containers before executing the tests.

Let’s modify our application to actually use the Redis cluster:

const redis = require('redis').createClient({
  host: config.redis.host,
  port: config.redis.port,
})

...

app.route('/api/users').post(async (req, res, next) => {
  try {
    const { email, firstname } = req.body;
    // ... validate inputs here ...
    const userData = { email, firstname };
    const result = await db('users').returning('id').insert(userData);
    const id = result[0];

    // Once the user is created store the data in the Redis cluster
    await redis.set(id, JSON.stringify(userData));

    res.status(201).send({ id, ...userData });
  } catch (err) {
    console.log(`Error: Unable to create user: ${err.message}. ${err.stack}`);
    return next(err);
  }
});

Let's now change our tests to check that the Redis cluster is populated with the right data. That's why the myapp-tests container also gets the Redis host and port configuration in docker-compose.yml.

it("should create a new user", done => {
  chai
    .request(SERVER_URL)
    .post("/api/users")
    .send(TEST_USER)
    .end((err, res) => {
      if (err) throw err;
      res.should.have.status(201);
      res.should.be.json;
      res.body.should.be.a("object");
      res.body.should.have.property("id");
      res.body.should.have.property("email");
      res.body.should.have.property("firstname");
      res.body.id.should.not.be.null;
      res.body.email.should.equal(TEST_USER.email);
      res.body.firstname.should.equal(TEST_USER.firstname);
      createdUserId = res.body.id;

      redis.get(createdUserId, (err, cacheData) => {
        if (err) throw err;
        cacheData = JSON.parse(cacheData);
        cacheData.should.have.property("email");
        cacheData.should.have.property("firstname");
        cacheData.email.should.equal(TEST_USER.email);
        cacheData.firstname.should.equal(TEST_USER.firstname);
        done();
      });
    });
});

See how easy this was. You can build a complex environment for your tests like you assemble Lego bricks.

We can see another benefit of this kind of containerized full environment testing. The tests can actually look into the environment's components. Our tests can not only check that our API returns the proper response codes and data. We can also check that data in the Redis cluster have the proper values. We could also check the database content.

Adding API mocks

A common element for API components is to call other API components.

Let's say our API needs to check for spammy user emails when creating a user. The check is done using a third party service:

const validateUserEmail = async (email) => {
  const res = await fetch(`${config.app.externalUrl}/validate?email=${email}`);
  if(res.status !== 200) return false;
  const json = await res.json();
  return json.result === 'valid';
}

app.route('/api/users').post(async (req, res, next) => {
  try {
    const { email, firstname } = req.body;
    // ... validate inputs here ...
    const userData = { email, firstname };

    // We don't just create any user. Spammy emails should be rejected
    const isValidUser = await validateUserEmail(email);
    if(!isValidUser) {
      return res.sendStatus(403);
    }

    const result = await db('users').returning('id').insert(userData);
    const id = result[0];
    await redis.set(id, JSON.stringify(userData));
    res.status(201).send({ id, ...userData });
  } catch (err) {
    console.log(`Error: Unable to create user: ${err.message}. ${err.stack}`);
    return next(err);
  }
});

Now we have a problem for testing anything. We can't create any users if the API to detect spammy emails is not available. Modifying our API to bypass this step in test mode is a dangerous cluttering of the code.

Even if we could use the real third party service, we don't want to do that. As a general rule our tests should not depend on external infrastructure. First of all, because you will probably run your tests a lot as part of your CI process. It’s not that cool to consume another production API for this purpose. Second of all the API might be temporarily down, failing your tests for the wrong reasons.

The right solution is to mock the external APIs in our tests.

No need for any fancy framework. We'll build a generic mock in vanilla JS in ~20 lines of code. This will give us the opportunity to control what the API will return to our component. It allows to test error scenarios.

Now let’s enhance our tests.

const express = require("express");

...

const MOCK_SERVER_PORT = process.env.MOCK_SERVER_PORT || 8002;

// Some object to encapsulate attributes of our mock server
// The mock stores all requests it receives in the `requests` property.
const mock = {
  app: express(),
  server: null,
  requests: [],
  status: 404,
  responseBody: {}
};

// Define which response code and content the mock will be sending
const setupMock = (status, body) => {
  mock.status = status;
  mock.responseBody = body;
};

// Start the mock server
const initMock = async () => {
  mock.app.use(bodyParser.urlencoded({ extended: false }));
  mock.app.use(bodyParser.json());
  mock.app.use(cors());
  mock.app.get("*", (req, res) => {
    mock.requests.push(req);
    res.status(mock.status).send(mock.responseBody);
  });

  mock.server = await mock.app.listen(MOCK_SERVER_PORT);
  console.log(`Mock server started on port: ${MOCK_SERVER_PORT}`);
};

// Destroy the mock server
const teardownMock = () => {
  if (mock.server) {
    mock.server.close();
    delete mock.server;
  }
};

describe("Users", () => {
  // Our mock is started before any test starts ...
  before(async () => await initMock());

  // ... killed after all the tests are executed ...
  after(() => {
    redis.quit();
    teardownMock();
  });

  // ... and we reset the recorded requests between each test
  beforeEach(() => (mock.requests = []));

  it("should create a new user", done => {
    // The mock will tell us the email is valid in this test
    setupMock(200, { result: "valid" });

    chai
      .request(SERVER_URL)
      .post("/api/users")
      .send(TEST_USER)
      .end((err, res) => {
        // ... check response and redis as before
        createdUserId = res.body.id;

        // Verify that the API called the mocked service with the right parameters
        mock.requests.length.should.equal(1);
        mock.requests[0].path.should.equal("/api/validate");
        mock.requests[0].query.should.have.property("email");
        mock.requests[0].query.email.should.equal(TEST_USER.email);
        done();
      });
  });
});

The tests now check that the external API has been hit with the proper data during the call to our API.

We can also add other tests checking how our API behaves based on the external API response codes:

describe("Users", () => {
  it("should not create user if mail is spammy", done => {
    // The mock will tell us the email is NOT valid in this test ...
    setupMock(200, { result: "invalid" });

    chai
      .request(SERVER_URL)
      .post("/api/users")
      .send(TEST_USER)
      .end((err, res) => {
        // ... so the API should fail to create the user
        // We could test that the DB and Redis are empty here
        res.should.have.status(403);
        done();
      });
  });

  it("should not create user if spammy mail API is down", done => {
    // The mock will tell us the email checking service
    //  is down for this test ...
    setupMock(500, {});

    chai
      .request(SERVER_URL)
      .post("/api/users")
      .send(TEST_USER)
      .end((err, res) => {
        // ... in that case also a user should not be created
        res.should.have.status(403);
        done();
      });
  });
});

How you handle errors from third party APIs in your application is of course up to you. But you get the point.

To run these tests we need to tell the container myapp what is the base URL of the third party service:

  myapp:
    environment:
      APP_EXTERNAL_URL: http://myapp-tests:8002/api
    ...

  myapp-tests:
    environment:
      MOCK_SERVER_PORT: 8002
    ...

Conclusion and a few other thoughts

Hopefully this article gave you a taste of what Docker compose can do for you when it comes to API testing. The full example is here on GitHub.

Using Docker compose makes tests run fast in an environment close to production. It requires no adaptations to your component code. The only requirement is to support environment variables driven configuration.

The component logic in this example is very simple but the principles apply to any API. Your tests will just be longer or more complex. They also apply to any tech stack that can be put inside a container (that's all of them). And once you are there you are one step away from deploying your containers to production if need be.

If you have no tests right now this is how I recommend you should start: end to end testing with Docker compose. It is so simple you could have your first test running in a few hours. Feel free to reach out to me if you have questions or need advice. I'd be happy to help.

I hope you enjoyed this article and will start testing your APIs with Docker Compose. Once you have the tests ready you can run them out of the box on our continuous integration platform Fire CI.

One last idea to succeed with automated testing.

When it comes to maintaining large test suites, the most important feature is that tests are easy to read and understand. This is key to motivate your team to keep the tests up to date. Complex tests frameworks are unlikely to be properly used in the long run.

Regardless of the stack for your API, you might want to consider using chai/mocha to write tests for it. It might seem unusual to have different stacks for runtime code and test code, but if it gets the job done ... As you can see from the examples in this article, testing a REST API with chai/mocha is as simple as it gets. The learning curve is close to zero.

So if you have no tests at all and have a REST API to test written in Java, Python, RoR, .NET or whatever other stack, you might consider giving chai/mocha a try.

If you wonder how to get start with continuous integration at all, I have written a broader guide about it. Here it is: How to get started with Continuous Integration

Originally published on the Fire CI Blog.

Consumer driven contract testing is a great way to improve the reliability of interconnected systems. Integration testing becomes way easier and more self contained. It opens the door for independent deployments, and leads to faster iterations and more granular feedback. Unlike your insurance, it doesn't have any fine print. This article is about setting it up in a delivery pipeline, in the context of doing continuous delivery.

I want to show how Contract Tests help split the deployment of the front end and the back end of a small application. I have a React client and a Spring Boot backend written in Kotlin.

What is a Contract Test?

I am not talking about smart contracts. There is no blockchain whatsoever in this article. Sorry for that (Contract Tests for Smart Contracts sounds like a conference talk that the world badly needs, though!).

In a nutshell, a Contract Test is a specification of the interactions between a consumer and a provider. In our case, the communication happens using REST. The consumer defines the actions sent to the provider and the responses that will be returned. In our case, the frontend is the consumer and the backend is the provider. A contract is generated. Both sides test against this contract.

It is not really about any particular technology. There are a bunch of different frameworks, but some simple scripts could do the trick.

Why have it as part of the delivery pipeline?

First of all, running these tests continuously ensures that they keep working at all times. The big benefit, however, is that we can separate the deployment of the front end and back end. If both sides are fulfilling the contract, it is likely that they work together correctly. Thus, we can consider avoiding expensive integrated tests. They tend to work pretty badly anyways.

Setting up some contracts

There are two sides to set up, consumer and provider. The tests will run in the pipelines that build the front end and the back end, respectively. We are going to use the Pact framework for our examples, which is the tool that I am most familiar with. Because of that, I tend to use pact and contract interchangeably. Our pipelines are written for CircleCI, but they should be fairly easy to port to other CI Tools.

The consumer side

As mentioned, the consumer leads the creation of the contract. Having the client driving this might sound counterintuitive. Often, APIs are created before the clients that will use them. Flipping it around is a nice habit to get into. It forces you to really think in terms of what the client will actually do, instead of bikeshedding a super generic API that will never need most of its features. You should give it a try!

The pact is defined through interactions specified in unit tests. We specify what we expect to be sent to the back end, and then use the client code to trigger requests. Why? We can compare expectations against actual requests, and fail the tests if they don't match.

Let's have a look at an example. We are using Jest to run the tests. We'll start with some initialization code:

import path from 'path'
import Pact from 'pact'

const provider = () =>
  Pact({
    port: 8990,
    log: path.resolve(process.cwd(), 'logs', 'pact.log'),
    dir: path.resolve(process.cwd(), 'pacts'),
    spec: 2,
    consumer: 'frontend',
    provider: 'backend'
  })

export default provider

Then we have the code for an actual test. The test consists of two parts. First we define the expected interaction. This is not very different from mocking an http library, with something like axios. It specifies the request that we will send (URL, headers, body and so forth), and the response that we will get.

const interaction: InteractionObject = {
  state: 'i have a list of recipes',
  uponReceiving: 'a request to get recipes',
  withRequest: {
    method: 'GET',
    path: '/rest/recipes',
    headers: {
      Accept: 'application/json',
      'X-Requested-With': 'XMLHttpRequest'
    }
  },
  willRespondWith: {
    status: 200,
    headers: { 'Content-Type': 'application/json; charset=utf-8' },
    body: [
      {
        id: 1,
        name: 'pasta carbonara',
        servings: 4,
        duration: 35
      }
    ]
  }
}

Then we have the test itself, where we call the actual client code that will trigger the request. I like to encapsulate these requests in services that convert the raw response into the domain model that will be used by the rest of the app. Through some assertions, we make sure that the data that we are delivering from the service is exactly what we expect.

it('works', async () => {
  const response = await recipeList()

  expect(response.data.length).toBeGreaterThan(0)
  expect(response.data[0]).toEqual({
    id: 1,
    name: 'pasta carbonara',
    servings: 4,
    duration: 35
  })
})

Note that even if recipeList is properly typed with TypeScript, that won't help us here. Types disappear at runtime, so if the method is returning an invalid Recipe we won't realize it, unless we explicitly test for it.

Finally we need to define some extra methods that will ensure that the interactions are verified. If there are interactions missing, or they don't look like they should, the test will fail here. After that, all that remains is writing the pact to disk.

beforeAll(() => provider.setup())
afterEach(() => provider.verify())
afterAll(() => provider.finalize())

In the end, the pact gets generated as a JSON file, reflecting all the interactions that we have defined throughout all our tests.

Flexible matching

Our pact thus far is specifying the exact values that it will get from the backend. That won't be maintainable in the long run. Certain things are inherently harder to pin down to exact values (for example, dates).

A pact that breaks constantly will lead to frustration. We are going through this whole process to make our life easier, not harder. We'll avoid that by using matchers. We can be more flexible and define how things will look like, without having to provide exact values. Let's rewrite our previous body:

willRespondWith: {
  status: 200,
  headers: { 'Content-Type': 'application/json; charset=utf-8' },
  body: Matchers.eachLike({
    id: Matchers.somethingLike(1),
    name: Matchers.somethingLike('pasta carbonara'),
    servings: Matchers.somethingLike(4),
    duration: Matchers.somethingLike(35)
  })
}

You can be more specific. You can set the expected length of a list, use regexes and a bunch of other things.

Integrating it in the pipeline

The pact tests rely on an external process, and having multiple tests hitting it can lead to non deterministic behavior. One solution is to run all the tests sequentially:

npm test --coverage --runInBand

If you want to run the pact tests independently, we can build our own task to run them separately:

"scripts": {
  "pact": "jest --transform '{\"^.+\\\\.ts$\": \"ts-jest\"}' --testRegex '.test.pact.ts$' --runInBand"
}

Which will become an extra step in our pipeline:

jobs:
  check:
    working_directory: ~/app

    docker:
      - image: circleci/node:12.4

    steps:
      - checkout
      - run: npm
      - run: npm run linter:js
      - run: npm test --coverage --runInBand
      - run: npm pact

Storing the pact

Our pact is a json file that we are going to commit directly in the frontend repository, after running the tests locally. I've found that this tends to work well enough. Making the pipeline itself commit the pact to git does not seem to be necessary.
We'll get to extending the pact and in a second.

The provider side

At this point we have a working pact, that is being verified by the consumer. But that is only half of the equation. Without a verification from the provider side, we haven't accomplished anything. Maybe even less than that, because we might get a false sense of security!

To do this, we are going to start the back end as a development server and run the pact against it. There is a gradle provider that takes care of this. We need to configure it and provide a way of finding the pact (which is stored in the front end repository). You can fetch the pact from the internet, or from a local file, whichever is more convenient.

buildscript {
    dependencies {
        classpath 'au.com.dius:pact-jvm-provider-gradle_2.12:3.6.14'
    }
}

apply plugin: 'au.com.dius.pact'

pact {
    serviceProviders {
        api {
            port = 4003

            hasPactWith('frontend') {
                pactSource = url('https://path-to-the-pact/frontend-backend.json')
                stateChangeUrl = url("http://localhost:$port/pact")
            }
        }
    }
}

What remains is starting the server and running the pact against it, which we do with a small script:

goal_test-pact() {
  trap "stop_server" EXIT

  goal_build
  start_server

  ./gradlew pactVerify
}

start_server() {
  artifact=app.jar
  port=4003

  if lsof -i -P -n | grep LISTEN | grep :$port > /dev/null ; then
    echo "Port[${port}] is busy. Server won't be able to start"
    exit 1
  fi

  nohup java -Dspring.profiles.active=pact -jar ./build/libs/${artifact} >/dev/null 2>&1 &

  # Wait for server to answer requests
  until curl --output /dev/null --silent --fail http://localhost:$port/actuator/health; do
    printf '.'
    sleep 3
  done
}

stop_server() {
  pkill -f 'java -Dspring.profiles.active=pact -jar'
}

Fixtures

If you are running your back end in development mode, it will have to deliver some data, so that the contract is fulfilled. Even if we are not using exact matching, we have to return something, otherwise it won't be possible to verify it.

You can use mocks, but I've found that avoiding them as much as possible leads to more trustworthy results. Your app is closer to what will happen in production. So what other options are there? Remember that when we were defining interactions, we had a state. That's the cue for the provider. One way of using it is the stateChangeUrl. We can provide a special controller to initialize our back end based on the state:

private const val PATH = "/pact"

data class Pact(val state: String)

@RestController
@RequestMapping(PATH, consumes = [MediaType.APPLICATION_JSON_VALUE])
@ConditionalOnExpression("\${pact.enabled:true}")
class PactController(val repository: RecipeRepository) {
    @PostMapping
    fun setup(@RequestBody body: Pact): ResponseEntity<Map<String,String>> {
        when(body.state) {
            "i have a list of recipes" -> initialRecipes()
            else -> doNothing()
        }

        return ResponseEntity.ok(mapOf())
    }
}

Note that this controller is only active for a specific profile, and won't exist outside of it.

Integrating it in the pipeline

As with the provider, we will run the check as part of our pipeline

version: 2
jobs:
  build:

    working_directory: ~/app

    docker:
      - image: circleci/openjdk:8-jdk

    steps:

      - checkout
      - run: ./go linter-kt
      - run: ./go test-unit
      - run: ./go test-pact

There is a slight difference, though. Our contract gets generated by the consumer. That means that a change in the front end could lead to a pact that does not verify properly anymore, even though no code was changed in the back end. So ideally, a change in the pact should trigger the back end pipeline as well. I haven't found a way to represent this elegantly in CircleCI, unlike say ConcourseCI.

How the contract influences the relationship between front end and back end

It's nice that we got this set up. Never touch a running system, right? Well, we might! After all, quick change is why we invest in all this tooling. How would you introduce a change that requires extending the API?

1. We start with the client. We want to define what the client will get that is not there yet. As we learned, we do that through a test in the front end that defines the expectation for the new route, or the new fields. That will create a new version of the pact.
2. Note that at this point the back end does not fulfill the pact. A new deployment of the back end will fail. But also, the existing backend does not fulfill the pact either right now. The change you introduced has to be backwards compatible. The front end should not be relying on the changes, either.
3. Now it's time to fulfill the new pact from the back end side. If this takes a long time, you will block your deployment process, which is not good. Consider doing smaller increments in that case. Anyways, you've got to implement the new functionality. The pact test will verify that your change is what's actually expected.
4. Now that the back end is providing the new functionality, you can freely integrate it in your front end.

This flow can get a bit awkward in the beginning. It is really important to work with the smallest quantum of functionality. You don't want to block your deployment process.

Next steps

For the integration between your own front end and back end I have found this setup to be sufficient in practice. However, as complexity grows, versioning will become important. You'll want to help multiple teams collaborate more easily. For that, we can use a broker. This is a lot harder to implement, so you should ask yourself if you really need it. Don't fix problems that you don't have yet.

Conclusion

To summarize, this is the setup we arrived at:

Think about all the time you have spent writing tests to check that your back end is sending the right data. That is a lot more convenient to do with a contract. Moreover, releasing front end and back end independently means being faster, releasing smaller pieces of functionality. It might feel scary at first, but you will realize that you actually are much more aware of what's going out that way.

Once you have adopted this for one service, there is no reason not to do it for all of them. I really don't miss running costly end to end test suites just to verify that my back end is working. Here is the code that I used in the examples for the front end and the back end. It is a full running (albeit small) application. Good luck with your contracts!

Of course, having your CI/CD work everywhere is a tall order. Popular CI apps for GitHub repositories alone use a multitude of configuration languages spanning Groovy, YAML, TOML, JSON, and more… all with differing syntax, of course. Porting workflows from one tool to another is more than a one-cup-of-coffee process.

The introduction of GitHub Actions has the potential to add yet another tool to the mix; or, for the right set up, greatly simplify a CI/CD workflow.

Prior to this article, I accomplished my CD flow with several lashed-together apps. I used AWS Lambda to trigger site builds on a schedule. I had Netlify build on push triggers, as well as run image optimization, and then push my site to the public Pages repository. I used Travis CI in the public repository to test the HTML. All this worked in conjunction with GitHub Pages, which actually hosts the site.

I’m now using the GitHub Actions beta to accomplish all the same tasks, with one portable Makefile of build instructions, and without any other CI/CD apps.

Appreciating the shell

What do most CI/CD tools have in common? They run your workflow instructions in a shell environment! This is wonderful, because that means that most CI/CD tools can do anything that you can do in a terminal… and you can do pretty much anything in a terminal.

Especially for a contained use case like building my static site with a generator like Hugo, running it all in a shell is a no-brainer. To tell the magic box what to do, we just need to write instructions.

While a shell script is certainly the most portable option, I use the still-very-portable Make to write my process instructions. This provides me with some advantages over simple shell scripting, like the use of variables and macros, and the modularity of rules.

I got into the nitty-gritty of my Makefile in my last post. Let’s look at how to get GitHub Actions to run it.

Using a Makefile with GitHub Actions

To our point on portability, my magic Makefile is stored right in the repository root. Since it’s included with the code, I can run the Makefile locally on any system where I can clone the repository, provided I set the environment variables. Using GitHub Actions as my CI/CD tool is as straightforward as making Make go worky-worky.

I found the GitHub Actions workflow syntax guide to be pretty straightforward, though also lengthy on options. Here’s the necessary set up for getting the Makefile to run.

The workflow file at .github/workflows/make-master.yml contains the following:

name: make-master

on:
  push:
    branches:
      - master
  schedule:
    - cron: '20 13 * * *'

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@master
        with:
          fetch-depth: 1
      - name: Run Makefile
        env:
          TOKEN: ${{ secrets.TOKEN }}
        run: make all

I’ll explain the components that make this work.

Triggering the workflow

Actions support multiple triggers for a workflow. Using the on syntax, I’ve defined two triggers for mine: a push event to the master branch only, and a scheduled cron job.

Once the make-master.yml file is in your repository, either of your triggers will cause Actions to run your Makefile. To see how the last run went, you can also add a fun badge to the README.

One hacky thing

Because the Makefile runs on every push to master, I sometimes would get errors when the site build had no changes. When Git, via my Makefile, attempted to commit to the Pages repository, no changes were detected and the commit would fail annoyingly:

nothing to commit, working tree clean
On branch master
Your branch is up to date with 'origin/master'.
nothing to commit, working tree clean
Makefile:62: recipe for target 'deploy' failed
make: *** [deploy] Error 1
##[error]Process completed with exit code 2.

I came across some solutions that proposed using diff to check if a commit should be made, but this may not work for reasons. As a workaround, I simply added the current UTC time to my index page so that every build would contain a change to be committed.

Environment and variables

You can define the virtual environment for your workflow to run in using the runs-on syntax. The obvious best choice one I chose is Ubuntu. Using ubuntu-latest gets me the most updated version, whatever that happens to be when you're reading this.

GitHub sets some default environment variables for workflows. The actions/checkout action with fetch-depth: 1 creates a copy of just the most recent commit your repository in the GITHUB_WORKSPACE variable. This allows the workflow to access the Makefile at GITHUB_WORKSPACE/Makefile. Without using the checkout action, the Makefile won't be found, and I get an error that looks like this:

make: *** No rule to make target 'all'.  Stop.
Running Makefile
##[error]Process completed with exit code 2.

While there is a default GITHUB_TOKEN secret, this is not the one I used. The default is only locally scoped to the current repository. To be able to push to my separate GitHub Pages repository, I created a personal access token scoped to public_repo and pass it in as the secrets.TOKEN encrypted variable. For a step-by-step, see Creating and using encrypted secrets.

Portable tooling

The nice thing about using a simple Makefile to define the bulk of my CI/CD process is that it’s completely portable. I can run a Makefile anywhere I have access to an environment, which is most CI/CD apps, virtual instances, and, of course, on my local machine.

One of the reasons I like GitHub Actions is that getting my Makefile to run was pretty straightforward. I think the syntax is well done - easy to read, and intuitive when it comes to finding an option you’re looking for. For someone already using GitHub Pages, Actions provides a pretty seamless CD experience; and if that should ever change, I can run my Makefile elsewhere. ¯_(ツ)_/¯

Fun fact: I first launched my GitHub Pages site 1,018 days ago.

Since then, we’ve grown together. From early cringe-worthy commit messages, through eighty-six versions of Hugo, and up until last week, a less-than-streamlined multi-app continuous integration and deployment (CI/CD) workflow.

If you know me at all, you know I love to automate things. I’ve been using a combination of AWS Lambda, Netlify, and Travis CI to automatically build and publish this site. My workflow for the task includes:

• Build with Hugo on push to master, and on a schedule (Netlify and Lambda);
• Optimize and resize images (Netlify);
• Test with HTMLProofer (Travis CI); and
• Deploy to my separate, public, GitHub Pages repository (Netlify).

Thanks to the introduction of GitHub Actions, I’m able to do all the above with just one portable Makefile.

Next week I’ll cover my Actions set up; today, I’ll take you through the nitty-gritty of my Makefile so you can write your own.

Makefile portability

POSIX-standard-flavour Make runs on every Unix-like system out there. Make derivatives, such as GNU Make and several flavours of BSD Make also run on Unix-like systems, though their particular use requires installing the respective program. To write a truly portable Makefile, mine follows the POSIX standard. (For a more thorough summation of POSIX-compatible Makefiles, I found this article helpful: A Tutorial on Portable Makefiles.) I run Ubuntu, so I’ve tested the portability aspect using the BSD Make programs bmake, pmake, and fmake. Compatibility with non-Unix-like systems is a little more complicated, since shell commands differ. With derivatives such as Nmake, it’s better to write a separate Makefile with appropriate Windows commands.

While much of my particular use case could be achieved with shell scripting, I find Make offers some worthwhile advantages. I enjoy the ease of using variables and macros, and the modularity of rules when it comes to organizing my steps.

The writing of rules mostly comes down to shell commands, which is the main reason Makefiles are as portable as they are. The best part is that you can do pretty much anything in a terminal, and certainly handle all the workflow steps listed above.

My continuous deployment Makefile

Here’s the portable Makefile that handles my workflow. Yes, I put emojis in there. I’m a monster.

.POSIX:
DESTDIR=public
HUGO_VERSION=0.58.3

OPTIMIZE = find $(DESTDIR) -not -path "*/static/*" \( -name '*.png' -o -name '*.jpg' -o -name '*.jpeg' \) -print0 | \
xargs -0 -P8 -n2 mogrify -strip -thumbnail '1000>'

.PHONY: all
all: get_repository clean get build test deploy

.PHONY: get_repository
get_repository:
    @echo "? Getting Pages repository"
    git clone https://github.com/victoriadrake/victoriadrake.github.io.git $(DESTDIR)

.PHONY: clean
clean:
    @echo "? Cleaning old build"
    cd $(DESTDIR) && rm -rf *

.PHONY: get
get:
    @echo "❓ Checking for hugo"
    @if ! [ -x "$$(command -v hugo)" ]; then\
        echo "? Getting Hugo";\
        wget -q -P tmp/ https://github.com/gohugoio/hugo/releases/download/v$(HUGO_VERSION)/hugo_extended_$(HUGO_VERSION)_Linux-64bit.tar.gz;\
        tar xf tmp/hugo_extended_$(HUGO_VERSION)_Linux-64bit.tar.gz -C tmp/;\
        sudo mv -f tmp/hugo /usr/bin/;\
        rm -rf tmp/;\
        hugo version;\
    fi

.PHONY: build
build:
    @echo "? Generating site"
    hugo --gc --minify -d $(DESTDIR)

    @echo "? Optimizing images"
    $(OPTIMIZE)

.PHONY: test
test:
    @echo "? Testing HTML"
    docker run -v $(GITHUB_WORKSPACE)/$(DESTDIR)/:/mnt 18fgsa/html-proofer mnt --disable-external

.PHONY: deploy
deploy:
    @echo "? Preparing commit"
    @cd $(DESTDIR) \
    && git config user.email "hello@victoria.dev" \
    && git config user.name "Victoria via GitHub Actions" \
    && git add . \
    && git status \
    && git commit -m "? CD bot is helping" \
    && git push -f -q https://$(TOKEN)@github.com/victoriadrake/victoriadrake.github.io.git master
    @echo "? Site is deployed!"

Sequentially, this workflow:

1. Clones the public Pages repository;
2. Cleans (deletes) the previous build files;
3. Downloads and installs the specified version of Hugo, if Hugo is not already present;
4. Builds the site;
5. Optimizes images;
6. Tests the built site with HTMLProofer, and
7. Prepares a new commit and pushes to the public Pages repository.

If you’re familiar with command line, most of this may look familiar. Here are a couple bits that might warrant a little explanation.

Checking if a program is already installed

I think this bit is pretty tidy:

if ! [ -x "$$(command -v hugo)" ]; then\
...
fi

I use a negated if conditional in conjunction with command -v to check if an executable (-x) called hugo exists. If one is not present, the script gets the specified version of Hugo and installs it. This Stack Overflow answer has a nice summation of why command -v is a more portable choice than which.

Image optimization

My Makefile uses mogrify to batch resize and compress images in particular folders. It finds them automatically using the file extension, and only modifies images that are larger than the target size of 1000px in any dimension. I wrote more about the batch-processing one-liner in this post.

There are a few different ways to achieve this same task, one of which, theoretically, is to take advantage of Make’s suffix rules to run commands only on image files. I find the shell script to be more readable.

Using Dockerized HTMLProofer

HTMLProofer is installed with gem, and uses Ruby and Nokogiri, which adds up to a lot of installation time for a CI workflow. Thankfully, 18F has a Dockerized version that is much faster to implement. Its usage requires starting the container with the built site directory mounted as a data volume, which is easily achieved by appending to the docker run command.

docker run -v /absolute/path/to/site/:/mounted-site 18fgsa/html-proofer /mounted-site

In my Makefile, I specify the absolute site path using the default environment variable GITHUB_WORKSPACE. I’ll dive into this and other GitHub Actions features in the next post.

In the meantime, happy Making!

At Qlector we are committed to develop and deliver high quality software and take into account best engineering practices as listed in 12…

At Qlector, we are committed to developing and delivering high quality software and we take into account the best engineering practices as listed in 12 factor apps.

In the following post, we describe how we introduced Continuous Delivery. By doing so, we’ve reduced waste and the…

This article is about creating iOS builds through Jenkins BOT, remotely, without the need of a developer.

Before starting, I want to say that this is my first article. So feel free to leave a comment if something can be improved :)

Why is this a good idea?

When a developer makes a feature, they QA test it before pushing it to production. So a build has to be shared with the QA team with some test configurations.

Xcode (the IDE) takes a significant amount of time to compile and generate this build. This means that any person that needs the build would have to install the IDE, clone the repository, create a signing identity and certificate and then create the build themselves. Or depend on the developer to create one for them.

During the build creation process, the IDE is unusable. This severely impacts the productivity of the developer. In my company, the average build time of an .ipa is around 20 mins. On average, a developer makes 2–3 builds daily.
This means 5 working hours a week get wasted.

How much more time will it take to build?

But what if there was an automated system which could generate the builds on its own? This would free the developers from this responsibility. It would also make it possible for anyone to get a build easily.

Jenkins is one of the solutions to our problem.

Making builds easily available to testers and developers ensures that people are able to test features faster and ship to production more easily. This improves the productivity of development teams. It also improves the quality of products pushed to production.

Let’s get started now.

Prerequisites

You will require:

• macOS Machine (it is better to run it on Mac products)
• 10 GB of drive space (for Jenkins)
• Java 8 installed (either a JRE or Java Development Kit (JDK) is fine)
  http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html

Additional Plugins to be installed

• homebrew
• wget
• RVM Plugin
  Installation guide
  https://rvm.io/rvm/security

Make one Branch with a file in it with the name Jenkinsfile with sample code:
_node {_
_sh ‘echo HelloWorld’_
_}_
Let's name it jenkins-integration. Will come back to this later.

• Install Xcode on your machine from the App Store
• Install Fastlane on your machine. Jenkins will internally use fastlane commands to generate builds.

Now let’s go through it step by step.

Step 1. Install Jenkins on your machine

You can install on a MacBook or mac-mini. Mac-mini is preferred as it can be kept alive.

Download Jenkins -&gt; https://jenkins.io/

Run java -jar jenkins.war --httpPort=8080 in the command line. If you’re getting an error in the terminal, try a different port (for example, 9090) as sometimes some ports are not available.

Browse to http://localhost:8080 and follow the instructions to complete the installation.

Then add admin credentials and don’t forget them (as did I :P). Later you can go to Jenkins > Manage Jenkins > Manager Users and do your changes if needed.

Step 2. Creating Your first Pipeline

Create a new job and choose Pipeline Project.

To check out your project, under the section Pipeline, in Definition, choose Pipeline Script from SCM and in SCM choose Git

Then add your repo URL and add the credentials if its a private repo. In branches to build , add /*jenkins-integration, the branch which we created earlier.

Make sure Script Path is Jenkinsfile which we have created in our new branch. All the scripts will be written in this Jenkinsfile.

Click on Save and Jenkins will automatically scan your repo with the mentioned branch and will run the Jenkinsfile script.

Now we are ready to configure our Jenkinsfile to create builds

Step 3. Add parameters to the Job

User input is required for

• branch
• environment (test or prod)

For that we need to configure our project to take input parameters for a job.

Go to the Configure section and check This project is parameterised.
Then select add parameter and add the same accordingly.

When you click on save, you will see a new section on left side -&gt; Build with Parameters. This will be the user interface to make builds.

These params will be used in our Jenkins script.

Step 3. Configure Jenkins Script

Will create multiple steps in our Jenkinsfile, each having one responsibility, and it will create a nice UI when it is built.

Go to your Jenkinsfile and replace the script with the following:

First, check out the branch through the parameter which we added earlier. Add your repo and GitHub token.

Now the GitHub token should not be visible to others. To do this, go to Manage Jenkins-&gt; Configure System ->Global properties and add githubToken as an environment variable.

Then invoke the script to change the environment.

Next, invoke fastlane to clean (remove derived data, clean, delete .dsym files etc).

If code signing is required, do that next using ad-hoc. You can use development or app store based on your needs.

Next, create builds using the gym command in fastlane.

Step 4. Run the Job

Now our script is ready. Go to Jenkins and open Build with Parameters.

It will start to run the script and will create a nice UI with multiple steps as mentioned in the Jenkinsfile.

When the job is completed, go to the project Users/agammahajan/.jenkins/workspace/iOS_Build_Systems
and you will see that the .ipa has been created. Voilà!

Now you can share this build with others. You can use the Slack plugin to upload the builds to Slack if you want.

Wrapping up

So to conclude here, we can see how easy it is to setup an automated bot which enables any person to trigger builds in just 2 steps: Give Branch->Test Environment-> Done.

This has helped me and my fellow developers improve productivity and ship faster. It has also helped the QA team, so that they don’t have to depend on developers every time they need to test something. I hope it benefits you and your company also.

From here, the possibilities are endless.

1. You can make scheduled jobs to generate nightly builds.
2. Upload builds directly to the App Store.
3. Cache the builds, so builds with the same configuration are not generated again.
4. Distributing the IPA in house for OTA (Over the air) installation.
5. Make a CI-CD pipeline to run automated tests on every commit and make them production ready.