This can be daunting in situations where multiple developers are actively working on a codebase, making changes to different branches, or managing multiple branch-specific configurations.

Like with every pull request or change pushed to a branch, you’ll need to review every line of code that’s been changed, added, or removed before deciding what gets merged or not. Configuration files in codebases are not exempt from this, making them prone to errors, as a simple change can affect your entire Continuous Integration setup.

When changes get made to the staging or production branch and a build is triggered, you’ll want to ensure that the correct resources attached to a branch are maintained. In some cases, you may need to define different redirect rules for each respective client, custom build commands, or other settings for each branch.

In this article, I’ll walk through how to manage branch-specific configurations including redirects for multiple branches automatically, using a simple bash script. I’ll also show you how to safely merge context-specific rules for your staging and production branches on Netlify.

What we’ll cover:

• Project Structure and Scenario

• What are Redirects/Rewrites?

• How Netlify Processes Redirects

  • Using the _redirects file syntax

  • Using the netlify.toml configuration file syntax

• The Problem: Managing Multiple netlify.toml Files for Different Branches

• How to Write the Script to Automatically Create Our Configuration File(s)

  • Sample Netlify.toml file

  • Step 1: Create the script folder and add the script file

  • Step 2: Modify package.json to include the script command

• How to Deploy Our Client to Netlify

  • First deployment of your project to Netlify

  • Subsequent Deployments / How to Set Up Branch Deployments

    • Step 1: Set Up Environment Variables in Netlify for each branch context — production, staging, and so on

    • Step 2: Trigger a new deploy

• Inspect Your Deployments

• Conclusion

Project Structure and Scenario

Consider a situation where you have two separate servers deployed for a project: one to serve requests to a staging environment (deployed to Render), and another to the production environment (deployed to Google Cloud Run).

You also have two separate client deployments on Netlify, each with their respective API_BASE_URLs, that are served by their respective servers as illustrated below:

The image below is a sample-project repository, with api and client folders/directories within it. This is an overview of the structure in each of the branches outlined above. Each directory contains its own package.json file, is treated as an independent component, and can be deployed to two separate services.

In your frontend deployment for each of the clients, all your requests made to endpoints that begin with /api/v1/ are routed to the server. Other routes remain within the frontend to direct you to pages within the client. So you’re required to write the correct rules to guide your client on how to process these requests. These are called redirect rules or rewrites.

What are Redirects/Rewrites?

Redirects, or rewrites, are rules you can create to have certain URLs automatically go to a new location anywhere on the internet (source: WPengine). These are also generally known as URL forwarding and you can use them anywhere – in entire websites, sections of a website, or an entire web application.

In web applications, redirects are often utilized to determine how to process requests. Web hosting platforms such as Netlify and Vercel use them as well, giving developers the option to determine how their web applications process requests.

How Netlify Processes Redirects

Netlify has two possible ways to specify redirect rules. You can do it using the _redirects file syntax or using the netlify.toml configuration file syntax. They achieve the same goal, but the netlify.toml syntax gives you more options and capabilities.

Using the _redirects file syntax

If you opt to use the redirect syntax, you should simply create a _redirects file in the public folder of your client app, and specify the redirect rules within it. That’s as easy as it gets. Below is an example of a redirect rule within the file:

The above rule can be interpreted as:

1. Send all my requests that match /api/v1 to the API URL specified, and return a 200 success status code. The asterisks (*) after /api/v1/ as seen in /api/v1/* instruct it to append any other extension of the original URL to the stated API URL. For example, if you have a /api/v1/users route in your frontend, that request will be redirected to https://your-api-base-url.com/api/v1/users. The :splat seen in the API URL is simply a placeholder.

2. Serve all other default routes through the index.html folder. This is required to ensure that you don’t encounter broken pages when you navigate to other pages and attempt to visit the previous page using the “back” button.

Using the netlify.toml configuration file syntax

The netlify.toml configuration file gives you a lot more flexibility when specifying redirect rules, including but not limited to matching the original request route, the required destination, the preferred status code response, header rules, signatures, country restrictions, roles and more.

This is a sample netlify.toml file sourced from Netlify’s documentation:

Quick Note: using the redirects file for redirecting certain requests to our API is perfectly fine. But it can be considered a security risk adding our API URL in plain text in the redirects file if the API_BASE_URL is supposed to be private. This is because any file in the public folder is what it sounds like – public – and anyone can access it.

If the direct locations you desire to have in your app are public URLs, then feel free to utilize the _redirects file syntax. But if you prefer to have a private URL(s), utilizing a netlify.toml configuration file in combination with the environment variables is generally a better idea.

The Problem: Managing Multiple netlify.toml Files for Different Branches

When you use the netlify.toml file to define your build commands and environment-specific settings, and you make changes that are pushed to your repository and open pull requests, you have to manually ignore or edit each netlify.tomlin each branch. This eventually becomes very stressful and susceptible to errors.

In addition to this, we want to avoid having our API URLs hardcoded in either our _redirects or netlify.tomlfile within our project codebase for security reasons. We will use environment variables as provided within our Netlify UI for production and staging contexts.

To avoid the above problems, we will use a small script in our codebase to dynamically generate the correct netlify.toml files for each branch. This approach eliminates conflicts and removes the need for manual intervention when switching between branches or handling pull requests.

How to Write the Script to Automatically Create Our Configuration File(s)

Sample Netlify.toml file

Below is a screenshot of a sample netlify.toml file we are trying to achieve for each build. You can see that all our requests that match api/v1/ in our codebase will be routed to our API.

You could have your API endpoint requests structured differently, for example /api/your-endpoint – just make sure to adjust the script accordingly. In this sample project, we use api/v1/your-endpoint as our structure.

Step 1: Create the script folder and add the script file

In the client directory, create a scripts/ directory and a configure-netlify.sh script file. You are required to do this for each branch in your repo. The content remains the same.

Open the configure-netlify.sh script file and paste the following content within it:

#!/bin/bash

# Ensure API_BASE_URL is set
if [ -z "$API_BASE_URL" ]; then
  echo "Error: API_BASE_URL environment variable is not set."
  exit 1  # Exit the script to stop the deployment
fi

echo "Using API endpoint: $API_BASE_URL"

# Define the desired Netlify configuration
NETLIFY_CONFIG="
[build]
  command = \"npm install && npm run build\"
  base = \"client\"
  publish = \"dist\"

[[redirects]]
  from = \"/api/v1/*\"
  to = \"$API_BASE_URL/:splat\"
  status = 200
  force = true

[[redirects]]
  from = \"/*\"
  to = \"/index.html\"
  status = 200
"

# Create or update the netlify.toml file
if [ ! -f "netlify.toml" ]; then
  echo "Creating netlify.toml file..."
else
  echo "Updating existing netlify.toml file..."
fi

echo "$NETLIFY_CONFIG" > netlify.toml

# Confirm successful configuration
echo "netlify.toml file has been configured successfully!"

The script does the following:

1. It checks the environment variables to ensure that the API_BASE_URL is set. If this isn’t set, it exits the build and causes it to fail. We did this to ensure that you don’t mistakenly create a successful deployment but with invalid URLs in production.

2. It then creates the content of the netlify.toml file as shown in the sample above. If your API endpoints use a different structure from api/v1/your-endpoint, you can adjust the script to fit your desired structure.

3. It checks if there already exists a netlify.toml file. If it doesn’t exist, it creates one and writes the content into it. If it exists, it updates it with the correct content during the build, using the API_BASE_URL set in the environment variables.

Step 2: Modify package.json to include the script command

To integrate this script with your build process, we will add a script command to the package.json file to call this script before running the actual build.

Add the configure-netlify command as follows: "configure-netlify": "bash scripts/configure-netlify.sh" within the scripts in your package.json file.

Update your build command to run the script before running the actual build: "build": "npm run configure-netlify && vite build".

Don’t forget to save these changes and push them to your remote repository.

How to Deploy Our Client To Netlify

When deploying our client to Netlify, we are given three options:

1. importing an existing project (that is, a project that exists on a git repository service such as GitHub and GitLab),

2. importing from a template, or

3. manually deploying a static site using the Netlify drop (drag and drop) interface.

For the configuration in our repository to work as expected during our build process, you’ll need to use the option that requires importing from an existing project such as GitHub. Using the drag-and-drop interface won’t work. If you must use this, then opt for the _redirects file syntax option to define your redirects.

First deployment of your project to Netlify

When deploying your project for the first time, you are given the option of deploying only one branch initially. You can only add and specify other options, such as other branches, in subsequent deployments.

To deploy your project, take the following steps:

1. Sign in to Netlify > netlify.com

2. Click "Add new site" > "Import an existing project" > "Deploy with GitHub"

3. Click "Configure Netlify on GitHub" > Search for your repository > Select it

4. Enter a unique site name for your project

5. Configure deploy settings. Here you are required to select the preferred branch to deploy. For the initial deployment, we will deploy the main branch which we use as the production branch.

   • Branch: main/master

   • Build command: npm run build

   • Publish directory: dist (Select the directory where your static file lives. In this sample project, it’s exported into the dist directory. Some tools export into build)

6. Enter the environment variables for your project. Don’t forget to enter your API_BASE_URL from your server. This is an essential requirement according to the bash script.

7. Click "Deploy site"

Your project should deploy correctly, and you’ll be able to see the netlify.toml configuration generated by the script by inspecting the deployment details at the bottom of the successful deployment page.

You can download this file to your local machine to see the configuration generated. It should match the sample netlify.toml file above. You can also test that it works using your generated site link.

Subsequent Deployments / How to Set Up Branch Deployments

Step 1: Set Up Environment Variables in Netlify for each branch context  — production, staging, and so on

When your project has been deployed successfully, you can set up deployments for your staging branch. To edit the configurations, you’ll need to:

1. Navigate to the list of your sites

2. Select your successfully deployed site

3. Click on “site configuration” on the left menu

4. Select “environment variables” > click the “Add a variable” button.

You will be given the option of adding variables individually or importing an entire .env file. You can choose either option. In the image below, I’ve selected “Import from a .env file”.

Seeing that our production site, deployed from the main branch (with the production environmental variables), has already been deployed, you’ll need to:

1. Uncheck the production branch (to prevent replacing the initially deployed main branch. Be careful not to mix up your environment variables for different branches)

2. Select “branch deploys”

3. Copy and paste the content of your .env file in the input section

4. Don’t forget to add the API_BASE_URL environment variable for your staging environment

Note that in selecting branch deploys, the environment variables imported here will affect all branch deploys, apart from the production branch. You can further customize your contexts by selecting a custom branch, but that’s an entirely different approach which may require you to further customize your netlify.toml configuration file or the bash script.

If you decide to import each environment variable individually, you are given a similar option as seen below. Ensure that you select the correct context for each branch.

DON’T USE THE SAME VALUES FOR ALL CONTEXTS. As seen in the image below, selecting “different value for each deploy context” will allow you to define the values for each one. In this case, we define the values for branch deploys. Your initially used production variable should already exist.

When all the variables have been imported, you can inspect them to confirm that they have been correctly imported by selecting the dropdown on the right beside each variable and inspecting their values.

Step 2: Trigger a new deploy

When all your environment variables have been imported for the different contexts – production and staging in this case – navigate to “deploys” on the left panel of your screen. Then hit the “Trigger deploy” button, clear the cache, and initiate a new deployment.

Inspect Your Deployments

You can confirm that your script works as expected by selecting any of the deployments and selecting the building dropdown in the “Deploy log”. You will see the command run, as well as your output and API URL for that deployment, as defined by your context.

Conclusion

By following the steps in this guide, and using your script and updated commands in each branch in your repo, when you push changes then Netlify will automatically generate or update the netlify.tomlfile in each branch. This ensures that the correct configurations and environment variables for each environment are used at build time.

Your script remains the same across all the branches. This lets you focus on other code changes while your script handles the correct configuration for you safely and easily.

Push changes to any branch to see this in action.

Feel free to connect with me on Twitter (@francisihej) or LinkedIn!

These terms might seem complex or interchangeable at first glance, leaving you wondering: What do they actually mean? How do they differ from one another? 🤔

In this handbook, I’ll break down these concepts in a clear and approachable way, drawing on relatable analogies to make each term easier to understand. 🧠💡 Beyond just theory, we’ll dive into a hands-on tutorial where you’ll learn how to set up a CI/CD workflow step by step.

Together, we’ll:

• Set up a Node.js project. ✨

• Implement automated tests using Jest and Supertest. 🛠️

• Set up a CI/CD workflow using GitHub Actions, triggered on push, and pull requests, or after a new release. ⚙️

• Build and publish a Docker image of your application to Docker Hub. 📦

• Deploy your application to a staging environment for testing. 🚀

• Finally, roll it out to a production environment, making it live! 🌐

By the end of this guide, not only will you understand the difference between CI/CD concepts, but you’ll also have practical experience in building your own automated pipeline. 😃

Table of Contents

1. What is Continuous Integration, Deployment, and Delivery?

2. Differences Between Continuous Integration, Continuous Delivery, and Continuous Deployment

3. How to Set Up a Node.js Project with a Web Server and Automated Tests

4. How to Create a GitHub Repository to Host Your Codebase

5. How to Set Up the CI and CD Workflows Within Your Project

6. Set Up a Docker Hub Repository for the Project's Image and Generate an Access Token for Publishing the Image

7. Create a Google Cloud Account, Project, and Billing Account

8. Create a Google Cloud Service Account to Enable Deployment of the Node.js Application to Google Cloud Run via the CD Pipeline

9. Create the Staging Branch and Merge the Feature Branch into It (Continuous Integration and Continuous Delivery)

10. Merge the Staging Branch into the Main Branch (Continuous Integration and Continuous Deployment)

11. Conclusion

What is Continuous Integration, Deployment, and Delivery? 🤔

Continuous Integration (CI)

Imagine you’re part of a team of six developers, all working on the same project. Without a proper system, chaos would ensue.

Let’s say Mr. A is building a new login feature, Mrs. B is fixing a bug in the search bar, and Mr. C is tweaking the dashboard UI—all at the same time. If everyone is editing the same "folder" or codebase directly, things could go horribly wrong: "Hey! Who just broke the app?!" 😱

To keep everything in order, teams use Version Control Systems (VCS) like GitHub, GitLab, or BitBucket. Think of it as a digital workspace where everyone can safely collaborate without stepping on each other’s toes. 🗂️✨

Here’s how Continuous Integration fits into this process step-by-step:

1. The Main Branch: The General Folder ✨

At the heart of every project is the main branch—the ultimate source of truth. It contains the stable codebase that powers your live app. It’s where every team member contributes their work, but with one important rule: only tested and approved code gets merged here. 🚀

2. Feature Branches: Personal Workspaces 🔨

When someone like Mr. A wants to work on a new feature, they create a feature branch. This branch is essentially a personal copy of the main branch where they can tinker, write code, and test without affecting others. Mrs. B and Mr. C are also working on their own branches. Everyone’s experiments stay neatly organized. 🧪💡

3. Merging Changes: The CI Workflow 🎉

When Mr. A is satisfied with his feature, he doesn’t just shove it into the main branch—CI ensures it’s done safely:

• Automated Tests: Before merging, CI tools automatically run tests on Mr. A’s code to check for bugs or errors. Think of it as a bouncer guarding the main branch, ensuring no bad code gets in. 🕵️‍♂️

• Build Verification: The feature branch code is also "built" (converted into a deployable version of the app) to confirm it works as intended.

Once these checks are passed, Mr. A’s feature branch is merged into the main branch. This frequent merging of changes is what we call Continuous Integration.

Continuous Delivery (CD)

Continuous Delivery (CD) often gets mixed up with Continuous Deployment, and while they share similarities, they serve distinct purposes in the development lifecycle. Let’s break it down! 🧐

The Need for a Staging Area 🌉

In the Continuous Integration (CI) process we discussed above, we primarily dealt with feature branches and the main branch. But directly merging changes from feature branches into the main branch (which powers the live product) can be risky. Why? 🛑

While automated tests and builds catch many errors, they’re not foolproof. Some edge cases or bugs might slip through unnoticed. This is where the staging branch and staging environment come into play! 🎭

Think of the staging branch as a “trial run.” Before unleashing changes to real customers, the codebase from feature branches is merged into the staging branch and deployed to a staging environment. This environment is an exact replica of the production environment, but it’s used exclusively by the Quality Assurance (QA) team for testing.

The QA team takes the role of a “test driver,” running the platform through its paces just as a real user would. They check for usability issues, edge cases, or bugs that automated tests might miss, and provide feedback to developers for fixes. 🚦 If everything passes, the codebase is cleared for deployment to production.

Continuous Delivery in Action 📦

The process of merging changes into the staging branch and deploying them to the staging environment is what we call Continuous Delivery. 🛠️ It ensures that the application is always in a deployable state, ready for the next step in the pipeline.

Unlike Continuous Deployment (which we’ll discuss later), Continuous Delivery doesn’t automatically push changes to production (live platform). Instead, it pauses to let humans—namely the QA team or stakeholders—decide when to proceed. This adds an extra layer of quality assurance, reducing the chances of errors making it to the live product. 🕵️‍♂️

Continuous Deployment (CD)

Continuous Deployment (CD) takes automation to its peak. While it shares similarities with Continuous Delivery, the key difference lies in the final step: there’s no manual approval required. The final process—merging the codebase and deploying it live for end users (the QA testers or the team lead could do this).

Let’s explore what makes Continuous Deployment so powerful (and a little scary)! 😅

The Last Mile of the CI/CD Pipeline 🛣️

Imagine you’ve gone through the rigorous process of Continuous Integration: teammates have merged their feature branches, automated tests were run, and the codebase was successfully deployed to the staging environment during Continuous Delivery.

Now, you’re confident that the application is free of bugs and ready to shine in the production environment—the live version of your platform used by real customers.

In Continuous Deployment, this final step of deploying changes to the live environment happens automatically. The pipeline triggers whenever specific events occur, such as:

• A Pull Request (PR) is merged into the main branch.

• A new release version is created.

• A commit is pushed directly to the production branch (though this is rare for most teams).

Once triggered, the pipeline springs into action, building, testing, and finally deploying the updated codebase to the production environment. 📡

Differences Between Continuous Integration, Continuous Delivery, and Continuous Deployment 🔍

Now that we’ve untangled the mysteries of Continuous Integration, Continuous Delivery, and Continuous Deployment, it’s time to roll up our sleeves and put theory into practice 😁.

How to Set Up a Node.js Project with a Web Server and Automated Tests ✨

In this hands-on section, we’ll build a Node.js web server with automated tests using Jest. From there, we’ll create a CI/CD pipeline with GitHub Actions that automates testing for every pull request to the staging and main branches. Finally, we’ll publish an Image of our application to DockerHub and deploy the image to Google Cloud Run, first to a staging environment for testing and later to the production environment for live use.

Ready to bring your project to life? Let’s get started! 🚀✨

Step 1: Install Node.js 📥

To get started, you’ll need to have Node.js installed on your machine. Node.js provides the JavaScript runtime we’ll use to create our web server.

1. Visit https://nodejs.org/en/download/package-manager

2. Choose your operating system (Windows, macOS, or Linux) and download the installer.

3. Follow the installation instructions to complete the setup.

To verify that Node.js was installed successfully, open your terminal and run node -v. This should display the installed version of Node.js

Step 2: Clone the Starter Repository 📂

The next step is to grab the starter code from GitHub. If you don’t have Git installed, you can download it at https://git-scm.com/downloads. Choose your OS and follow the instructions to install Git. Once you’re set, it’s time to clone the repository.

Run the following command in your terminal to clone the boilerplate code:

git clone --single-branch --branch initial https://github.com/onukwilip/ci-cd-tutorial

This will download the project files from the initial branch, which contains the starter template for our Node.js web server.

Navigate into the project directory:

cd ci-cd-tutorial

Step 3: Install Dependencies 📦

Once you’re in the project directory, install the required dependencies for the Node.js project. These are the packages that power the application:

npm install --force

This will download and set up all the libraries specified in the project. Alright, dependencies installed? You’re one step closer!

Step 4: Run Automated Tests ✅

Before diving into the code, let’s confirm that the automated tests are functioning correctly. Run:

npm test

You should see two successful test results in your terminal. This indicates that the starter project is correctly configured with working automated tests.

Step 5: Start the Web Server 🌐

Finally, let’s start the web server and see it in action. Run the following command:

npm start

Wait for the application to start running. Open your browser and visit http://localhost:5000. 🎉 You should see the starter web server up and running, ready for your CI/CD magic:

How to Create a GitHub Repository to Host Your Codebase 📂

Step 1: Sign In to GitHub

1. Go to GitHub: Open your browser and visit GitHub - https://github.com.

2. Sign In: Click on the Sign In button in the top-right corner and enter your username and password to log in, OR create an account if you don’t have one by clicking the Sign up button.

Step 2: Create a New Repository

Once you're signed in, on the main GitHub page, you’ll see a "+" sign in the top-right corner next to your profile picture. Click on it, and select “New repository” from the dropdown.

Now it’s time to set the repository details. You’ll include:

• Repository Name: Choose a name for your repository. For example, you can call it ci-cd-tutorial.

• Description (Optional): You can add a short description, like “A tutorial project for CI/CD with Docker and GitHub Actions.”

• Visibility: Choose whether you want your repository to be public (accessible by anyone) or private (only accessible by you and those you invite). For the sake of this tutorial, make it public.

• Do Not Check the Add a README File Box: Important: Make sure you do not check the option to Add a README file. This will automatically create a README.md file in your repository, which could cause conflicts later when you push your local files. We'll add the README file manually if needed later.

After filling out the details, click on “Create repository”.

Step 3: Change the Remote Destination and Push to Your New Repository

Update the Remote Repository URL:

Since you've already cloned the codebase from my repository, you need to update the remote destination to point to your newly created GitHub repository.

Copy your repository URL (the URL of the page you were redirected to after creating the repository). It should look similar to this: https://github.com/<username>/<repo-name>.

Open your terminal in the project directory and run the following commands:

git remote set-url origin <your-repo-url>

Replace <your-repo-url> with your GitHub repository URL which you copied earlier.

Rename the Current Branch to main:

If your branch is named something other than main, you can rename it to main using:

git branch -M main

Push to Your New Repository:

Finally, commit any changes you’ve made and push your local repository to the new remote GitHub repository by running:

git add .
git commit -m 'Created boilerplate'
git push -u origin main

Now your local codebase is linked to your new GitHub repository, and the files are successfully pushed there. You can verify by visiting your repository on GitHub.

How to Set Up the CI and CD Workflows Within Your Project ⚙️

Now it’s time to create the CI and CD workflows for our project! These workflows won’t run on your local PC but will be automatically triggered and executed in the cloud once you push your changes to the remote repository. GitHub Actions will detect these workflows and run them based on the triggers you define.

Step 1: Prepare the Workflow Directory 📂

Before adding the CI/CD pipelines, it's a good practice to first create a feature branch. This step mirrors the workflow commonly used in teams, where new features or changes are made in separate branches before they are merged into the main codebase.

To create and switch to a new branch, run the following command:

git checkout -b feature/ci-cd-pipeline

This will create a new branch called feature/ci-cd-pipeline and switch to it. Now, you can safely add and test the CI/CD workflows without affecting the main branch.

Once you finish, you’ll be able to merge this feature branch back into main or staging as part of the pull request process.

In the project’s root directory, create a folder named .github. Inside .github, create another folder called workflows.

Any YAML file placed in the .github/workflows directory is automatically recognized as a GitHub Actions workflow. These workflows will execute based on specific triggers, such as pull requests, pushes, or releases.

Step 2: Create the Continuous Integration Workflow 🚀

We’ll now create a CI workflow that automatically tests the application whenever a pull request is made to the main or staging branches.

First, inside the workflows directory, create a file named ci-pipeline.yml.

Paste the following code into the file:

name: CI Pipeline to staging/production environment
on:
  pull_request:
    branches:
      - staging
      - main
jobs:
  test:
    runs-on: ubuntu-latest
    name: Setup, test, and build project
    env:
      PORT: 5001
    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Install dependencies
        run: npm ci

      - name: Test application
        run: npm test

      - name: Build application
        run: |
          echo "Run command to build the application if present"
          npm run build --if-present

Explanation of the CI Workflow

Here’s a breakdown of each section in the workflow:

1. name: CI Pipeline to staging/production environment: This is the title of your workflow. It helps you identify this pipeline in GitHub Actions.

2. on: The on parameter is what determines the events that trigger your workflow. When the workflow YAML file is pushed to the remote GitHub repository, GitHub Actions automatically registers the workflow using the configured triggers in the on field. These triggers act as event listeners that tell GitHub when to execute the workflow

   For example:

   If we set pull_request as the value for the on parameter and specify the branches we want to monitor using the branches key, GitHub sets up event listeners for pull requests to those branches.

    on:
      pull_request:
        branches:
          - main
          - staging
   

   This configuration means that GitHub will trigger the workflow whenever a pull request is made to the main or staging branches.

   Multiple Triggers:
   You can define multiple event listeners in the on parameter. For instance, in addition to pull requests, you can add a listener for push events.

    on:
      pull_request:
        branches:
          - main
          - staging
      push:
        branches:
          - main
   

   This configuration ensures that the workflow is triggered when:

   • A pull request is made to either the main or staging branch.

   • A push is made directly to the main branch.

📘 Learn more about triggers: Check out the official GitHub documentation here.

3. jobs: The jobs section outlines the specific tasks (or jobs) that the workflow will execute. Each job is an independent unit of work that runs on a separate virtual machine (VM). This isolation ensures a clean, unique environment for every job, avoiding potential conflicts between tasks.

   Key Points About Jobs:

   1. Clean VM for Each Job: When GitHub Actions runs a workflow, it assigns a dedicated VM instance to each job. This means the environment is reset for every job, ensuring there’s no overlap or interference between tasks.

   2. Multiple Jobs: Workflows can have multiple jobs, each responsible for a specific task. For example:

      • A Test job to install dependencies and run automated tests.

      • A Build job to compile the application.

   3. Job Organization: Jobs can be organized to run:

      • Sequentially: Ensures one job is completed before the next starts, for example the Test job must finish before the Build job. This sequential flow mimics the "pipeline" structure.

      • Simultaneously: Multiple jobs can run in parallel to save time, especially if the jobs are independent of one another.

   4. Single Job in This Workflow: In our current workflow, there is only one job, test, which:

      • Installs dependencies.

      • Runs automated tests.

      • Builds the application.

📘 Learn more about jobs: Dive into the GitHub Actions jobs documentation here.

4. runs-on: ubuntu-latest: Specifies the operating system the job will run on. GitHub provides pre-configured virtual environments, and we’re using the latest Ubuntu image.

5. env: Sets environment variables for the job. Here, we define the PORT variable used by our application.

6. Steps: Steps define the individual actions to execute within a job:

   • Checkout: Uses the actions/checkout action to clone the repository containing the codebase in the feature branch into the virtual machine instance environment. This step ensures the pipeline has access to the project files.

   • Install dependencies: Runs npm ci to install the required Node.js packages.

   • Test application: Runs the automated tests using the npm test command. This validates the codebase for errors or failing test cases.

   • Build application: Builds the application if a build script is defined in the package.json. The --if-present flag ensures this step doesn’t fail if no build script is present.

Now that we’ve completed the CI pipeline, which runs on pull requests to the main or staging branches, let’s move on to setting up the Continuous Delivery (CD) and Continuous Deployment pipelines. 🚀

Step 3: The Continuous Delivery and Deployment Workflow

First, create the Pipeline File:
In the .github/workflows folder, create a new file called cd-pipeline.yml. This file will define the workflows for automating delivery and deployment.

Next, paste the configuration:
Copy and paste the following configuration into the cd-pipeline.yml file:

name: CD Pipeline to Google Cloud Run (staging and production)
on:
  push:
    branches:
      - staging
  workflow_dispatch: {}
  release:
    types: published

env:
  PORT: 5001
  IMAGE: ${{vars.IMAGE}}:${{github.sha}}
jobs:
  test:
    runs-on: ubuntu-latest
    name: Setup, test, and build project
    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Install dependencies
        run: npm ci

      - name: Test application
        run: npm test
  build:
    needs: test
    runs-on: ubuntu-latest
    name: Setup project, Authorize GitHub Actions to GCP and Docker Hub, and deploy
    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Authenticate for GCP
        id: gcp-auth
        uses: google-github-actions/auth@v0
        with:
          credentials_json: ${{ secrets.GCP_SERVICE_ACCOUNT }}

      - name: Set up Cloud SDK
        uses: google-github-actions/setup-gcloud@v0

      - name: Authenticate for Docker Hub
        id: docker-auth
        env:
          D_USER: ${{secrets.DOCKER_USER}}
          D_PASS: ${{secrets.DOCKER_PASSWORD}}
        run: |
          docker login -u $D_USER -p $D_PASS
      - name: Build and tag Image
        run: |
          docker build -t ${{env.IMAGE}} .
      - name: Push the image to Docker hub
        run: |
          docker push ${{env.IMAGE}}
      - name: Enable the Billing API
        run: |
          gcloud services enable cloudbilling.googleapis.com --project=${{secrets.GCP_PROJECT_ID}}
      - name: Deploy to GCP Run - Production environment (If a new release was published from the master branch)
        if: github.event_name == 'release' && github.event.action == 'published' && github.event.release.target_commitish == 'main'
        run: |
          gcloud run deploy ${{vars.GCR_PROJECT_NAME}} \
          --region ${{vars.GCR_REGION}} \
          --image ${{env.IMAGE}} \
          --platform "managed" \
          --allow-unauthenticated \
          --tag production \
      - name: Deploy to GCP Run - Staging environment
        if: github.ref != 'refs/heads/main'
        run: |
          echo "Deploying to staging environment"
          # Deploy service with to staging environment
          gcloud run deploy ${{vars.GCR_STAGING_PROJECT_NAME}} \
          --region ${{vars.GCR_REGION}} \
          --image ${{env.IMAGE}} \
          --platform "managed" \
          --allow-unauthenticated \
          --tag staging \

The CD pipeline configuration combines Continuous Delivery and Continuous Deployment workflows into a single file for simplicity. It builds on the concepts of CI/CD we discussed earlier, automating testing, building, and deploying the application to Google Cloud Run.

Explanation of the CD pipeline:

1. Workflow Triggers (on)

• push: Workflow triggers on pushes to the staging branch.

• workflow_dispatch: Enables manual execution of the workflow via the GitHub Actions interface.

• release: Triggers when a new release is published.
  Example: When a release is published from the main branch, the app deploys to the production environment.

2. Job 1 – Testing the Codebase: The first job in the pipeline, Test, ensures the codebase is functional and error-free before proceeding with delivery or deployment

3. Job 2 – Building and Deploying the Application: Aha! Moment ✨: These jobs run sequentially. 😃 The Build job begins only after the Test job is completed successfully. It prepares the application for deployment and manages the actual deployment process.

   Here's what happens:

   • Authorization for GCP and Docker Hub: The workflow authenticates with both Google Cloud Platform (GCP) and Docker Hub. For GCP, it uses the google-github-actions/auth@v0 action to handle service account credentials stored as secrets. Similarly, it logs into Docker Hub with stored credentials to enable image uploads.

   • Build and Push Docker Image: The application is built into a Docker image and tagged with a unique identifier (${{env.IMAGE}}). This image is then pushed to Docker Hub, making it accessible for deployment.

   • Deploy to Google Cloud Run: Based on the event that triggered the workflow, the application is deployed to either the staging or production environment in Google Cloud Run. A push to the staging branch deploys to the staging environment (Continuous Delivery), while a release from the main branch deploys to production (Continuous Deployment).

To ensure the security and flexibility of our pipeline, we rely on external variables and secrets rather than hardcoding sensitive information directly into the workflow file.

Why? Workflow configuration files are part of your repository and accessible to anyone with access to the codebase. If sensitive data, like API keys or passwords, is exposed here, it can be easily compromised. 😨

Instead, we use GitHub’s Secrets to securely store and access this information. Secrets allow us to define variables that are encrypted and only accessible by our workflows. For example:

• DockerHub Credentials: We’ll add a Docker username and access token to the repository’s secrets. These are essential for authenticating with DockerHub to upload the built Docker images.

• Google Cloud Service Account Key: This key will grant the pipeline the necessary permissions to deploy the application on Google Cloud Run securely.

We'll set up these variables and secrets incrementally as we proceed, ensuring each step is fully secure and functional. 🎯

Set Up a Docker Hub Repository for the Project's Image and Generate an Access Token for Publishing the Image 📦

Before we dive into the steps, let’s quickly go over what we’re about to do. In this section, you’ll learn how to create a Docker Hub repository, which acts like an online storage space for your application’s container image.

Think of a container image as a snapshot of your application, ready to be deployed anywhere. To ensure smooth and secure access, we’ll also generate a special access token, kind of like a revokable password that our CI/CD pipeline can use to upload your app’s image to Docker Hub. Let’s get started! 🚀

Step 1: Sign Up for Docker Hub

Here are the steps to follow to sign up for Docker Hub:

1. Go to the Docker Hub website: Open your web browser and visit Docker Hub - https://hub.docker.com/.

2. Create an account: On the Docker Hub homepage, you’ll see a button labelled "Sign Up" in the top-right corner. Click on it.

3. Fill in your details: You'll be asked to provide a few details like your username, email address, and password. Choose a strong password that you can remember.

4. Agree to the terms: You’ll need to check a box to agree to Docker’s terms of service. After that, click “Sign Up” to create your account.

5. Verify your email: Docker Hub will send you an email to verify your account. Open that email and click on the verification link to complete your account creation.

Step 2: Sign In to Docker Hub

After verifying your email, go back to Docker Hub, and click on "Sign In" at the top right. Then you can use the credentials you just created to log in.

Step 3: Generate an Access Token (for the CI/CD pipeline)

Now that you have an account, you can create an access token. This token will allow your GitHub Actions workflow to securely sign into Docker Hub and upload Docker images.

Once you’re logged into Docker Hub, click on your profile picture (or avatar) in the top right corner. This will open a menu. From the menu, click “Account Settings”.

Then in the left-hand menu of your account settings, scroll to the "Security" tab. This section is where you manage your tokens and passwords.

Now you’ll need to create a new access token. In the Security tab, you’ll see a link labelled “Personal access tokens” – click on it. Click the button labelled “Generate new token”.

You’ll be asked to give your token a description. You can name it something like "GitHub Actions CI/CD" so that you know what it's for.

After giving it a description, click on the “Access permissions dropdown“ and select “Read & Write“, or “Read, Write, Delete“. Click “Generate“

Now, you need to copy the credentials. After clicking the generate button, Docker Hub will create an access token. Immediately copy this token along with your username and save it somewhere safe, like in a file (don’t worry, we’ll add it to our GitHub secrets). You won’t be able to see this token again, so make sure you save it!

Step 4: Add the Token to GitHub as a Secret

To do this, open your GitHub repository where the codebase is hosted. In the GitHub repo, click on the Settings tab (located near the top of your repo page).

Then on the left sidebar, scroll down and click on “Secrets and Variables”, then choose “Actions”.

Here are the steps to create and manage your new secret:

1. Add a new secret: Click on the “New repository secret” button.

2. Set up the secret:

   • In the Name field, type DOCKER_PASSWORD.

   • In the Value field, paste the access token you copied earlier.

3. Save the secret: Finally, click Add secret to save your Docker access token securely in GitHub.

Then you’ll repeat the process for your Docker username. Create a new secret called DOCKER_USER and add your Docker username that you copied earlier.

And that’s it! Now your CI/CD pipeline can use this token to securely log in to Docker Hub and upload images automatically when triggered. 🎉

Step 5: Creating the Dockerfile for the Project

Before you can build and publish the Docker image to Docker Hub, you need to create a Dockerfile that contains the necessary instructions to build your application.

Follow the steps below to create the Dockerfile in the root folder of your project:

1. Navigate to your project’s root folder.

2. Create a new file named Dockerfile.

3. Open the Dockerfile in a text editor and paste the following content into it:

FROM node:18-slim

WORKDIR /app

COPY package.json .

RUN npm install -f

COPY . .

# EXPOSE 5001
EXPOSE 5001

CMD ["npm", "start"]

Explanation of the Dockerfile:

• FROM node:18-slim: This sets the base image for the Docker container, which is a slim version of the official Node.js image based on version 18.

• WORKDIR /app: Sets the working directory for the application inside the container to /app.

• COPY package.json .: Copies the package.json file into the working directory.

• RUN npm install -f: Installs the project dependencies using npm.

• COPY . .: Copies the rest of the project files into the container.

• EXPOSE 5001: This tells Docker to expose port 5001, which is the port our app will run on inside the container.

• CMD ["npm", "start"]: This sets the default command to start the application when the container is run, using npm start.

Create a Google Cloud Account, Project, and Billing Account ☁️

In this section, we’re laying the foundation for deploying our application to Google Cloud. First, we’ll set up a Google Cloud account (don’t worry, it’s free to get started!). Then, we’ll create a new project where all the resources for your app will live.

Finally, we’ll enable billing so you can unlock the cloud services needed for deployment. Think of this as setting up your workspace in the cloud—organized, ready, and secure! Let’s dive in! ☁️

Step 1: Create or Sign in to a Google Cloud Account 🌐

First, go to Google Cloud Console. If you don’t have a Google Cloud account, you’ll need to create one.

To do this, click on Get Started for Free and follow the steps to set up your account (you’ll need to provide payment information, but Google offers $300 in free credits to get started). If you already have a Google account, simply sign in using your credentials.

Once you’ve signed in, you’ll be taken to your Google Cloud dashboard. This is where you can manage all your cloud projects and resources.

Step 2: Create a New Google Cloud Project 🏗️

At the top left of the Google Cloud Console, you’ll see a drop-down menu beside the Google Cloud logo. Click on this drop-down to display your current projects.

Now it’s time to create a new project. In the top-left corner of the pop-up modal, click on the New Project button.

You’ll be redirected to a page where you’ll need to provide some basic details for your new project. So now enter the following information:

• Project Name: Enter a name of your choice for the project (for example, gcr-ci-cd-project).

• Location: Select a location for your project. You can leave it as the default "No organization" if you're just getting started.

Once you've entered the project name, click the Create button. Google Cloud will now start creating your new project. It may take a few seconds.

Step 3: Access Your New Project 🛠️

After a few seconds, you’ll be redirected to your Google Cloud dashboard.

Click on the drop-down menu beside the Google Cloud logo again, and you should now see your newly created project listed in the modal where you can select it.

Then click on the project name (for example, gcr-ci-cd-project) to enter your project’s dashboard.

Step 4: Link A Billing Account To Your Project 💳

To access the billing page, in the Google Cloud Console, find the Navigation Menu (the three horizontal lines) at the top left of the screen. Click on it to open a list of options. Scroll down and click on Billing. This will take you to the billing section of your Google Cloud account.

If you haven't set up a billing account yet, you'll be prompted to do so. Click on the "Link a billing account" button to start the process.

Now you can create a new billing account (if you don’t have one). You’ll be redirected to a page where you can either select an existing billing account or create a new one. If you don't already have a billing account, click on "Create a billing account".

Provide the necessary details, including:

• Account name (for example, "Personal Billing Account" or your business name).

• Country: Choose the country where your business or account is based.

• Currency: Choose the currency in which you want to be billed.

  

Next, enter your payment information (credit card or bank account details). Google Cloud will verify your payment method, so make sure the information is correct.

Read and agree to the Google Cloud Terms of Service and Billing Account Terms. Once you’ve done this, click "Start billing" to finish setting up your billing account

After setting up your billing account, you’ll be taken to a page that asks you to link it to your project. Select the billing account you just created or an existing billing account you want to use. Click Set Account to link the billing account to your project.

After you’ve linked your billing account to your project, you should see a confirmation message indicating that billing has been successfully enabled for your project.

You can always verify this by returning to the Billing section in the Google Cloud Console, where you’ll see your billing account listed.

Create a Google Cloud Service Account to Enable Deployment of the Node.js Application to Google Cloud Run via the CD Pipeline 🚀

Why Do We Need a Service Account and Key? 🤔

A service account allows our CI/CD pipeline to authenticate and interact with Google Cloud services programmatically. By assigning specific roles (permissions), we ensure the service account can only perform tasks related to deployment, such as managing Google Cloud Run.

The service account key is a JSON file containing the credentials used for authentication. We securely store this key as a GitHub secret to protect sensitive information.

Step 1: Open the Service Accounts Page

Here are the steps you can follow to set up your service account and get your key:

First, visit the Google Cloud Console at https://console.cloud.google.com/. Ensure you’ve selected the correct project (e.g. gcr-ci-cd-project). To change projects, click the drop-down menu next to the Google Cloud logo at the top-left corner and select your project.

Then navigate to the Navigation Menu (three horizontal lines in the top-left corner) and click on IAM & Admin > Service Accounts.

Step 2: Create a New Service Account

Click on the "Create Service Account" button. This will open a form where you’ll define your service account details.

Next, enter the Service Account details:

• Name: Enter a descriptive name (for example, ci-cd-sa).

• ID: This will auto-fill based on the name.

• Description: Add a description to help identify its purpose, such as “Used for deploying Node.js app to Cloud Run.”

• Click Create and Continue to proceed.

Step 3: Assign Necessary Roles (Permissions)

On the next screen, you’ll assign roles to the service account. Add the following roles one by one:

• Cloud Run Admin: Allows management of Cloud Run services.

• Service Account User: Grants the ability to use service accounts.

• Service Usage Admin: Enables control over enabling APIs.

• Viewer: Provides read-only access to view resources.

To add a role:

• Click on "Select a Role".

• Use the search bar to type the role name (for example, "Cloud Run Admin") and select it.

• Repeat for all four roles.

Your screen should look similar to this:

After assigning the roles, click Continue.

Step 4: Skip Granting Users Access to the Service Account

On the next screen, you’ll see an option to grant additional users access to this service account. Click Done to complete the creation process.

Step 5: Generate a Service Account Key 🔑

You should now see your newly created service account in the list. Find the row for your service account (for example, ci-cd-sa) and click the three vertical dots under the “Actions” column. Select "Manage Keys" from the drop-down menu.

To add a new key:

• Click on "Add Key" > "Create New Key".

• In the pop-up dialog, select JSON as the key type.

• Click Create.

  

Now, download the key file. A JSON file will automatically be downloaded to your computer. This file contains the credentials needed to authenticate with Google Cloud.

Make sure you keep the key secure and store it in a safe location. Don’t share it – treat it as sensitive information.

Step 6: Add the Service Account Key to GitHub Secrets 🔒

Start by opening the downloaded JSON file using a text editor (like Notepad or VS Code). Then select and copy the entire contents of the file.

Then navigate to the repository you created for this project on GitHub. Click on the Settings tab at the top of the repository. Scroll down and find the Secrets and variables > Actions section.

Now you need to add a new secret. Click the "New repository secret" button. In the Name field, enter GCP_SERVICE_ACCOUNT. In the Value field, paste the JSON content you copied earlier. Click Add secret to save it.

Do the same for the GCP_PROJECT_ID secret, but now add your Google Project ID as the value. To get your project ID, follow these steps:

1. Navigate to the Google Cloud Console: Open Google Cloud Console at https://console.cloud.google.com/.

2. Locate the Project Dropdown: At the top-left of the screen, next to the Google Cloud logo, you will see a drop-down that shows the name of your current project.

3. View the Project ID: Click the drop-down, and you'll see a list of all your projects. Your Project ID will be displayed next to the project name. It is a unique identifier used by Google Cloud.

4. Copy the Project ID: Copy the Project ID that is displayed, and add it as the value of the GCP_PROJECT_ID secret.

Step 7: Adding External Variables to the GitHub Repository 🔧

Before proceeding with deployment, we need to define some external variables that were referenced in the CD workflow. These variables ensure that the pipeline knows critical details about your Google Cloud Run services and Docker container registry.

Here are the steps you’ll need to follow to do this:

1. First, go to your repository on GitHub.

2. Click the Settings tab at the top of the repository. Scroll down to Secrets and variables > Actions.

3. Click on the Variables tab next to Secrets. Click "New repository variable" for each variable. Then you’ll need to define these variables:

   • GCR_PROJECT_NAME: Set this to the name of your Cloud Run service for the production/live environment. For example, gcr-ci-cd-app.

   • GCR_STAGING_PROJECT_NAME: Set this to the name of your Cloud Run service for the staging/test environment. For example, gcr-ci-cd-staging.

   • GCR_REGION: Enter the region where you’d like to deploy the services. For this tutorial, set it to us-central1.

   • IMAGE: Specify the name of the Docker image/container registry where the published image will be uploaded. For example, <dockerhub-username>/ci-cd-tutorial-app.

4. After entering each variable name and value, click Add variable.

Enabling the Service Usage API on the Google Cloud Project 🌐

To deploy your application, the Service Usage API must be enabled in your Google Cloud project. This API allows you to manage Google Cloud services programmatically, including enabling/disabling APIs and monitoring their usage.

Follow these steps to enable it:

1. First, visit the Google Cloud Console at https://console.cloud.google.com/.

2. Then make sure you’re in the correct project. Click the project drop-down menu near the Google Cloud logo at the top-left corner. Select gcr-ci-cd-project , or the name you gave your project from the list of projects.

3. Next you’ll need to access the API library. Open the Navigation Menu (three horizontal lines in the top-left corner). Select APIs & Services > Library from the menu.

4. In the API Library, use the search bar to search for "Service Usage API".

5. Click on the Service Usage API from the search results. On the API’s details page, click Enable.

6. To verify, go to APIs & Services > Enabled APIs & Services in the Google Cloud Console. Confirm that the Service Usage API appears in the list of enabled APIs.

   

Create the Staging Branch and Merge the Feature Branch into It (Continuous Integration and Continuous Delivery) 🌟

When changes from the feature/ci-cd-pipeline branch are merged into the staging branch, we complete the Continuous Integration (CI) process, and the workflow ci-pipeline.yml will run. This ensures that the changes made in the feature branch are tested and integrated into a shared branch.

Once the pull request (PR) is merged into staging, the Continuous Delivery (CD) pipeline automatically triggers, deploying the application to the staging environment. This simulates how updates are tested in a safe environment before being pushed to production.

Create the staging Branch on the Remote Repository

To enable the CI/CD pipeline, we’ll first create a staging branch on the remote GitHub repository. This branch will serve as the test environment where changes are deployed before they reach the production environment.

To create the staging branch directly on GitHub, follow these steps:

1. First, navigate to your repository on GitHub. Open your web browser and go to the GitHub repository where you want to create the new staging branch.

2. Then, switch to the main branch. On the top of the repository page, locate the Branch dropdown (usually labelled as main or the current branch name). Click on the dropdown and make sure you are on the main branch.

3. Next, create the staging branch. In the same dropdown where you see the main branch, type staging into the text box. Once you start typing, GitHub will offer you the option to create a new branch called staging. Select the Create branch: staging option from the dropdown.

4. Finally, verify the branch**.** After creating the staging branch, GitHub will automatically switch to it. You should now see staging in the branch dropdown, confirming the new branch was created.

   

Merge Your Feature Branch into the Staging Branch via a Pull Request (PR)

This process combines both Continuous Integration (CI) and Continuous Delivery (CD). You will commit changes from your feature branch, push them to the remote feature branch, and then open a PR to merge those changes into the staging branch. Here's how to do it:

Step 1: Commit Local Changes on Your Feature Branch

First, you’ll want to make sure that you are on the correct branch (the feature branch) by running:

git status

If you are not on the feature/ci-cd-pipeline branch, switch to it by running:

git checkout feature/ci-cd-pipeline

Now, it’s time to add your changes you made for the commit:

git add .

This stages all changes, including new files, modified files, and deleted files.

Next, commit your changes with a clear and descriptive message:

git commit -m "Set up CI/CD pipelines for the project"

Then you can verify your commit by running:

git log

This will display your most recent commits, and you should see the commit message you just added.

Step 2: Push Your Feature Branch Changes to the Remote Repository

After committing your changes, push them to the remote repository:

git push origin feature/ci-cd-pipeline

This pushes your local changes on the feature/ci-cd-pipeline branch to the remote GitHub repository.

Once the push is successful, visit your GitHub repository in a web browser, and confirm that the feature/ci-cd-pipeline branch is updated with your new commit.

Step 3: Create a Pull Request to Merge the Feature Branch into Staging

Go to your repository on GitHub and ensure that you are on the main page of the repository.

You should see an alert at the top of the page suggesting you create a pull request for the recently pushed branch (feature/ci-cd-pipeline). Click the Compare & Pull Request button next to the alert.

Now, it’s time to choose the base and compare branches. On the PR creation page, make sure the base branch is set to staging (this is the branch you want to merge your changes into). The compare branch should already be set to feature/ci-cd-pipeline (the branch you just pushed). If they’re not selected correctly, use the dropdowns to change them.

You’ll want to come up with a good PR description for this. Write a clear title and description for the pull request, explaining what changes you're merging and why. For example:

• Title: "Merge CI/CD setup changes from feature branch"

• Description: "This pull request adds the CI/CD pipelines for GitHub Actions and Docker Hub integration to the project. It includes the configurations for both CI and CD workflows."

Now GitHub will show a list of all the changes that will be merged. Take a moment to review them and ensure everything looks correct.

If all looks good after reviewing, click on the Create pull request button. This will create the PR and notify team members (if any) that changes are ready to be reviewed and merged.

Wait a few seconds, and you should see a message indicating that all the checks have passed. Click on the link with the description "CI Pipeline to staging/production environment...". This should direct you to the Continuous Integration workflow, where you can view the steps that ran

The Continuous Integration (CI) Process

The CI process begins when a Pull Request is made to the staging branch. It triggers the GitHub Actions workflow defined in the .github/workflows/ci-pipeline.yml file. The workflow runs the necessary steps to set up the environment, install dependencies, and build the Node.js application.

It then runs automated tests (using npm test) to ensure that the changes do not break any functionality in the codebase. If all these steps are completed successfully, the CI pipeline confirms that the feature branch is stable and ready to be merged into the staging branch for further testing and deployment.

Step 4: Merge the Pull Request

If your team or collaborators are part of the project, they may review your PR. This step may involve discussing any changes or improvements. If everything looks good, a reviewer will merge the PR.

Once the PR has been reviewed and approved, you can merge the PR. To do this, just click on the Merge pull request button. Choose Confirm merge when prompted.

After merging, you can go to the staging branch to verify that the changes were successfully merged.

Navigating to the Actions Page After Merging the PR

Once you have successfully merged your pull request from the feature/ci-cd-pipeline branch into the staging branch, the Continuous Delivery (CD) pipeline will be triggered. To view the progress of the CD pipeline, navigate to the Actions tab in your GitHub repository. Here's how to do it:

1. Go to your GitHub repository.

2. At the top of the page, you will see the Actions tab next to the Code tab. Click on it.

3. On the Actions page, you will see a list of workflows that have been triggered. Look for the one labelled CD Pipeline to Google Cloud Run (staging and production). It should appear as a new run after the PR merge.

4. Click on the workflow run to view its progress and see the detailed logs for each step.

This will allow you to monitor the status of the CD pipeline and check if there are any issues during deployment.

If you look at the CD steps and workflow, you'll see that the step to deploy the application to the production environment was skipped, while the step to deploy to the staging environment was executed.

Continuous Delivery (CD) pipeline – what’s going on:

The Continuous Delivery (CD) Pipeline automates the process of deploying the application to Google Cloud Run (testing environment). This workflow is triggered by a push to the staging branch, which happens after the changes from the feature branch are merged into staging. It can also be manually triggered via workflow_dispatch or upon a new release being published.

The pipeline consists of multiple stages:

1. Test Job: The pipeline begins by setting up the environment and running tests using the npm test command. If the tests pass, the process moves forward.

2. Build Job: The next step builds the Docker image of the Node.js application, tags it, and then pushes it to Docker Hub.

3. Deployment to GCP: After the image is pushed, the workflow authenticates to Google Cloud and deploys the application. If the event is a release (that is, a push to the main branch), the application is deployed to the production environment. If the event is a push to staging, the app is deployed to the staging environment.

The CD process ensures that any changes made to the staging branch are automatically tested, built, and deployed to the staging environment, ready for further validation. When a release is published, it will trigger deployment to production, ensuring your app is always up to date.

Accessing the Deployed Application in the Staging Environment on Google Cloud Run 🌐

Once the deployment to Google Cloud Run is successfully completed, you'll want to access your application running in the staging environment. Follow these steps to find and visit your deployed application:

1. Navigate to the Google Cloud Console

Open the Google Cloud Console in your browser by visiting https://console.cloud.google.com. If you're not already signed in, make sure you log in with your Google account.

2. Go to the Cloud Run Dashboard

In the Google Cloud Console, use the Search bar at the top or navigate through the left-hand menu: Go to Cloud Run (you can type this into the search bar, or find it under Products & services > Compute > Cloud Run). Click on Cloud Run to open the Cloud Run dashboard.

3. Select Your Staging Service

In the Cloud Run dashboard, you should see a list of all your services deployed across various environments. Find the service associated with the staging environment. The name should be similar to what you defined in your workflow (for example, gcr-ci-cd-staging).

4. Access the Service URL

Once you've selected your staging service, you’ll be taken to the Service details page. This page provides all the important information about your deployed service.
On this page, look for the URL section under the Service URL heading. The URL will look something like: https://gcr-ci-cd-staging-<unique-id>.run.app.

5. Visit the Application

Click on the Service URL, and it will open your staging environment in a new tab in your browser. You can now interact with your application as if it were live, but in the staging environment.

Merge the Staging Branch into the Main Branch (Continuous Integration and Continuous Deployment) 🌐

In this section, we'll take the updates in the staging branch, merge them into the main branch, and trigger the CI/CD pipeline. This process not only ensures your changes are production-ready but also deploys them to the production/live environment. 🚀

Step 1: Push Local Changes and Open a Pull Request

Why? The first step involves merging the staging branch into the main branch. Just like in the previous Continuous Delivery process, this ensures the integration of thoroughly tested updates.

Here’s how to do it:

First, visit the GitHub repository where your project is hosted.

Then go to the Pull Requests tab. Click New Pull Request. Choose staging as the source branch (base branch) and main as the target branch. Add a clear title and description for the Pull Request, explaining why these updates are ready for production deployment.

Step 2: Continuous Integration (CI) Pipeline Execution

After merging the pull request, the Continuous Integration (CI) pipeline will automatically execute to validate that the changes are still stable when integrated into the main branch.

Pipeline Steps:

• Code Checkout: The workflow fetches the latest code from the main branch.

• Dependency Installation: The pipeline installs all required dependencies.

• Testing: Automated tests are run to validate the application's stability.

Step 3: Create a New Release

The Continuous Deployment (CD) workflow to deploy to the production environment is triggered by the creation of a new release from the main branch.

Let’s walk through the steps to create a release.

On your GitHub repository page, click on the Releases section (located under the Code tab).

Next, click Draft a new release. Set the Target branch to main. Enter a Tag version (for example, v1.0.0) following semantic versioning. Add a Release title and an optional description of the changes.

Then, click Publish Release to finalize.

Why run the Continuous Deployment pipeline on release instead of on push? 🤔

In our setup, we decided not to trigger the Continuous Deployment (CD) pipeline every time changes are pushed to the main branch. Instead, we trigger it only when a new release is created. This gives the team more control over when updates are deployed to the production environment.

Imagine a scenario where developers are working on new features—they may push changes to the main branch as part of their regular workflow, but these features might not be complete or ready for users yet. Automatically deploying every push could accidentally expose unfinished features to your users, which can be confusing or disruptive.

By requiring a release to trigger the deployment, the team gets a chance to finalize and polish all changes before they go live.

For example, developers can test new features in the staging environment, fix any issues, and merge those changes into the main branch without worrying about them immediately appearing in production. This workflow ensures that only well-tested and complete features make their way to your end users.

Ultimately, this approach helps maintain a smooth user experience. Instead of seeing half-built features or unexpected changes, users only see updates that are ready and functional. It also gives the team the flexibility to push changes to the main branch frequently—preventing merge conflicts and making collaboration easier—while keeping control over what gets deployed live. 🚀

Step 4: Navigate to the Actions Page

After the release is published, the CD pipeline for the production environment is triggered. To monitor this repeat the process taken for the Continuous Delivery workflow, follow these steps:

1. Go to the GitHub Actions tab: In your GitHub repository, click on the Actions tab.

2. Locate the deployment workflow: Look for the CD Pipeline to Google Cloud Run (staging and production) workflow. You’ll notice that the workflow has been triggered on the main branch due to the push event.

3. Open the workflow details: Click on the workflow to view detailed steps, logs, and statuses for each part of the deployment process.

This time, the Continuous delivery workflow deploys the application to the production/live environment.

Step 5: Access the Live Application

Once the deployment is complete, go to Google Cloud Console at https://console.cloud.google.com.

Navigate to Cloud Run from the menu. Select the service corresponding to the production environment (for example, gcr-ci-cd-app).

Locate the Service URL in the service details page. Open the URL in your browser to access the live application.

And now, congratulations – you’re done!

Conclusion 🌟

In this article, we explored how to build and automate a CI/CD pipeline for a Node.js application, using GitHub Actions, Docker Hub, and Google Cloud Run.

We set up workflows to handle Continuous Integration by testing and integrating code changes and Continuous Delivery to deploy those changes to a staging environment. We also containerized our app using Docker and deployed it seamlessly to Google Cloud Run.

Finally, we implemented Continuous Deployment, ensuring updates to the production environment happen only when a release is created from the main branch.

This approach gives teams the flexibility to push and test incomplete features without impacting end users. By following these steps, you've built a robust pipeline that makes deploying your application smoother, faster, and more reliable.

Study Further 📚

If you would like to learn more about Continuous Integration, Delivery, and Deployment you can check out the courses below:

• Continuous Integration and Continuous Delivery (CI/CD) (from IBM Coursera)

• GitHub Actions - The Complete Guide (from Udemy)

• Learn CI/CD by buliding a project (freeCodeCamp tutorial)

About the Author 👨‍💻

Hi, I’m Prince! I’m a software engineer passionate about building scalable applications and sharing knowledge with the tech community.

If you enjoyed this article, you can learn more about me by exploring more of my blogs and projects on my LinkedIn profile. You can find my LinkedIn articles here. And you can visit my website to read more of my articles as well. Let’s connect and grow together! 😊

In this tutorial, you'll learn what CI/CD is, and I'll help you set up a CI/CD pipeline using Husky and GitHub Actions in a Next.js application.

This tutorial assumes that you already have knowledge of React and Next.js or other modern JavaScript frameworks. You will need also a GitHub account, and basic knowledge of Git will be strongly beneficial.

If you already have a working web app that is not built with Next.js, you might still find this article useful. All the concepts and most of the configurations will work with little adaptation in apps created with other frameworks.

Here's What We'll Cover:

1. What is CI/CD?
   – What is CI?
   – What is CD?
   – What is a CI/CD pipeline and what are its benefits?
2. How to Set Up a CI/CD Pipeline
   – Step 1: Set Up a Next.js App with Vitest
   – Step 2: Set a Git Hook
   – Step 3: Create a GitHub Actions Workflow
   – Step 4: Deploy the Project
3. Conclusion

What is CI/CD?

Continuous Integration/Continuous Delivery or Continuous Deployment (CI/CD) is a practice that involves automating the process of building, testing, and deploying software.

Its main benefit is speeding up the entire development process. It also increases productivity by ensuring smooth code integration, standards, and security best practices adoption. It also helps produce a shorter feedback cycle with early issue detection, among other advantages explained below.

CI/CD is an essential tool in today’s software development practices, enabling teams to deliver high-quality software quickly, efficiently, and reliably.

Let’s learn more about it in detail.

What is CI?

Continuous Integration is a software practice that means that developers in a team merge code changes into a central repository multiple times a day.

Instead of having independent dev environments and merging at a specific time, developers frequently integrate their changes to an application into a shared branch or “trunk”.

What is CD?

The CD in CI/CD usually refers to Continuous Delivery. It's a practice that, on top of CI, automates the software integration, testing, and release process. The automation stops just before deploying to production, where a human-controlled step is needed.

But CD can also refer to Continuous Deployment, which adds automation to the step of releasing software to a production environment.

Even though CD usually refers to Continuous Delivery, both terms are sometimes used interchangeably. The difference between them is the amount of automation implemented in a project.

What is a CI/CD pipeline and what are its benefits?

When put together, these two practices create a CI/CD pipeline. Adding CI/CD to your project brings the following benefits:

• Faster development: reduces the time required to deliver new features thanks to automating the build, test and deploy.
• Enhanced Collaboration: encourages frequent code integrations and reduces integration conflicts.
• Improved Code Quality: enforces the adoption of coding standards and best practices throughout the codebase.
• Early Detection of Issues: makes the feedback cycle smaller, as issues can be caught in advance.
• Increased Productivity: prevents developers from needing to work on repetitive tasks.

These are some of the reasons why CI/CD is a core practice in modern software development and why it is such an important topic to learn about. The following steps will guide you through the process of setting up a CI/CD pipeline for your project.

How to Set Up a CI/CD Pipeline

Step 1: Set Up a Next.js App

If you already have a working web app, you can skip this and go directly to the first step.

Otherwise, let's set up a basic Next.js app with the default ESLint configuration and Vitest, and push it to a GitHub repo.

Create a Next.js app

Navigate into the directory where you want to create the new project folder, then run the following command in your terminal:

npx create-next-app@latest

When prompted with the installation options, make sure you choose to use ESLint in your project. This will ensure that ESLint is properly installed and a lint script is created in the package.json.

Wait for create-next-app to create the folder and install the project dependencies. Once it's done, navigate into the new folder and start the dev server:

cd <your-project-name>
npm run dev

Set up Vitest

Let's add Vitest to the project and add some automated tests to run in the CI/CD pipeline.

First, install vitest and the dev dependencies needed:

npm install -D vitest @vitejs/plugin-react jsdom @testing-library/react

Create a vitest.config.js file (or vitest.config.ts if using TypeScript) with the following content:

import { defineConfig } from 'vitest/config'
import react from '@vitejs/plugin-react'

export default defineConfig({
  plugins: [react()],
  test: {
    environment: 'jsdom',
  },
})

And finally, add the test script to the package.json:

 "test": "vitest --no-watch"

Note that I added the no-watch option to the test script. This prevents Vitest from starting in the default watch mode in dev environment.

Now, you can add tests for your project. If you don't know how to start, you can check out this guide for some examples.

Push the project to GitHub

Log in into your GitHub account and create new repository. Once your are done, you can connect the local repo with the one you just created, adding this repo as the remote. Then push the changes:

git add .
git commit -m "first commit"
git remote add origin git@github.com:<your-user-name>/<your-repo-name>.git
git push origin main

You should be now ready to continue to the interesting part of this tutorial. :)

Step 2: Set a Git Hook

A Git hook is a script that allows you to run some event within the Git lifecycle. In this case we will be using Husky.

Husky is a pre-commit hook for Git that allows you maintain code quality by executing some task upon committing or pushing. You can run various checks before making a commit with new changes, such as linting the code and running automated tests.

By implementing these checks, you can avoid wasting time and resources by catching issues in advance before triggering the GitHub Actions workflow.

Let’s start by adding Husky to the project with the following command:

npm install --save-dev husky

Next, let’s set up the project using the Husky init command:

npx husky init

After running this command, you will notice that a pre-commit file was created under ./husky. Also, a “prepare” script was added in the package.json.

If you open the pre-commit file inside ./husky, you will find the following content:

npm test

As its name suggests, this file contains the code that executes before completing a commit. With everything set up as described, tests will run each time you attempt to create a new commit and new commits will be added only if all tests pass.

Adding more git hooks

Now, let’s change the content in the pre-commit file so the code linter also executes before creating a new commit.

You can open your preferred code editor and add npm run lint (or the corresponding ESLint script if you’re not using Next.js) in a new line in the pre-commit file. Alternatively, you can simply run the following command from the root folder of your project:

echo "npm run lint" >> ./.husky/pre-commit

Now, each time you attempt to make a new commit, the tests and the linter will run, and the commit will be created – only if all tests are passing and no errors are found in the code.

Setting up lint-staged

You can go one step further and include a tool called lint-staged. This tool will be especially useful if your project is large, because it allows you to run the Git hooks only for staged files. In this case, it will lint only the files that will be committed, avoiding wasting time by linting the entire project.

To start using lint-staged, let's add it as a dev dependency to the project:

npm install --save-dev lint-staged

There are different ways to configure lint-staged and you can choose the one that best suits your needs. I will add a lint-staged script and object to the package.json of my project with the following content:

  "scripts": {
    "dev": "next dev",
    "build": "next build",
    "start": "next start",
    "lint": "next lint",
    "test": "vitest --no-watch",
    "prepare": "husky"
  },
  "lint-staged": {
    "*.{js, jsx,ts,tsx}": [
        "eslint --fix"
        ]
    },

Now, I can replace npm run lint with npm run lint-staged in the pre-commit file.

Each time I make a new commit, any js, jsx, ts, or tsx staged files will be linted and, if there are fixable issues, they will be automatically fixed.

Let's test that the pre-commit hook is working as expected by:

1. Running git add .
2. Running git commit
3. Waiting for the linter to run and entering a commit message when prompted
4. Running git log to confirm that the commit was properly created

If you want, you can add more checks to your pre-commit file to fit your project's needs. For example, you could run a tool like Prettier to automatically format your code, or commitlint to lint your commit messages.

Now, let’s move on to setting up a GitHub Actions workflow for the project.

Step 3: Create a GitHub Actions Workflow

With the first part complete, we can move on to the next step. Here, you will add a GitHub Actions workflow to ensure the smooth integration of changes into the entire project.

GitHub Actions Basics

GitHub Actions is a CI/CD platform that allows you to automate the building, testing, and deployment of your project. It also lets you perform actions when certain activities happen in your repository, such as opening a pull request or creating an issue.

GitHub Actions are configured through workflows defined in YAML files. These workflows typically run when triggered by an event in the repository, but they can also be scheduled or run manually.

Workflows are located in the .github/workflows folder and run different jobs. Each job includes a set of steps that run in order on the same runner or server. A step can be either a shell script or an action (a reusable piece of code that helps reduce repetitive code in your workflows).

Let's put all this together by creating the first workflow.

Creating a workflow to execute when you push to main branch

First create a .github/workflows/ under your project root. Then create a run-test.yml file. You will be adding content to this file to create a CI workflow.

The first line is optional and includes a name for the workflow. It will appear at the "Actions" tab in the GitHub repo:

name: Run linter and tests on push

Then, you will use the on key to define the event or events that will trigger the workflow run. This can be an event in your repo or a time schedule. In this case, let's set it to run each time a push to the repo happens:

on:
  push

You can also set options below the on keyword to limit the execution of a workflow to some branch or files – for example to run only on push to main branch:

on:
  push:
    branches:
      - main

Below this, you will add the jobs key. It groups all the jobs in the workflow, followed by the name of the first job, in this case run-linter-and-tests.

The lines below that define workflow properties, configuring it to run on the latest version of an Ubuntu Linux runner and grouping all the steps that run on this job.

jobs:
  run-linter-and-tests:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Install dependencies
        run: npm i

      - name: Lint code
        run: npm run lint

      - name: Run tests
        run: npm test

As mentioned before, each step can be either a shell script or an action. You can see the difference between the first and the second step in the previous code.

The first one specifies with the uses keyword that will run the actions/checkout. This action is used to checkout the repository onto the runner so the workflow can use the repository code. The second step Install dependencies uses the run keyword to tell the job to execute the npm i command on the runner.

This is the complete resulting file:

name: CI workflow
on:
  push

jobs:
  run-linter-and-tests:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: npm install
        run: npm i

      - name: Lint code
        run: npm run lint

      - name: Run tests
        run: npm test

Let's commit the changes and push them to the GitHub repository.

Now each time you push to your repository, the workflow will trigger. If you click on the "Actions" tab in your GitHub repository navigation bar, you will find a list of all the runs from all your workflows and its complete logs.

"Actions" tab in a GitHub repository navigation bar

Also, you will see that in the GitHub repository's "Code" tab, a green checkmark appears next to the last commit message. This means that workflows ran and finished successfully.

When jobs are still running, you'll see a brown dot, and a red cross when a workflow finished with an error.

Adding a second workflow to run when a PR is created

Each repository can have one or more workflows, so let's add a second workflow to run each time a PR is created. Let's run the code coverage report each time a PR is opened against the main branch of the repo.

First, create and checkout a new add-wf branch:

git checkout -b add-wf

Then, create a new YAML file under the .github/workflows directory and start adding some content on it.

First, let's add the name and when to run the workflow with the on keyword:

name: Run Coverage on PR
on: pull_request

After that, you will use the jobs keyword to describe the jobs to run. Let's define the first one as build-and-run-coverage to run in ubuntu-latest runner:

jobs:
  build-and-run-coverage:
    runs-on: ubuntu-latest

Now, let's add steps for this job:

  steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Install dependencies
        run: npm i

      - name: Build code
        run: npm run build

      - name: Run tests and coverage
        run: npm run coverage

Following is the complete resulting code:

name: Run Coverage on PR
on: pull_request

jobs:
  build-and-run-coverage:
    runs-on: ubuntu-latest

      steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Install dependencies
        run: npm i

      - name: Build code
        run: npm run build

      - name: Run tests and coverage
        run: npm run coverage

Now, you can push the change to your GitHub repo:

git add .
git commit -m 'add a wf to run on opened PR'
git push origin add-wf

Now you can open a PR against your main branch and and wait for the workflow to complete.

Comment coverage report in the PR

As mentioned earlier in this article, actions are reusable pieces of code that avoid repetitive code in the workflow. One cool thing about them is that there are many already written by the community that you can use in your workflows, saving lots of time.

To complete the workflow we created, let's add a new step that uses an action to report coverage results as a comment on the pull request.

First, let's modify the permissions keyword to ensure the workflow has the right access to content and to create comments:

 permissions:
      contents: read
      pull-requests: write

Then, let's use the Vitest Coverage Report action by adding a step into the build-and-run-coverage job:

- name: Report Coverage
        uses:  davelosert/vitest-coverage-report-action@v2

The final yaml file will look like this:

name: Run Coverage on PR
on: pull_request

jobs:
  build-and-run-coverage:
    runs-on: ubuntu-latest

    permissions:
      contents: read
      pull-requests: write

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Install dependencies
        run: npm i

      - name: Build
        run: npm run build

      - name: Run test and coverage
        run: npm run coverage

      - name: Report Coverage
        uses:  davelosert/vitest-coverage-report-action@v2

There is one more step to ensure all works as expected. You must add the json-summary reporter in the Vitest configuration:

import { defineConfig } from "vitest/config";
import react from "@vitejs/plugin-react";

export default defineConfig({
  plugins: [react()],
  test: {
    environment: "jsdom",
    coverage: {
      provider: "v8",
      extension: [".tsx"],
      reporter: ['text', 'json-summary', 'json'],
    },
  },
});

Now, make some changes in your project and add corresponding tests to check if the workflow is working as expected.

Once you push your changes to the GitHub repo, open a PR against the main branch of your project. After the workflows finish running, you should see a comment showing the coverage result:

Coverage Report in a pull request comment

Step 4: Deploy the Project

As a last step in this tutorial, let's deploy the project on Vercel. You will set up an automatic deployment through Git that will trigger a redeploy each time new changes are pushed or merged into the main branch.

First, log in to your Vercel account, or create one if you don't already have one. Then, in your dashboard, click on "Add New Project" and click on the "Import" button next to your repository name in the "Import Git Repository" section.

If you don't see your repository listed, it may be due to your GitHub app permissions configuration. You can manage them in your settings section in your GitHub account.

Finally, choose a name for the project in the "Configure Project" section and click on the "Deploy" button. You can now see the deploy details by clicking on the "Deployment" link.

Vercel automatic deployments ensure that the deployed project is always updated with the latest changes. They also have the benefit of Preview Deployments, a preview URL that lets you test new features in advance of merging changes into production.

If you have followed along with the tutorial, with this step completed, you'll have completed the CD part of the CI/CD pipeline for your project. Now, you can be sure any code that is pushed to the main branch is linted and tested, and once all checks pass, it is automatically pushed to production.

Conclusion

In this guide, you learned about the importance of CI/CD in today’s software development ecosystem and its main benefits. You also took your first steps in this area by creating your own CI/CD pipeline for your project, learning how to use Husky and GitHub Actions.

Now, you can keep learning more about these tools and improve your CI/CD pipeline by customizing it to better fit your project's needs.

I hope you were able to gain some new knowledge and enjoyed following along. Thanks for reading!

But unfortunately, they often become targets for hackers and malicious individuals seeking to exploit any vulnerabilities to compromise data and disrupt functionality. As a result, you'll need to prioritize the security of your web server by updating it and implementing safeguards against threats.

To enhance the security of your web server, one effective approach is to use Continous Integration (CI). CI is a DevOps technique that allows the automated merging of code modifications from software engineers into a single repository. This practice enhances code quality, minimizes bugs and speeds up code delivery.

By using CI, you can automate the testing, building, and deployment processes for your web servers’ code and configuration. You can also ensure that your web server consistently maintains a stable state.

In this tutorial, I'll guide you through the process of strengthening the security of your web server by using two popular and powerful tools: NGINX and CircleCI.

NGINX, which is an open source web server, provides a range of features and modules that can greatly enhance the security of your web server. These include SSL/TLS encryption, security headers, and support for HTTP/2.

On the other hand, CircleCI offers both cloud-based and self-hosted options for Continous Integration (CI) and Continuous Delivery (CD), enabling seamless deployment processes.

By following this guide, you will learn how to:

• Configure NGINX to use SSL/TLS encryption and security headers

• Create a GitHub repository and push your NGINX configuration files to it

• Create a CircleCI project and link it to your GitHub repository

• Create a CircleCI configuration file and define your CI pipeline

• Test and deploy your web server with CircleCI

Here is what we'll cover:

• Prerequisites

• Step 1: Configure NGINX to Use SSL/TLS Encryption

• Step 2: Configure NGINX to Include Security Headers

• Step 3: Create a GitHub Repository and Push Your NGINX Configuration

• Step 4: Create a CircleCI Project and Link it to Your GitHub Repository

• Step 5: Create a CircleCI Configuration File and Define Your CI Pipeline

• Step 6: Test and Deploy Your Web Server with CircleCI

Let's get started!

Prerequisites

Before you start this guide, you need to ensure you have the following:

• A web server running NGINX. If you don't have one, you can follow this guide to install NGINX on Ubuntu 20.04. You can also use any other operating system or cloud provider that supports NGINX.

• A GitHub account. If you don't have one, you can sign up for free here.

• A CircleCI account. If you don't have one, you can sign up for free here. You will also need to link your GitHub account to your CircleCI account.

• Some basic knowledge of web development and Linux commands. You should be familiar with the concepts of web servers, SSL/TLS encryption, security headers, and CI. You should also be comfortable with using the command line and editing configuration files.

Once you have these, you are ready to proceed with the next steps.

Step 1: Configure NGINX to Use SSL/TLS Encryption

SSL/TLS encryption ensures the transmission of data between your web server and clients. It safeguards against interception or manipulation of information. It also plays a role in verifying the identity and reliability of your web server.

You need an SSL/TLS certificate to use SSL/TLS for your web server. An SSL/TLS certificate contains information about your web server, such as its domain name, owner, and public key. The validity of the certificate is verified through the unique digital signature from a Certificate Authority (CA).

You can either purchase an SSL/TLS certificate from a commercial CA, such as DigiCert, Symantec, or GlobalSign, or you can just get one for free from a non-profit CA, such as Let's Encrypt. You can also create your own self-signed certificate, but this is not recommended for production use, as it will not be trusted by most browsers and clients.

In this guide, you will use Let's Encrypt to get a free SSL/TLS certificate for your web server. To use Let's Encrypt, you need to install a software client on your web server that can communicate with the CA and perform the necessary tasks.

One of the most common and recommended clients for Let's Encrypt is Certbot. Certbot is a command-line tool that can automatically request, install, and renew certificates for your web server. It can also configure your web server to use the certificates and enable HTTPS.

To install Certbot on your web server, run the following commands:

sudo apt update
sudo apt install certbot python3-certbot-nginx

After installing Certbot, use it to request and install a certificate for your web server. You need to provide your domain name and your email address for the certificate.

To request and install a certificate for your web server, run the following command:

sudo certbot --nginx -d yourdomain.com

Replace yourdomain.com with your actual domain name.

Follow the prompts and answer the questions. Certbot will automatically verify your domain ownership, obtain a certificate, and install it on your web server. It will also ask you whether you want to redirect all HTTP traffic to HTTPS. Choose option 2 to enable the redirection.

After the process is completed, you will see a message like this:

You have now successfully configured NGINX to use SSL/TLS encryption with a certificate from Let's Encrypt. You can now access your web server using HTTPS and see the lock icon in your browser.

You can also test your web server security using online tools, such as SSL Labs. You should see a grade of A or higher.

Note: Let's Encrypt certificates are valid for 90 days. Certbot can automatically renew them for you before they expire.

To enable the automatic renewal, you need to create a CRON job or a systemd timer that runs the following command at least once per day:

sudo certbot renew

You can also test the renewal process manually by running the following command:

sudo certbot renew --dry-run

This will perform a trial run without making any changes.

If you encounter any errors or issues, you can check the Certbot documentation or the Let's Encrypt community forum for help.

Step 2: Configure NGINX to Include Security Headers

Security headers help instruct the browser to apply certain security policies or restrictions when handling your web content. They can prevent or mitigate common web attacks, such as cross-site scripting (XSS), clickjacking, content injection, and more.

In this step, you will be adding security headers to your NGINX configuration. These headers include X Frame Options, X Content Type Options, X XSS Protection, and Content Security Policy.

X-Frame-Options

The X-Frame-Options header tells the browser whether or not to allow your web page to be displayed in a frame, iframe, embed, or object element. This can help you prevent clickjacking attacks, where an attacker overlays a hidden frame on top of your web page and tricks the user into clicking on it.

There are three possible values for this header:

• DENY: This value prevents your web page from being displayed in any frame.

• SAMEORIGIN: This value allows your web page to be displayed in a frame only if the frame is from the same origin as your web page.

• ALLOW-FROM URI: This value allows your web page to be displayed in a frame only if the frame is from the specified URI.

To enable the X-Frame-Options header in NGINX, add the following line to your server block in your NGINX configuration file (/etc/nginx/sites-enabled/example.conf):

add_header X-Frame-Options "SAMEORIGIN";

This will allow your web page to be displayed in a frame only if the frame is from the same origin as your web page. You can change the value to DENY or ALLOW-FROM uri according to your needs.

Save the file and restart NGINX to apply the changes.

X-Content-Type-Options

The X Content Type Options header instructs the browser to perform MIME-type sniffing. This feature attempts to determine the content type of a resource by analyzing its content or file extension.

By using this header you can safeguard against content injection attacks. These attacks involve uploading a file with a content type to exploit the browser’s interpretation of it.

There is only one possible value for this header:

• nosniff: This value prevents the browser from performing MIME type sniffing.

To enable the X-Content-Type-Options header in NGINX, add the following line to your server block in your NGINX configuration file (/etc/nginx/sites-enabled/example.conf):

add_header X-Content-Type-Options "nosniff";

This will prevent the browser from performing MIME type sniffing on your web resources.

Save the file and restart NGINX to apply the changes.

X-XSS-Protection

The X-XSS-Protection header tells the browser to enable or disable its built-in XSS filter, which can detect and block some types of XSS attacks. This can help you prevent XSS attacks, where an attacker injects malicious code into your web page that executes in the browser.

There are three possible values for this header:

• 0: This value disables the XSS filter.

• 1: This value enables the XSS filter and sanitizes the page if an XSS attack is detected.

• 1; mode=block: This value enables the XSS filter and blocks the page if an XSS attack is detected.

To enable the X-XSS-Protection header in NGINX, add the following line to your server block in your NGINX configuration file (/etc/nginx/sites-enabled/example.conf):

add_header X-XSS-Protection "1; mode=block";

This will enable the XSS filter and block the page if an XSS attack is detected. You can change the value to 0 or 1 according to your needs.

Save the file and restart NGINX to apply the changes.

Content-Security-Policy

The Content-Security-Policy header tells the browser to enforce a set of rules that restrict what sources and types of content can be loaded and executed on your web page. This can help you prevent XSS, content injection, and other types of attacks that rely on loading malicious or untrusted content.

The value of this header is a complex policy that consists of multiple directives and values. Each directive specifies a type of content and a list of sources that are allowed or disallowed for that content.

For example, the following policy allows only scripts styles from the same origin, and images from the same origin or yourdomain.com:

Content-Security-Policy: default-src 'none'; script-src 'self'; style-src 'self'; img-src 'self' yourdomain.com;

The Content-Security-Policy header is very powerful and flexible, but also very complicated and error-prone. You need to carefully design and test your policy to ensure that it does not break your web functionality or introduce new vulnerabilities. You can use tools like CSP Evaluator or CSP Scanner to check and improve your policy.

To enable the Content-Security-Policy header in NGINX, add the following line to your server block in your NGINX configuration file (/etc/nginx/sites-enabled/example.conf):

add_header Content-Security-Policy "default-src 'none'; script-src 'self'; style-src 'self'; img-src 'self' yourdomain.com;";

This will enforce the policy that you described above. You can change the policy according to your needs.

Save the file and restart NGINX to apply the changes.

Step 3: Create a GitHub Repository and Push Your NGINX Configuration

To create a GitHub repository and push your NGINX configuration files to it, follow these steps:

Create a GitHub repository

First, log in to your GitHub account and go to the GitHub homepage.

Click on the plus icon in the top right corner of the page, and select "New repository" from the dropdown menu.

On the next page, enter a name for your repository in the "Repository name" field. This should be a short and descriptive name that accurately reflects the contents of the repository. For example, you can name it "nginx-config".

In the "Description" field, you can enter a longer description of the repository if you want. This is optional, but it can be helpful to provide more information about the purpose of the repository.

For example, you can write "A repository for storing and managing my NGINX configuration files".

You can set the visibility to whatever you prefer. If you want others to be able to see your work, set it to "Public". Otherwise, set it to "Private".

Leave the "Initialize this repository with a README" option unchecked, as you want to create an empty repository.

Click on the "Create repository" button to create the repository.

Your new empty repository will be created and you will be taken to the repository page.

Push your NGINX configuration files to the GitHub repository

On your web server, navigate to the directory where your NGINX configuration files are located. By default, this is /etc/nginx on most Linux distributions.

Initialize a new Git repository in this directory by running the following command:

git init

This will create a new .git directory in the current directory, which will be used to store all the version control information for your project.

Add all the configuration files that you want to include in the repository by running the following command:

git add .

This will add all the files in the current directory and its subdirectories to the repository. You can also specify individual files to be added by replacing the (.) with the file names, separated by spaces.

Then commit the files to the repository by running the following command:

git commit -m "Initial commit"

This will create the first commit in the repository, which will include all the files that were added in the previous step. The -m flag is used to specify a commit message, which should briefly describe the changes that were made in this commit.

Go back to your GitHub repository page and copy the URL of your repository. You can find it under the "Code" section. It should look something like this:

On your web server, add the URL of your GitHub repository as a remote for your Git repository by running the following command:

git remote add origin https://github.com/username/nginx-config.git

Replace username with your GitHub username and nginx-config with your repository name. The origin is the name of the remote, which you can change to anything you want.

Push your local Git repository to the GitHub repository by running the following command:

git push -u origin master

This will push your master branch, which is the default branch in Git, to the origin remote, which is the GitHub repository that you created. The -u flag is used to set the upstream for your branch, which means that you can use git push or git pull without specifying the remote or the branch in the future.

Enter your GitHub username and password when prompted. If you have enabled two-factor authentication, you will need to use a personal access token instead of your password. You can generate one from your GitHub settings page.

You have successfully created a GitHub repository and pushed your NGINX configuration files to it. You can now view and manage your configuration files on GitHub.

Step 4: Create a CircleCI Project and Link it to Your GitHub Repository

CircleCI is a platform that offers cloud-based and self-hosted options for continuous integration and delivery. It allows you to create and run pipelines that automate and streamline your web server deployment and update process.

To use CircleCI, you need to create a CircleCI project and link it to your GitHub repository. This will enable CircleCI to access your code and configuration files, and trigger builds whenever you push to GitHub.

To create a CircleCI project and link it to your GitHub repository, follow these steps:

Sign up for CircleCI and connect your GitHub account

Start by logging in to your CircleCI account or sign up for free here.

On the CircleCI dashboard, click on the "Create Project" button in the top right corner of the page.

On the next page, select "GitHub" as your version control provider and click on the "Connect with GitHub" button.

On the next page, authorize CircleCI to access your GitHub account by clicking on the "Authorize circleci" button.

On the next page, Enter a “Project Name” and follow the remaining instructions to successfully create a CircleCI project.

Create a CircleCI project and link it to your GitHub repository

Next, create a new SSH key pair in your terminal:

ssh-keygen -t ed25519 -f ~/.ssh/project_key -C email@example.com

Then copy the private key generated:

pbcopy < ~/.ssh/project_key

Next, enter it in the private key field:

You will see a list of your GitHub repositories. Find the repository that you created in the previous step and click on the "Create Project" button next to it.

Now you will see a list of templates for different languages and frameworks. Since you are using Python and Flask, select the "Python" template and click on the "Use this config" button.

On the next page, you will see the generated CircleCI configuration file (config.yml) that defines your pipeline. You can review and edit the file if you want, or leave it as it is for now. Click on the "Start building" button to create the project and link it to your GitHub repository.

Your new CircleCI project will be created and linked to your GitHub repository.

You have now successfully created a CircleCI project and linked it to your GitHub repository. You can now configure and run your pipeline on CircleCI.

Step 5: Create a CircleCI Configuration File and Define Your CI Pipeline

A CircleCI configuration file is a YAML file that defines your CI pipeline. A CI pipeline is a sequence of jobs that run whenever you push changes to your GitHub repository. Each job consists of steps that perform specific tasks, such as running commands, installing dependencies, or deploying your web server.

In this step, you will create a CircleCI configuration file and define your CI pipeline. You will use the Python template that you selected in the previous step as a starting point, and modify it to suit our needs. You will also explain what each step in the pipeline does and how it helps to automate and secure your web server deployment.

Create a CircleCI configuration file

On your web server, navigate to the directory where your NGINX configuration files are located. By default, this is /etc/nginx on most Linux distributions.

Create a new directory called .circleci in this directory by running the following command:

mkdir .circleci

This is where you will store our CircleCI configuration file.

Then create a new file called config.yml in the .circleci directory by running the following command:

touch .circleci/config.yml

This is your CircleCI configuration file.

Open the config.yml file with your preferred text editor and paste the following code:

version: 2.1

jobs:
  build:
    docker:
      - image: cimg/python:3.9
    steps:
      - checkout
      - run:
          name: Install dependencies
          command: |
            pip install -r requirements.txt
      - run:
          name: Run tests
          command: |
            pytest
  deploy:
    machine:
      image: ubuntu-2004:202101-01
    steps:
      - checkout
      - add_ssh_keys:
          fingerprints:
            - "YOUR_FINGERPRINT"
      - run:
          name: Deploy Nginx configuration
          command: |
            scp -r nginx root@YOUR_IP:/etc
            ssh root@YOUR_IP "systemctl restart nginx"

workflows:
  version: 2
  build-and-deploy:
    jobs:
      - build
      - deploy:
          requires:
            - build
          filters:
            branches:
              only: main

This is your CircleCI configuration file that defines your CI pipeline. You will explain each part of the file in the next section.

Finally, fave and close the file.

Define your CI pipeline

Let's go through each part of the config.yml file and see what it does.

• Line 1: This indicates the version of the CircleCI platform you are using. 2.1 is the most recent version.

• Line 3: The jobs level contains a collection of children, representing your jobs. You specify the names for these jobs, for example, build, test, deploy.

• Line 6: This is the Docker image. The example shows cimg/python:3.9, which is a CircleCI-provided image that contains Python 3.9 and other common tools.

• Line 9: The run directive executes a shell command or script. You can specify a name and a command for each run directive.

• Line 11: The command attribute is a list of shell commands that you want to execute. In this case, you are installing the dependencies for your web application using pip.

• Line 12: This is another run directive that runs the tests for your web application using pytest.

• Line 13: deploy is the second child in the jobs collection. This job is responsible for deploying your NGINX configuration to your web server.

• Line 14: This specifies that you are using a machine executor for this job. A machine executor provides a full virtual machine with root access and various tools installed.

• Line 15: This is the machine image. The example shows ubuntu-2004:202101-01, which is a CircleCI-provided image that contains Ubuntu 20.04 and other common tools.

• Line 16: The steps collection is a list of run directives and other commands that you want to execute in this job.

• Line 18: The add_ssh_keys command adds your SSH keys to the machine. You need to provide the fingerprints of the keys that you want to use. You can generate and add SSH keys from your CircleCI settings page.

• Line 21: The command attribute is a list of shell commands that you want to execute. In this case, you are using SCP to copy your NGINX configuration files from the machine to your web server, and SSH to restart the NGINX service on your web server. You need to replace YOUR_FINGERPRINT with the fingerprint of your SSH key, and YOUR_IP with the IP address of your web server.

• Line 24: This indicates the version of the workflow syntax you are using. 2 is the most recent version.

• Line 29: The required attribute specifies the dependencies of this job. In this case, you are saying that the deploy job requires the build job to finish successfully before running.

• Line 30: The filters attribute specifies the conditions for running this job. In this case, you are saying that the deploy job should only run on the main branch of our GitHub repository.

Push your CircleCI configuration file to your GitHub repository

On your web server, add, commit, and push your CircleCI configuration file to your GitHub repository by running the following commands:

git add .circleci/config.yml
git commit -m "Add CircleCI config file"
git push origin main

This will trigger a new build on CircleCI and run your CI pipeline.

Go to your CircleCI dashboard and click on the build-and-deploy workflow.

You can click on each job to see the details and logs of the steps.

Wait for the workflow to finish.

You have successfully created a CircleCI configuration file and defined your CI pipeline. You can now automate and secure your web server deployment with CircleCI. You can also modify and improve your configuration file according to your needs.

Step 6: Test and Deploy Your Web Server with CircleCI

Now that you have created a CircleCI project and a configuration file for your CI pipeline, you can test and deploy your web server with CircleCI. You can trigger and monitor your CI pipeline from the CircleCI web app or the command line. You can also verify that your web server is deployed and secured correctly by using online tools or by accessing your web server using HTTPS.

Trigger and monitor your CI pipeline from the CircleCI web app

To trigger and monitor your CI pipeline from the CircleCI web app, follow these steps:

• Go to the CircleCI dashboard.

• On the dashboard, you will see a list of your projects and pipelines. Find the project that you created in the previous step and click on it.

• On the project page, you will see a list of your branches and workflows. Find the branch that you pushed your CircleCI configuration file to in the previous step and click on the build-and-deploy workflow.

• On the workflow page, you will see a graphical representation of your pipeline, showing the status and duration of each job and step. You can click on each job or step to see the details and logs of the commands that were executed.

• Wait for the workflow to finish. If everything goes well, you will see a green check mark next to each job and step, indicating that they were successful.

You have successfully triggered and monitored your CI pipeline from the CircleCI web app. You can also trigger and monitor your CI pipeline from the command line.

Trigger and monitor your CI pipeline from the command line

To trigger and monitor your CI pipeline from the command line, follow these steps:

• On your web server, navigate to the directory where your NGINX configuration files are located. By default, this is /etc/nginx on most Linux distributions.

• Make some changes to your configuration files, such as adding or removing security headers, and save them.

• Add, commit, and push your changes to your GitHub repository by running the following commands:

git add .
git commit -m "Update Nginx configuration"
git push origin main

This will trigger a new build on CircleCI and run your CI pipeline.

To monitor your CI pipeline from the command line, you can use the CircleCI CLI, which is a tool that allows you to interact with CircleCI from your terminal. You can install the CircleCI CLI by following the instructions on the official website.

After installing the CircleCI CLI, you can use the circleci command to perform various actions, such as listing your projects, pipelines, workflows, jobs, and artifacts. You can also use the --help flag to see the available options and arguments for each command.

To monitor your CI pipeline from the command line, you can use the circleci pipeline command to list and describe your pipelines.

For example, you can run the following command to list the pipelines for your project:

circleci pipeline list --org-slug <VCS>/<your-vcs-org-or-username> --project-slug <VCS>/<your-repo-name>

Replace <VCS> with either gh or bb depending on your version control system. Replace <your-vcs-org-or-username> with your GitHub or Bitbucket organization or username. Replace <your-repo-name> with your repository name. You will see something like this:

You can use the pipeline ID or number to describe a specific pipeline and see its details, such as the status, workflows, jobs, and steps. For example, you can run the following command to describe the latest pipeline for your project:

circleci pipeline describe --org-slug <VCS>/<your-vcs-org-or-username> --project-slug <VCS>/<your-repo-name> --pipeline-number <number>

Replace <number> with the pipeline number that you want to describe.

Wait for the pipeline to finish. If everything goes well, you will see a success message for each job and step, indicating that they were successful.

You have successfully triggered and monitored your CI pipeline from the command line. You can also verify that your web server is deployed and secured correctly.

Verify that your web server is deployed and secured correctly

To verify that your web server is deployed and secured correctly, you can use online tools or access your web server using HTTPS. Here are some examples:

To verify that your web server is using the latest Nginx configuration that you pushed to GitHub, you can use a tool like curl or wget to make a request to your web server and inspect the response headers.

For example, you can run the following command to see the security headers that your web server is sending:

curl -I https://www.yourdomain.com

Replace yourdomain.com with your actual domain name.

You can compare the headers with the ones that you configured in your NGINX configuration file and see if they match.

To verify that your web server is using the SSL/TLS certificate that you installed with Certbot, you can use a tool like SSL Labs or HTBridge to scan your web server and check its SSL/TLS configuration and rating. You can check the details of your certificate, such as the issuer, validity, and chain. You can also check the grade of your SSL/TLS configuration, which should be A or higher.

To verify that your web server is accessible and functional using HTTPS, you can simply open your web browser and visit your web server using HTTPS. For example, you can go to https://www.yourdomain.com and see your web page.

You can check the lock icon in your browser, which indicates that your connection is secure. You can also click on the icon and see the details of your certificate and connection.

Conclusion

In this article, you have learned how to secure your web server using NGINX and CicleCI. NGINX and CircleCI, when used together, can provide a powerful solution for ensuring the continuous security of your web applications.

Stay ahead of the curve by integrating these technologies into your workflow, and empower your team to deliver secure and reliable web services.

Please don't forget to share this guide with your colleagues, friends, and online communities if you find it insightful.

We're going to review what this practice is about, how it compares to the previous approach in the software development industry, and finally see a practical example of how we can implement it in our projects.

Let's go!

Table of Contents

• Intro

• How CI/CD works

• The key benefits of CI/CD

• Tools for CI/CD

• How to set up a CI/CD pipeline with GitHub Actions

  • Initializing the project

  • What are GitHub Actions and how do they work?

  • Setting up our workflow

  • The magic

• Wrapping up

Intro

Continuous Integration and Continuous Delivery (CI/CD) is a software development approach that aims to improve the speed, efficiency, and reliability of software delivery. This approach involves frequent code integration, automated testing, and continuous deployment of software changes to production.

Before the adoption of CI/CD in the software development industry, the common approach was a traditional, waterfall model of software development.

In this approach, developers worked in silos, with each stage of the software development life cycle completed in sequence. The process typically involved gathering requirements, designing the software, coding, testing, and deployment.

The disadvantages of this traditional approach include:

1. Slow Release Cycles: Since each stage of the software development life cycle was completed in sequence, the release cycle was slow, which made it difficult to respond quickly to changing customer needs.

2. High Failure Rates: Software projects were prone to failure due to a lack of automated testing, which meant that developers had to rely on manual testing, leading to errors and bugs in the code.

3. Limited Collaboration: The traditional approach did not encourage collaboration between developers, testers, and other stakeholders, which made it difficult to identify and fix issues.

4. High Cost: The manual nature of software development meant that it was expensive, with high costs associated with testing, debugging, and fixing errors.

5. Limited Agility: Since the traditional approach was linear, it was not possible to make changes to the software quickly or respond to customer needs in real-time.

CI/CD emerged as a solution to these disadvantages, by introducing a more agile and collaborative approach to software development. CI/CD enables teams to work together, integrating their code changes frequently, and automating the testing and deployment process.

How CI/CD Works

CI/CD is an automated process that involves frequent code integration, automated testing, and continuous deployment of software changes to production.

Let's explain each step in a little more detail:

Code Integration

The first step in the CI/CD pipeline is code integration. In this step, developers commit their code changes to a remote repository (like GitHub, GitLab or BitBucket), where the code is integrated with the main codebase.

This step aims to ensure that the code changes are compatible with the rest of the codebase and do not break the build.

Automated Testing

Once the code is integrated, the next step is automated testing. Automated testing involves running a suite of tests to ensure that the code changes are functional, meet the expected quality standards, and are free of defects.

This step helps identify issues early in the development process, allowing developers to fix them quickly and efficiently.

If you're not familiar with the topic of testing, you can refer to this article I wrote a while ago.

Continuous Deployment

After the code changes pass the automated testing step, the next step is continuous deployment. In this step, the code changes are automatically deployed to a staging environment for further testing.

This step aims to ensure that the software is continuously updated with the latest code changes, delivering new features and functionality to users quickly and efficiently.

Production Deployment

The final step in the CI/CD pipeline is production deployment. In this step, the software changes are released to end-users. This step involves monitoring the production environment, ensuring that the software is running smoothly, and identifying and fixing any issues that arise.

The four steps of a CI/CD pipeline work together to ensure that software changes are tested, integrated, and deployed to production automatically. This automation helps to reduce errors, increase efficiency, and improve the overall quality of the software.

By adopting a CI/CD pipeline, development teams can achieve faster release cycles, reduce the risk of software defects, and improve the user experience.

Keep in mind that the pipeline stages might look different given the specific project or company we're talking about. Meaning, some teams might or might not use automated testing, some teams might or might not have a "staging" environment, and so on.

The key parts that make up the CI/CD practice are integration and deployment. This means that the code has to be continually integrated in a remote repository, and that this code has to be continually deployed to a given environment after each integration.

The Key Benefits of CI/CD

The key benefits of CI/CD include:

1. Faster Release Cycles: By automating the testing and deployment process, CI/CD enables teams to release software more frequently, responding quickly to customer needs.

2. Improved Quality: Automated testing ensures that software changes do not introduce new bugs or issues, improving the overall quality of the software.

3. Increased Collaboration: Frequent code integration and testing require developers to work closely together, leading to better collaboration and communication.

4. Reduced Risk: Continuous deployment allows developers to identify and fix issues quickly, reducing the risk of major failures and downtime.

5. Cost-Effective: CI/CD reduces the amount of manual work required to deploy software changes, saving time and reducing costs.

In summary, CI/CD emerged as a solution to the limitations of the traditional, linear approach to software development. By introducing a more agile and collaborative approach to software development, CI/CD enables teams to work together, release software more frequently, and respond quickly to customer needs.

Tools for CI/CD

There are several tools available for implementing CI/CD pipelines in software development. Each tool has its unique features, pros, and cons. Here are some of the most commonly used tools in CI/CD pipelines today:

Jenkins

Jenkins is an open-source automation server that is widely used in CI/CD pipelines. It is highly customizable and supports a wide range of plugins, making it suitable for various development environments. Some of its key features include:

Pros:

• Highly customizable with a wide range of plugins

• Supports integration with various tools and technologies

• Provides detailed reporting and analytics

Cons:

• Requires some technical expertise to set up and maintain

• Can be resource-intensive, especially for large projects

• Lack of a centralized dashboard for managing multiple projects

If you want to learn more about Jenkins, here's a full course for you.

Travis CI

Travis CI is a cloud-based CI/CD platform that provides automated testing and deployment for software projects. It supports several programming languages and frameworks, making it suitable for various development environments. Some of its key features include:

Pros:

• Easy to set up and use

• Cloud-based, so there's no need to set up and maintain infrastructure

• Supports a wide range of programming languages and frameworks

• Provides detailed reporting and analytics

Cons:

• Limited customization options

• Not suitable for large projects with complex requirements

• Limited support for on-premise installations

Here's a helpful tutorial about how to automate deployment on GitHub Pages with Travis CI.

GitHub actions

GitHub Actions is a powerful CI/CD tool that allows developers to automate workflows, run tests, and deploy code directly from their GitHub repositories.

Pros:

• Integrated with GitHub

• Easy to use

• Provides large ecosystem and good documentation

Cons:

• Limited build minutes

• Complex YAML syntax

Side comment: I mention GitHub actions here because it's a popular tool – but keep in mind that other online repository providers like GitLab and BitBucket also provide very similar options to GitHub actions.

Built-in CI/CD features by hosts

Popular hosts such as Vercel or Netlify have built-in in CI/CD features that allow you to link an online repository to a given site, and deploy to that site after a given event occurs in that repo.

Pros:

• Very simple to set up and use

Cons:

• Limited customization options

Each of these tools has its unique features, pros, and cons. The choice of tool will depend on the specific requirements of your project, your team's technical expertise, and your budget.

Here's a tutorial about how to deploy a front-end app with Netlify. And in this tutorial, you'll learn how to use Vercel to deploy a Next.js app.

How to Set Up a CI/CD Pipeline with GitHub Actions

Cool, so now that we have a clear idea of what CI/CD is, let's see how we can implement a simple example with an actual project using GitHub actions.

Initializing the Project

We'll start with a very basic React app built with Vite. You can do that by running yarn create vite in your console.

We'll focus on the CI/CD pipeline here, so there will be no complexity in the actual app code. But just to have an idea, the app.jsx component will have this code:

import './App.css';

function App() {

    return (
        <div className='App'>
            <h1>Vite + Reactooooo</h1>
        </div>
    );
}

export default App;

And then we'll have a test file that will check for that text to render:

import { describe, expect, it } from 'vitest';
import { render, screen } from './utils/test-utils/test-utils.jsx';

import App from 'src/App.jsx';

describe('App', async () => {
    it('should render while authenticating', () => {
        render(<App />);

        expect(screen.getByText('Vite + Reactooooo')).toBeInTheDocument();
    });
});

This test will run each time we run the yarn test command.

Next step should be to push our code to a GitHub repo. Then, let's talk a bit more about what GitHub actions are and how they work.

What are GitHub Actions and How Do They Work?

GitHub Actions is a CI/CD (Continuous Integration/Continuous Deployment) service provided by GitHub. It allows developers to automate workflows by defining custom scripts, known as "actions", that can be triggered by events such as pushes to a repository, pull requests, or issues.

Actions are defined in a YAML file, also known as a "workflow", which specifies the steps required to complete a task. GitHub Actions workflows can run on Linux, Windows, and macOS environments and support a wide range of programming languages and frameworks.

When an event triggers a GitHub Actions workflow, the service creates a fresh environment, installs dependencies, and runs the defined steps in the order specified. This can include tasks such as building, testing, packaging, and deploying code.

GitHub Actions also provides several built-in actions that can be used to simplify common tasks, such as checking out code, building and testing applications, publishing releases, and deploying to popular cloud providers like AWS, Azure, and Google Cloud.

GitHub Actions workflows can be run on a schedule, manually, or automatically when a specific event occurs, such as a pull request being opened or a new commit being pushed to a branch.

Setting Up Our Workflow

Great, so as we've seen, basically GitHub actions are a feature that allows us to define workflows four our projects. These workflows are nothing but a series of tasks or steps that will execute on GitHub's cloud after a given event we declare.

The way GitHub reads and executes these workflows is by automatically reading files within the .github/workflows directory in the root of our project. These workflow files should have the .yaml extension and use the YAML syntax.

To create a new workflow we just have to create a new YAML file within that directory. We'll call ours prod.yaml since we'll use it to deploy the production branch of our project.

Keep in mind a single project can have many different workflows that run different tasks on different occasions. For example, we could have a workflow for dev and staging branches as well, as those environments could require different tasks to execute and will probably deploy on different sites.

After creating this file, let's drop the following code in it:

# Name of our workflow
name: Production deploy

# Trigger the workflow on push to the main branch
on:
  push:
    branches:
      - main

# List of jobs
# A "job" is a set of steps that are executed on the same runner
jobs:
  # Name of the job
  test-and-deploy-to-netlify:
    # Operating system to run on
    runs-on: ubuntu-latest

    # List of steps that make up the job
    steps:
    # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
    - name: Checkout code
      uses: actions/checkout@v2

    # Setup Node.js environment
    - name: Use Node.js 16.x
      uses: actions/setup-node@v2
      with:
        node-version: '16.x'

    # Install dependencies
    - name: Install dependencies
      run: yarn install

    # Run tests
    - name: Run tests
      run: yarn test

    # Deploy to Netlify
    - name: Netlify Deploy
      uses: jsmrcaga/action-netlify-deploy@v2.0.0
      with:
        # Auth token to use with netlify
        NETLIFY_AUTH_TOKEN: ${{ secrets.NETLIFY_AUTH_TOKEN }}
        # Your Netlify site id
        NETLIFY_SITE_ID:  ${{ secrets.NETLIFY_SITE_ID }}
        # Directory where built files are stored
        build_directory: './dist'
        # Command to install dependencies
        install_command: yarn install
        # Command to build static website
        build_command: yarn build

So our workflow has the following tasks declared:

1. "Checkout code" step that checks out the latest commit on the current branch.

2. "Use Node.js 16.x" step that sets up the Node.js environment to version 16.x.

3. "Install dependencies" step that installs the project dependencies using the Yarn package manager.

4. "Run tests" step that runs the project's tests using the Yarn package manager.

5. "Netlify Deploy" step that deploys the project to Netlify using the jsmrcaga/action-netlify-deploy action. This step uses the Netlify authentication token and site ID secrets stored in the GitHub repository's secrets. The build directory, install command, and build command are also specified.

You probably noticed the first and last steps have the keyword uses. This keyword allows you to use actions or workflows developed by other GitHub users, and it's one of the best features of GitHub actions.

What's great is that by using this third party actions we can execute complex tasks such as deploying to an external host or building complex cloud infrastructure without the need to write every single line of necessary code.

As these tasks tend to be repetitive and frequently executed in many projects, we can just use a workflow developed by an official company account (such as Azure or AWS for example) or an independent open-source developer. Think about it as using a third party library. It's the same idea but taken to CI/CD workflows. Very convenient.

Another important thing to mention here is that in GitHub actions workflows tasks run sequentially, one after the other. And if a given task fails or throws and error, the next one won't execute. This is important because if we have a problem when installing our dependencies or a test fails, we don't want that code to be deployed.

Before we can push this code and see how the magic works, we first need to create a site on Netlify and get the NETLIFY_AUTH_TOKEN and NETLIFY_SITE_ID. This is quite straight forward even if you don't have previous experience with Netlify, so give it a try and Google a bit if you can't figure it out. ;)

Once you have these two tokens, you need to declare them as repository secrets in your GitHub repo. You can do this from the "settings" tab:

Configure both Netlify secret tokens in your repo

With this in place, now our prod.yaml file will be able to read these two tokens and execute the Netlify deploy action.

The Magic

Now that we have everything in place, let's push our code and see how it goes.

After pushing, if we go to the "actions" tabs of our repo, on the left we'll see a list of all the workflows we have in our repo. And on the right we'll see a list of each execution of the selected workflow. Since our workflow executes after each push, we should see a new execution each time we push.

A workflow execution

When the execution has a yellow light to the left of it, it means it's still running (executing tasks). If it has a green light it means it finished executing successfully and if the light is red, you know something went wrong, haha...

After clicking on the execution we can see a list of the workflow's jobs (we only had a single one).

The workflow's jobs

And after clicking on the job we can see a list of the job's tasks.

The tasks of the job

Each task is expansible and within it we can see logs corresponding to that task's execution. This is quite useful for debugging purposes. ;)

Now if we go to our previously set up Netlify site, we should see our app up and running!

And now that we have our CI/CD pipeline in place, we can deploy our app after each push to the main branch, all without lifting another finger. =D

Wrapping Up

CI/CD is a software development approach that provides several benefits to software development teams, including faster time-to-market, improved quality, increased collaboration, reduced risk, and cost-effectiveness.

By automating the software delivery pipeline, teams can quickly deploy new features and bug fixes, while reducing the risk of major failures and downtime.

With the availability of several CI/CD tools, it has become easier for teams to implement this approach and improve their software delivery process.

Well everyone, as always, I hope you enjoyed the article and learned something new.

If you want, you can also follow me on LinkedIn or Twitter. See you in the next one!

But I recently discovered a way to automate the deployment process for Next.js apps using AWS CodeDeploy and CodePipeline. It made my life so much easier, and I'm excited to share it with you.

In this tutorial, I'll guide you through the process of setting up auto-deployment for your Next.js app using the AWS services CodePipeline and CodeDeploy. By the end of it, you'll be able to save a lot of time by deploying your app automatically every time you push the code.

Let's get started!

Table of Contents:

1. Prerequisites
2. How to Deploy the Next.js App to AWS EC2
3. How to Run the Next.js App in Production Mode
4. How to Run a Next.js App Forever When the Console is Closed
5. What is CodeDeploy?
6. How to Setup Auto-Deployment using CodePipeline and CodeDeploy
7. How to Attach the IAM Role to EC2
8. How to Create the CodePipeline
9. Conclusion

Prerequisites

1. EC2 machine running Ubuntu
2. Very basic knowledge of EC2 and IAM AWS Services

How to Deploy the Next.js App to AWS EC2

To start simple, let's manually deploy the sample Next.js boilerplate app "hello-world" to EC2. The steps are almost the same for all Next.js applications.

Login to EC2

Login to the EC2 machine which you've created using the below command:

ssh -i /path/key-pair-name.pem instance-user-name@instance-IP-address

When you try to log into EC2, this is the common error that most people will encounter (I got it, too):

Permissions 0664 for .pem file is too open error

This error describes that the .pem file should be read-protected. Only the root user should be able to read it. So, you have to set the file permission to 400. Run the following command to achieve that:

chmod 400 key-pair-name.pem

EC2 by default comes with no software installed. Once you've logged into EC2, install NodeJS. There's an excellent article published by Digital Ocean and I use it every time I have to install Node on the server.

I have uploaded the boilerplate repo to Github. You can clone the repo by running the following command:

git clone https://github.com/5minslearn/deploy_nextjs_app.git

Navigate to the project and install the dependencies by running the below commands.

A quick note here. I'm a big fan of yarn for its lightning-fast dependency management. But I see most people use npm to manage their dependencies. If you like to use npm, you can replace yarn install with npm install in the below commands.

If you like to go with yarn, install yarn by following this tutorial first.

cd deploy_nextjs_app
yarn install

Let's run the application:

yarn dev

Hit "http://ec2-public-ip-address:3000/" on your browser and you should be able to see the following page:

Next.js Hello World App

There's another common issue that most people face here which we'll look at next.

How to fix the timeout error (EC2)

"Oh, My God! My site is loading for a long time and finally, it's throwing a timeout error. What could be the issue? Where did I made mistake?"

If this happens to you, then you can follow the below steps to fix it.

This issue basically occurs if your server does not expose port 3000. Remember, by default Next apps will be running on port 3000. But, you have to allow port 3000 from the Security Group of your EC2 console to access from your browser.

Login to your AWS console, select your EC2 instance, and then select the Security Group option. Click on the "Edit inbound rules" button. Add port 3000 to the list as shown in the below screenshot. Then hit the "Save rules" button.

Adding port 3000 to a security group

Visit the link "http://ec2-public-ip-address:3000/", and you'll be amazed to see that your page loads like magic.

So far, we've just run our app in development mode and verified that it's working.

How to Run the Next.js App in Production Mode

To deploy the app in Production Mode, you have to build your app first. Run yarn build to build the app and yarn start to start the app in production mode.

yarn build
yarn start

Hit "http://ec2-public-ip-address:3000/" again and this time you'll see that your app loads faster than before.

Apps running in Production mode will always be faster when compared to the ones running in Development mode. This is because Production apps will be optimized for performance.

How to Run a Next.js App Forever When the Console is Closed

So, you have your app running now. But you might notice that it's blocking you from closing your terminal and exiting from server connection. If you do so, your site will be down. That's where PM2 comes to play.

Basically, PM2 is a process manager that helps keep Node applications alive all the time. It runs in the background managing Node applications for you.

Install PM2 using the following command:

sudo yarn global add pm2

After PM2 installation, run the below command to run and manage your app in the background:

pm2 start yarn --name [name-of-your-app] -- start -p [port-number]

Replace [name-of-your-app] with your app name and [port-number] with 3000. Here's an example command,

pm2 start yarn --name next_hello_world_app -- start -p 3000

Hit "http://ec2-public-ip-address:3000/" and you'll again be amazed to see your app up and running.

It's always a best practice to save the PM2 process. When you reboot your instance, your PM2 instances will be lost. In order to restore it to its old state, you have to save the PM2 process. Here's the command for that:

pm2 save

Here's the command to restore your PM2 instances on reboot (don't execute this now, we'll come back to this shortly):

pm2 resurrect

We have successfully deployed the Next.js app manually. But remember, every time you make a code change and want to see the changes on your site, you have to login into EC2, pull the latest changes, build the app, and restart the app.

This will consume a lot of time and I'm too lazy to do it. So let's automate this in the next step!

Before setting up automatic deployment you have to know how CodeDeploy works.

What is CodeDeploy?

CodeDeploy lets you deploy your application automatically to any number of EC2 instances. We need to prepare two items before beginning this process:

1. CodeDeploy Agent must be installed in the EC2 instance. We use this to continuously poll CodeDeploy and deploy if any new changes are available.
2. A file called appspec.yml must be present in the root folder. This file describes the steps to be followed for the deployment.

There is awesome documentation by AWS to help you install CodeDeploy Agent. Please follow each and every step to install CodeDeploy Agent on your EC2 machine.

To verify that CodeDeploy agent is installed, run the below command. If you see active (running), Kudos to you! CodeDeploy was installed successfully.

sudo service codedeploy-agent status

CodeDeploy Agent running status

Now let's create the appspec.yml file. I've written the deployment instructions in the deploy.sh file. It's enough to run this file in the appspec.yml file. If you want to learn more about appspec.yml, check out the AWS official documentation.

Create a file called appspec.yml and add the following contents:

version: 0.0
os: linux
hooks:
  ApplicationStart:
    - location: deploy.sh
      timeout: 300
      runas: ubuntu

I hope you understand the instructions in the above file. If not, here's a super simple explanation. I'm advising the CodeDeploy Agent that I'm running a Linux OS in my instance and instructing it to run the deploy.sh file as ubuntu user with the timeout set to 300 seconds.

Here's my deploy.sh file:

#!/bin/bash
cd /path/to/project/on/EC2 
git pull origin master
yarn install &&
yarn build &&
pm2 restart [name]

This file contains instructions to navigate to the project folder on EC2, pull the latest code from source control, install dependencies, build the project, and restart the project instance.

This file is already available in the repo. No action for you here. Now it's time to set up automatic deployment.

How to Setup Auto-Deployment using CodePipeline and CodeDeploy

Two IAM roles have to be created to set up auto-deployment. Some complications will begin from here. To make things simple, I've attached screenshots with the appropriate items highlighted with red boxes.

Create an IAM Role for CodeDeploy

You have to create this role to deploy the code every time you push.

Navigate to IAM in the AWS Console by searching for "IAM" in the search bar at the top. Click Roles on the left pane and Click the "Create role" button at the top right.

Create IAM role

Choose AWS service in Trusted entity types and choose CodeDeploy in the Use cases section and proceed to the next step.

IAM role for CodeDeploy

Now, you can see that the AWSCodeDeployRole policy is the only policy available, and it'll be chosen by default in this (Permissions) step. Let's proceed to the next section. No action for you here.

AWSCodeDeploy Permission

Enter a name for your IAM role. You should choose a meaningful name to identify this in the future. I'm calling it service-role-for-code-deploy. Review the permission in the JSON and click the Create role button at the bottom.

AWSCodeDeploy Permission Review

Create an IAM role for EC2

Let's create the next role. This role is for EC2. Choose AWS service in the Trusted entity type, EC2 in the Common use cases section, and choose CodeDeploy in Use cases for other AWS services. Click Next to proceed to the next section.

IAM role for EC2

There are a lot of policies available for EC2 and CodeDeploy. In the Add permissions section, search for codedeploy (No space between code and deploy) and select "AmazonEC2RoleForCodeDeploy" and proceed to the next step.

Adding AmazonEC2RoleForCodeDeploy permission

No change in this step. Review and give a meaningful name (I'm naming it as "code-deploy-role-for-ec2") for your role and click "Create role" button.

How to Attach the IAM Role to EC2

Once the IAM role for EC2 is created, we have to attach it to the EC2 instance.

EC2 instance before attaching the IAM role

To attach the IAM role to the EC2 instance, open your EC2 instance, click on the "Actions" button on the top right, and select "Security" in the drop-down. Then select "Modify IAM role".

Modify IAM role for EC2 instance

Select the IAM role which you created last (code-deploy-role-for-ec2) and click the "Update IAM role" button. Reboot the EC2 for the changes to take effect.

Update IAM role for EC2 instance

EC2 instance after attaching IAM role

After rebooting the EC2, login to EC2 with SSH and run the pm2 resurrect command to restore the PM2 processes. Failing to do this may land you at "PM2 Process or Namespace not found error" while running automatic deployment.

PM2 process restore

PM2 process or namespace not found an error

How to Create the CodeDeploy Application

In the AWS Console, search "CodeDeploy" in the search bar at the top. Select "Applications" in the left pane. Click on the "Create application" button on the top right.

Navigate to CodeDeploy in AWS Console

Enter the Application name, choose the "EC2/On-premises" compute platform, and click the "Create application" button.

Create CodeDeploy application

Once it's done, you'll automatically be redirected to the Deployment groups section. We have to create a deployment group. Click on the "Create deployment group" button.

Create CodeDeploy Deployment group

Enter the deployment group name, select the service role (1st created role) you created, and select the deployment type as in-place:

Create CodeDeploy Deployment group

In the Environment configuration section, select "Amazon EC2 instances" and select the key as Name. Enter your EC2 instance name in the value.

Code Deployment Group Environment configuration

In the Agent configuration section, select Never, as we installed CodeDeployAgent already. Select "CodeDeployDefault.AllAtOnce" in the Deployment settings section. Leave the "Enable load balancing" checkbox unchecked. Finally, click the "Create a deployment group" button.

CodeDeploy deployment group configurations

How to Create the CodePipeline

AWS CodePipeline helps you to automate your release pipelines for fast and reliable application and infrastructure updates. Now it's time to create our CodePipeline. In the AWS Console, search for "CodePipeline" in the search bar.

Select "Pipelines" in the left pane and click on "Create pipeline" button.

Create CodePipeline

Enter the Pipeline name, and Role name. Remember, we created roles for EC2 and CodeDeploy, but not for CodePipeline. AWS by default creates it from here.

CodePipeline settings

Add Source Stage

In this step, we have to connect our repo with CodePipeline to deploy the changes immediately after the code is pushed.

We'll be using GitHub as our source. Choose GitHub (version 2) in the source provider, and click on the "Connect to GitHub" button. This will open up a new pop-up window. Click on the "Connect to GitHub" button.

CodePipeline adding source stage

This will take you to the GitHub authorization page where you have to sign into your GitHub account. Click on the "Install a new app" button.

CodePipeline Github Authorization

Choose "Only select repositories" and choose your repository below that.

Installing AWS connector for GitHub

Once installed, it will prompt you for the password. Click the "Connect" button once you're done with your authentication.

Connecting GitHub to AWS

After connecting to GitHub, select the Repository name and branch name. To start the CodePipline on code change, it's important to select the check box "Start the pipeline on source code change" – otherwise auto deployment will not happen. For "Output and artifact format", select "CodePipeline default" and click the "Next" button.

CodePipeline - Select source code repo

The next step is to add the build stage, but since we're deploying a simple app we don't need a build stage. Enterprise companies prefer to build their app using AWS CodeBuild service. Let's keep things simple and simply skip the build stage.

If you want me to write about CodeBuild let me know, I will try to cover it in my upcoming tutorials.

Add Deploy Stage

In the deployment stage step, choose "AWS CodeDeploy" for the "Deploy provider" and select the region where you created the above CodeDeploy application. Then select the "Application name" and "Deployment group" that we created in the previous steps and click the "Next" button.

CodePipleine - Adding Deployment stage

The last step is "Review". Review everything carefully and click on the "Create pipeline" button. Once the pipeline is created it will start the deployment process. If you followed all the above steps, the pipeline should read "Succeeded" on your very first build.

Pipeline Succeeded

How to Verify Auto-Deployment

Now let's verify if the Auto-deployment works properly. This is the Home page of our project:

Next.js Hello World App

Let's change the text from "Hello World" to "Welcome to 5minslearn" and push the code to GitHub.

Git code diff

Here we go! The CodePipeline has triggered automatically and the changes were successfully deployed.

CodeDeploy getting triggered automatically on code changes in Git

Now head to "http://ec2-public-ip-address:3000/", you will see the below page:

Next.js App after auto deployment

Congrats! 🎉 We successfully completed setting up auto-deployment for a Next.js app.

Conclusion

In this article, we learned how to deploy Next.js manually on EC2 and set up auto-deployment using AWS services such as CodeDeploy and CodePipeline.

Hope you enjoyed reading this article! If you are stuck at any point feel free to drop your queries to my email. I'll be happy to help you!

If you wish to learn more about AWS, subscribe to my newsletter (https://5minslearn.gogosoon.com/) and follow me on social media.

As you continue to develop your software, you must also continue to integrate it with previous code and deploy it to servers.

Manually doing this is a time-consuming process that can occasionally result in errors. So we need to do this in a continuous and automated manner – which is what you will learn in this article.

We'll go over how you can improve your MERN (MongoDB, Express, React, and NodeJs) app development process by setting up a CI/CD pipeline with Jenkins. You'll see how to automate deployment for faster, more efficient releases.

Let's Get Started

Prerequisites

• Basic understanding of MERN stack technologies.
• Basic understanding of Docker.
• Get source code from GitHub

The Problem

Consider this productivity app – it's a MERN project that we are going to use in this article. There are numerous steps we must complete, from building the application to pushing it to the Docker hub.

First, we must run tests with a command to determine whether all tests pass or not. If all tests pass, we build the Docker images and then push those images to Docker Hub. If your application is extremely complex, you may need to take additional steps.

Now, imagine that we're doing everything manually, which takes time and can lead to mistakes.

Waiting for deployment without devops meme

The Solution

To address this problem, we can create a CI/CD Pipeline. So, whenever you add a feature or fix a bug, this pipeline gets triggered. This automatically performs all of the steps from testing to deploying.

What is CI/CD and Why is it Important?

Continuous Integration and Continuous Deployment is a series of steps performed to automate software integration and deployment. CI/CD is the heart of DevOps.

CI/CD steps

From development to deployment, our MERN app goes through four major stages: testing, building Docker images, pushing to a registry, and deploying to a cloud provider. All of this is done manually by running various commands. And we need to do this every time a new feature is added or a bug is fixed.

But this will significantly reduce developer productivity, which is why CI/CD can be so helpful in automating this process. In this article, we will cover the steps up until pushing to the registry.

CI/CD meme

The Project

The project we are going to use in this tutorial is a very simple full-stack MERN application.

Project demo

It contains two microservices.

1. Frontend
2. Backend

You can learn more about the project here.

Both of these applications contains a Dockerfile. You can learn how to dockerize a MERN application here.

What is Jenkins?

To run a CI/CD pipeline, we need a CI/CD server. This is where all of the steps written in a pipeline run.

There are numerous services available on the market, including GitHub Actions, Travis CI, Circle CI, GitLab CI/CD, AWS CodePipeline, Azure DevOps, and Google Cloud Build. Jenkins is one of the popular CI/CD tools, and it's what we'll use here.

How to Set Up Jenkins Server on Azure

Because Jenkins is open source and it doesn't provide a cloud solution, we must either run it locally or self-host on a cloud provider. Now, running locally can be difficult, particularly for Windows users. As a result, I've chosen to self-host it on Azure for this demo.

If you want to run locally or self-host somewhere other than Azure (follow these guides by Jenkins), skip this section and proceed to the How to Configure Jenkins section.

First, you'll need to sign in to your Azure account (Create one if you don't have one already). Open Azure Cloud Shell.

Opening Azure Cloud Shell

Then create a directory called jenkins to store all the Jenkins config, and switch to that directory:

mkdir jenkins
cd jenkins

Create a file called cloud-init-jenkins.txt. Open with nano or vim,

touch cloud-init-jenkins.txt
nano cloud-init-jenkins.txt

and paste this code into it:

#cloud-config
package_upgrade: true
runcmd:
  - sudo apt install openjdk-11-jre -y
  - wget -qO - https://pkg.jenkins.io/debian-stable/jenkins.io.key | sudo apt-key add -
  - sh -c 'echo deb https://pkg.jenkins.io/debian-stable binary/ > /etc/apt/sources.list.d/jenkins.list'
  - sudo apt-get update && sudo apt-get install jenkins -y
  - sudo service jenkins restart

Here, we'll use this file to install Jenkins after creating a virtual machine. First, we install openjdk, which is required for Jenkins to function. The Jenkins service is then restarted after we install it.

Next, create a resource group. (A resource group in Azure is like a container that holds all the related resources of a project in one group. Learn more about resource groups here.)

az group create --name jenkins-rg --location centralindia

Note: make sure to change the location to the one closest to you.

Now, create a virtual machine.

az vm create \
--resource-group jenkins-rg \
--name jenkins-vm \
--image UbuntuLTS \
--admin-username "azureuser" \
--generate-ssh-keys \
--public-ip-sku Standard \
--custom-data cloud-init-jenkins.txt

You can verify the VM installation with this command:

az vm list -d -o table --query "[?name=='jenkins-vm']"

Don't be confused. This command simply displays JSON data in a tabular format for easy verification.

Jenkins server runs on port 8080, so we need to expose this port on our VM. You can do that like this:

az vm open-port \
--resource-group jenkins-rg \
--name jenkins-vm  \
--port 8080 --priority 1010

Now we can access the Jenkins dashboard in the browser with the URL http://<your-vm-ip>:8080. Use this command to get the VM IP address:

az vm show \
--resource-group jenkins-rg \
--name jenkins-vm -d \
--query [publicIps] \
--output tsv

You can now see the Jenkins application in your browser.

Jenkins dashboard

As you'll notice, Jenkins is asking us to provide an admin password which is automatically generated during its installation.

But first let's SSH into our virtual machine where Jenkins is installed.

ssh azureuser@<ip_address>

Now, type in the below command to get the password:

sudo cat /var/lib/jenkins/secrets/initialAdminPassword

Copy and paste it. Then click Continue.

How to Configure Jenkins

First, you'll need to click Install suggested plugins. It will take some time to install all the plugins.

Installing suggested plugins

An admin user is needed to restrict access to Jenkins. So go ahead and create one. After finishing, click Save and continue.

Create an admin user

Now you will be presented with the Jenkins dashboard.

The first step is to install the "Blue Ocean" plugin. Jenkins has a very old interface, which may make it difficult for some people to use. This blue ocean plugin provides a modern interface for some Jenkins components (like creating a pipeline).

To install plugins, go to Manage Jenkins -> click Manage Plugins under "System Configuration" -> Available plugins. Search for "Blue Ocean" -> check the box and click Download now and install after restart.

Blue ocean

Great, we're all set. Now let's create a pipeline.

How to Write a Jenkinsfile

To create a pipeline, we need a Jenkinsfile. This file contains all the pipeline configurations – stages, steps, and so on. Jenkinsfile is to Jenkins as a Dockerfile is to Docker.

Jenkinsfile uses the Groovy syntax. The syntax is very simple. You can understand everything by just looking at it.

Let's start by writing:

pipeline {

}

The word 'agent' should be the first thing you mention in the pipeline. An agent is similar to a container or environment in which jobs run. You can use multiple agents to run jobs in parallel. You can find more information about Jenkins agents can here.

pipeline {
    agent any
}

Here we are telling Jenkins to use any available agent.

We have a total of 5 stages in our pipeline:

CI/CD pipeline stages

Stage 1: Checkout code

Different CI/CD tools use different naming conventions. In Jenkins, these are referred to as stages. In each stage we write various steps.

Our first stage is checking out code from a source code management system (in our case, GitHub).

pipeline {
    agent any

    stages {
        stage('Checkout') {
            steps {
                checkout scm
            }
        }
    }
}

Commit the changes and push to your GitHub repo.

Since we haven't created any pipelines yet, let's do that now.

Before we begin, we must ensure that Git is installed on our system. If you followed my previous steps to install Jenkins on an Azure VM, Git is already installed.

You can test it by running the following command (make you are still SSHed into the VM):

git --version

If it isn't already installed, you can do so with:

sudo apt install git

Open blue ocean. Click Create new pipeline.

Creating new pipeline

Then select your source code management system. If you chose GitHub, you must provide an access token for Jenkins to access your repository. I recommend clicking on Create an access token here because it is a template with all of the necessary permissions. Then click Connect.

Selecting scm

After that, a pipeline will be created. Since our repository already contains a Jenkinsfile, Jenkins automatically detects it and runs the stages and steps we mentioned in the pipeline.

If everything went well, the entire page will turn green. (Other colors: blue indicates that the pipeline is running, red indicates that something went wrong in the pipeline, and gray indicates that we stopped the pipeline.)

Stage one successful

Stage 2: Run frontend tests

In general, all the CI/CD pipelines contains some tests that needs to be run before deploying. So I added simple tests to both the frontend and backend. Let's start with the frontend tests.

stage('Client Tests') {
    steps {
        dir('client') {
            sh 'npm install'
            sh 'npm test'
        }
    }
}

We're changing the directory to client/ because that's where the frontend code is. And then install the dependencies with npm install and run the tests with npm test in a shell.

Again, before we restart the pipeline, we have to make sure node and npm are installed or not. Install node and npm with these commands in the virtual machine:

curl -sL https://deb.nodesource.com/setup_16.x | sudo -E bash -

After that, run the following:

sudo apt-get install -y nodejs

Now, commit the code and restart the pipeline.

Run client tests

Stage 3: Run backend tests

Now do the same thing for the backend tests.

But there is one thing we need to do before we proceed. If you take a look at the codebase and activity.test.js, we are using a few environment variables. So let's add these environment varibales in Jenkins.

How to add environment variables in Jenkins

To add environment variables, go to Manage Jenkins -> click Manage Credentials under "Security" -> System -> Global credentials (unrestricted) -> click + Add Credentials.

For Kind select "Secret text", leave Scope default, and for Secret write the secret value and ID. This is what we use when using these environment variables in the Jenkinsfile.

Add the following env variables:

Environment variables

Then in the Jenkinsfile, use these env variables:

environment {
    MONGODB_URI = credentials('mongodb-uri')
    TOKEN_KEY = credentials('token-key')
    EMAIL = credentials('email')
    PASSWORD = credentials('password')
}

Add a stage to install dependencies, set these variables in the Jenkins environment, and run the tests:

stage('Server Tests') {
    steps {
        dir('server') {
            sh 'npm install'
            sh 'export MONGODB_URI=$MONGODB_URI'
            sh 'export TOKEN_KEY=$TOKEN_KEY'
            sh 'export EMAIL=$EMAIL'
            sh 'export PASSWORD=$PASSWORD'
            sh 'npm test'
        }
    }
}

Again, commit the code and restart the pipeline.

Run server tests

Stage 4: Build Docker images

Now, we have to specify a step to build the Docker images from the Dockerfiles.

Before we proceed, install Docker in the VM (if you don't already have it installed).

To install Docker:

sudo apt install docker.io

Add the user jenkins to the docker group so that Jenkins can access the Docker daemon – otherwise you'll get a permission denied error.

sudo usermod -a -G docker jenkins

Then restart the jenkins service.

sudo systemctl restart jenkins

Add a stage in the Jenkinsfile.

stage('Build Images') {
    steps {
        sh 'docker build -t rakeshpotnuru/productivity-app:client-latest client'
        sh 'docker build -t rakeshpotnuru/productivity-app:server-latest server'
    }
}

Commit the code and restart the pipeline.

Build docker images

Stage 5: Push images to the registry

As a final stage, we will push the images to Docker hub.

Before that, add your docker hub username and password to the Jenkins credentials manager, but for Kind choose "Username with password".

Username with password type credential

Add the final stage where we login and push images to Docker hub.

stage('Push Images to DockerHub') {
    steps {
        withCredentials([usernamePassword(credentialsId: 'dockerhub', passwordVariable: 'DOCKER_PASSWORD', usernameVariable: 'DOCKER_USERNAME')]) {
            sh 'docker login -u $DOCKER_USERNAME -p $DOCKER_PASSWORD'
            sh 'docker push rakeshpotnuru/productivity-app:client-latest'
            sh 'docker push rakeshpotnuru/productivity-app:server-latest'
        }
    }
}

Push images to dockerhub

Here is the complete Jenkinsfile:

// This is a Jenkinsfile. It is a script that Jenkins will run when a build is triggered.
pipeline {
    // Telling Jenkins to run the pipeline on any available agent.
    agent any

    // Setting environment variables for the build.
    environment {
        MONGODB_URI = credentials('mongodb-uri')
        TOKEN_KEY = credentials('token-key')
        EMAIL = credentials('email')
        PASSWORD = credentials('password')
    }

    // This is the pipeline. It is a series of stages that Jenkins will run.
    stages {
        // This state is telling Jenkins to checkout the source code from the source control management system.
        stage('Checkout') {
            steps {
                checkout scm
            }
        }

        // This stage is telling Jenkins to run the tests in the client directory.
        stage('Client Tests') {
            steps {
                dir('client') {
                    sh 'npm install'
                    sh 'npm test'
                }
            }
        }

        // This stage is telling Jenkins to run the tests in the server directory.
        stage('Server Tests') {
            steps {
                dir('server') {
                    sh 'npm install'
                    sh 'export MONGODB_URI=$MONGODB_URI'
                    sh 'export TOKEN_KEY=$TOKEN_KEY'
                    sh 'export EMAIL=$EMAIL'
                    sh 'export PASSWORD=$PASSWORD'
                    sh 'npm test'
                }
            }
        }

        // This stage is telling Jenkins to build the images for the client and server.
        stage('Build Images') {
            steps {
                sh 'docker build -t rakeshpotnuru/productivity-app:client-latest client'
                sh 'docker build -t rakeshpotnuru/productivity-app:server-latest server'
            }
        }

        // This stage is telling Jenkins to push the images to DockerHub.
        stage('Push Images to DockerHub') {
            steps {
                withCredentials([usernamePassword(credentialsId: 'dockerhub', passwordVariable: 'DOCKER_PASSWORD', usernameVariable: 'DOCKER_USERNAME')]) {
                    sh 'docker login -u $DOCKER_USERNAME -p $DOCKER_PASSWORD'
                    sh 'docker push rakeshpotnuru/productivity-app:client-latest'
                    sh 'docker push rakeshpotnuru/productivity-app:server-latest'
                }
            }
        }
    }
}

Pipeline ran successfully

Conclusion

In summary, let's review what we've covered:

• We explored the significance of implementing Continuous Integration and Continuous Deployment (CI/CD) in software development.
• We delved into the fundamentals of Jenkins and acquired knowledge on how to deploy a Jenkins server on the Azure cloud platform.
• We customized Jenkins to meet our specific requirements.
• Lastly, we wrote a Jenkinsfile and built a pipeline utilizing the user-friendly interface of Jenkins Blue Ocean.

That's all for now! Thanks for reading 🙂.

Connect with me on twitter.

In this article, we'll learn how to set up a CI/CD pipeline with GitHub Actions and AWS. I've divided the guide into three parts to help you work through it:

First, we'll cover some important terminology so you're not lost in a bunch of big buzzwords.

Second, we'll set up continuous integration so we can automatically run builds and tests.

And finally, we'll set up continuous delivery so we can automatically deploy our code to AWS.

Alright, that was a lot. Let's start by diving into each of these terms so you understand exactly what we're doing here.

Part One: Demystifying the Hefty Buzzwords

The key to making sense of the title of this piece lies in understanding the terms CI/CD Pipeline, GitHub Actions, and AWS.

If you already have a strong grasp of what these terms are, you can just skip to down to Part 2.

What is a CI/CD Pipeline?

A CI/CD Pipeline is simply a development practice. It tries to answer this one question: How can we ship quality features to our production environment faster? In other words, how can we hasten the feature release process without compromising on quality?

How does the CI/CD Pipeline help us hasten the feature release process, you might ask?

The diagram below depicts a typical feature delivery cycle with or without the CI/CD pipeline.

the feature release process. Source: Author

Without the CI/CD Pipeline, each step in the diagram above will be performed manually by the developer. In essence, to build the source code, someone on your team has to manually run the command to initiate the build process. Same thing with running tests and deployment.

The CI/CD approach is a radical shift from the manual way of doing things. It is entirely based on the premise that we can speed up the feature release process reasonably, if we automate steps 3-6 in the diagram above.

With the CI/CD Pipeline, we set up a mechanism that automatically starts the build process, runs the tests, deploys to the User Acceptance Testing (UAT) environment, and finally to the production environment each time a member of the team pushes their change to the shared repo, for example.

Continuous Integration happens each time the build process is initiated, and tests run on a new change.

Continuous Delivery happens when a newly integrated change is automatically deployed to the UAT environment and then manually deployed to the production environment from there.

Continuous Deployment happens when an update in the UAT environment is automatically deployed to the production environment as an official release.

Continuous Integration vs Continuous Deployment vs Continuous Delivery. Source: Author

Note: If the deployment from the UAT environment to the production environment is initiated manually, then it is a Continuous Integration/Continuous Delivery setup. Otherwise, it is a Continuous Integration/Continuous Deployment set up.

However, we can't help but ask, what is that entity that automates the different phases of the CI/CD Pipeline?

There are variety of tools we can use to automate the build, tests, and deployment steps in the CI/CD Pipeline – for example, CI Circle, Travis CI, Jenkins, GitHub Actions, and so on. In this article we will be focusing on GitHub Actions.

What are GitHub Actions?

For want of a better way of making the GitHub Actions term super comprehensible, I'm going to oversimplify this.

In the CI/CD Pipeline, GitHub Actions is the entity that automates the boring stuff. Think of it as some plugin that comes bundled with every GitHub repository you create.

The plugin exists on your repo to execute whatever task you tell it to. Usually, you'd specify what tasks the plugin should execute through a YAML configuration file. Whatever command you add to the configuration file, will translate to something like this in plain English:

"hey GitHub Actions, each time a PR is opened on X branch, automatically build and test the new change. And each time a new change is merged into or pushed to X branch, deploy that change to Y server."

At the core of GitHub Actions lies five concepts: jobs, workflows, events, actions, and runners.

Jobs are the tasks you command GitHub Actions to execute through the YAML config file. A job could be something like telling GitHub actions to build your source code, run tests, or deploy the code that has been built to some remote server.

Workflows are essentially automated processes that contain one or more logically related jobs. For example, you could put the build and run tests jobs into the same workflow, and the deployment job into a different workflow.

Remember, we mentioned that you tell GitHub Actions what job(s) to execute through a configuration file right? GitHub Actions considers each configuration file that you put in some folder in your repo a workflow. We will talk more about this folder in the next part.

So, to create a separate workflow for the deployment job and then a different workflow that combines the build and tests jobs, you'd have to add two config files to your repo. But if you are merging all the three jobs into a single workflow, then you'd need to add just one config file.

Events are literally the events that trigger the execution of a job by GitHub Actions. Recall we mentioned passing jobs to be executed through a config file? In that config file you'd also have to specify when a job should be executed.

For example, is it on-PR to main? Is it on-push to main? is it on-merge to main? A job can only be executed by a GitHub Action when some event happens.

Okay, let me quickly correct myself. It's not always the case that some event has to happen before a job could be executed. You could schedule jobs too.

For example, in your config file, instead of specifying that the event that should trigger the execution of, let's say, the build-and-test job, you could schedule it to happen a 2am everyday. In fact, you could both schedule a job and specify an event for that same job.

Actions are the reusable commands that you can reuse in your config file. You can write your custom actions or use existing ones.

A runner is the remote computer that GitHub Actions uses to execute the jobs you tell it to.

For example, when the build-and-test job is triggered based on some event, GitHub Actions will pull your code to that computer and execute the job.

The same thing happens in the case of the deployment job. The runner triggers the deployment of the built code to some remote server you specify. In our case, we will be using a service called AWS.

What is AWS?

AWS stands for Amazon Web Services. It is a platform owned by Amazon, and this platform allows you access to a broad range of cloud computing services.

Cloud computing – I thought you said no big words? Most times businesses and even individual developers build applications just so other people can use them. For that reason, these applications have to be available on the interwebs.

Making an application accessible to some target users, ideally, entails uploading that application to a special computer that runs 24/7 and is super fast.

Imagine if it were the case that, before you could make your applications available to other internet users, you'd have to own and set up such a computer. It is quite doable, but it comes with a lot of hurdles.

For example, what if you just want to test out the application? You'd go through all the stress of setting up a hardware infrastructure just for testing? Furthermore, what if you want to upload 1000 different applications or your application is beginning to handle 1billion concurrent requests? Things start to get complicated.

Cloud computing platforms like AWS exist to save you all that stress. These platforms already have numerous of these special computers setup and kept in buildings called data centers.

Instead of having to setup your own hardware infrastructure from scratch, they allow you upload your application to one of their pre-configured computers over the internet. In return, you pay some certain amount to them.

In fact, some of these platforms have free plans that allows you test out small applications. In addition to uploading your application's code, some of these platforms also allow you host your database and store your media files, amongst other features.

In its most simplistic form, Cloud Computing is primarily about storing or executing (sometimes both) certain things on someone else's computer, usually, over a network.

So when I said AWS gives access to a wide range of cloud services, I was just saying it provides businesses and individuals with some special computer where they could upload their code, databases, and media files as a service.

Okay, now that we fully understand the different parts of our title, we will now restate it in the form of objectives. These objectives will then dictate what the remaining two parts in this article will contain.

Objective 1: How to automatically build and run unit tests on push or on PR to the main branch with GitHub Actions.

Objective 2: How to automatically deploy to AWS on push or on PR to the main branch with GitHub Actions.

Part 2: Continuous Integration – How to Automatically Run Builds and Tests

In this section, we will be seeing how we can configure GitHub Actions to automatically run builds and tests on push or pull request to the main branch of a repo.

Prerequisites

• A Django project setup locally with at least one view that returns some response defined.
• A testcase written for the view(s) you've defined.

I have created a demo Django project which you can grab from this repository:

git clone [https://github.com/Nyior/django-github-actions-aws](https://github.com/Nyior/django-github-actions-aws)

After you download the code, create a virtualenv and install the requirements via pip:

pip install -r requirements.txt

Note: The project already has all the files we will be adding incrementally as we proceed. Maybe you could still download and try to make sense of the content of the files as we proceed. The project is certainly not interesting. It's just created for demo purposes.

Now that you have a Django project setup locally, let's configure GitHub Actions.

How to Configure GitHub Actions

Okay, so we have our project setup. We also have a testcase written for the view that we have defined, but most importantly we've pushed our shiny change to GitHub.

The goal is to have GitHub trigger a build and run our tests each time we push or open a pull request on main/master. We just pushed our change to main, but GitHub Actions didn't trigger the build or run our tests.

Why not? Because we haven't defined a workflow yet. Remember, a workflow is where we specify the jobs we want GitHub Actions to execute.

In fact, Nyior, how did you even know that no build was triggered and by extension no workflow defined? Every GitHub repo has an Action tab. If you navigate to that tab, you'll know if a repo has a workflow defined on it or not.

How? See the images below.

A GitHub Repo With a Workflow Defined on it

A GitHub Repo With No Workflow Defined on it

The first repo in the first image has a workflow defined on it named 'Lint and Test'. The second repo in the second image has no workflow – it's why you don't see a list with the heading 'All Workflows' as is the case with the first repo.

Oh okay, now I get it. So how do I define a workflow on my repo?

• Create a folder named .github in the root of your project directory.
• Create a folder named workflows in the .github directory: This is where you'll create all your YAML files.
• Let's create our first workflow that will contain our build and test jobs. We do that by creating a file with a .yml extension. Let's name this file build_and_test.yml
• Add the content below in the yaml file you just created:

name: Build and Test

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest

    steps:
    - name: Checkout code
      uses: actions/checkout@v2
    - name: Set up Python Environment
      uses: actions/setup-python@v2
      with:
        python-version: '3.x'
    - name: Install Dependencies
      run: |
        python -m pip install --upgrade pip
        pip install -r requirements.txt

    - name: Run Tests
      run: |
        python manage.py test

Let's make sense of each line in the file above.

• name: Build and Test This is the name of our workflow. When you navigate to the actions tab, each workflow you define will be identified by the name you give it here on that list.
• on: This is where you specify the events that should trigger the execution of our workflow. In our config file we passed it two events. We specified the main branch as the target branch.
• jobs: Remember, a workflow is just a collection of jobs.
• test: This is the name of the job we've defined in this workflow. You could name it anything really. Notice it's the only job and the build job isn't there? Well, it's Python code so no build step is required. This is why we didn't define the build job.
• runs-on GitHub provides Ubuntu Linux, Microsoft Windows, and macOS runners to run your workflows. This where you specify the type of runner you want to use. In our case, we are using the Ubuntu Linux runner.
• A job is made up of a series of steps that are usually executed sequentially on the same runner. In our file above, each step is marked by a hyphen. name represents the name of the step. Each step could either be a shell script that is being executed or an action . You define a step with uses if it's executing an action or you define a step with run if it's executing a shell script.

Now that you've defined a workflow by adding the config file in the designated folder, you can commit and push your change to your remote repo.

If you navigate to the Actions tab of your remote repo, you should see a workflow with the name Build and Test (the name which we've given it) listed there.

Part 3: Continuous Delivery – How to Automatically Deploy Our Code to AWS

In this section, we will see how we can have GitHub Actions automatically deploy our code to AWS on push or pull request to the main branch. AWS offers a broad range of services. For this tutorial, we will be using a compute service called Elastic Beanstalk.

Compute Service? Elastic Beanstalk? Come onnn :(

No worries, relax, you'll get it. Remember we mentioned that cloud computing is all about storing and executing certain things on someone else's computer via the internet right – certain things?

Yes. For example, we can store and execute source code or we can just store media files. Amazon knows this, and as a result, their cloud infrastructure encompasses a plethora of service categories. Each service category allows us do a certain thing out of the certain things that we can do.

For example, there is a service category that allows the upload and execution of source codes that power our applications (Compute Service). There is the service category that allows us persist our media files (Storage Service). Then there is the service category that allows us manage our databases (Database Service).

Each service category is made up of one or more services. Each service in a category just presents us with a different way of solving the problem that the category it belongs to addresses.

For example, each service in the compute category provides us with a different approach to deploying and executing our application code on the cloud – one problem, different approaches. Elastic Beanstalk is one of the services in the compute category. Others are, but not limited to, EC2 and Lambda.

Amazon S3 is one of the services in the storage category. And Amazon RDS is one of the services in the Database category.

Hopefully, you now understand what I mean by "we will be using a compute service called Elastic Beanstalk." Of all the compute services, why Elastic Beanstalk? Well, because it's one of the easiest to work with.

That Being Said, Let's Configure Stuff <3

For brevity's sake we are going with the Continuous Delivery setup. In addition, we are going to have just one one deployment environment that will serve as our UAT environment.

To help you get the big picture right, in summary, this is how our deployment setup is going to work: on push or pull request to main, GitHub Actions will test and upload our source code to Amazon S3. The code is then pulled from Amazon S3 to our Elastic Beanstalk environment. Picture the flow this way:

GitHub -> Amazon S3 -> Elastic Beanstalk

Why aren't we pushing directly to Elastic Beanstalk, you might ask?

The only other way we could upload code directly to an Elastic Beanstalk instance with our current setup, is if we were using the AWS Elastic Beanstalk CLI (EB CLI).

Using the EB CLI requires running some shell command that would then require that we respond with some input.

Now, if we are deploying from our local machine to Elastic Beanstalk, when we run the EB CLI commands, we'd be there to type in the required responses. But with our current setup, those commands would be executed on GitHub Runners. We wouldn't be there to provide the required responses. The EB CLI isn't the easiest deployment tool for our use case.

With the approach we've picked, we'd run a shell command that uploads our code to S3 and then another command that pulls the uploaded code to our Elastic Beanstalk instance. These commands, when run, do not require that we submit some responses. Having the Amazon s3 step is the easiest way to go about this.

To implement our approach and have our code deployed to Elastic Beanstalk, follow the steps below:

Step 1: Setup an AWS Account

Create an IAM. To keep things simple, when adding permissions, just add "Administrator Access" to the user (this has some security pitfalls, though). To accomplish this, follow the steps in modules 1 and 2 of this guide.

In the end, make sure to grab and keep your AWS secret and access keys. We will be needing them in the subsequent sections.

Now that we have an AWS account properly setup, it's time to set up our Elastic Beanstalk environment.

Step 2: Setup your Elastic Beanstalk Environment

Once logged into your AWS account, take the following steps to set up your Elastic Beanstalk environment.

First, search for "elastic beanstalk" in the search field as shown in the image below. Then click on the Elastic Beanstalk service.

searching for elastic beanstalk.

Once you click on the Elastic Beanstalk service in the previous step, you'll be taken to the page shown in the image below. On that page, click on the "Create a New Environment" prompt. Make sure to select "Web server environment" in the next step.

creating an environment

After selecting the "Web server environment" in the previous page you'll be taken to the page shown in the images below.

On that page, submit an application name, an environment name, and also select a platform. For this tutorial, we are going with the Python platform.

an application and an environment names

selecting a platform

Once you submit the form filled out in the previous step, after a while your application and its associated environment will be created. You should see the names you submitted displayed on the left side bar.

Grab the application name and the environment name. We will be needing them in the subsequent steps.

Now that we have our Elastic Beanstalk environment fully setup, it's time to configure GitHub Actions to trigger automatic deployment to AWS on push or pull request to main.

Step 3: Configure your Project for Elastic Beanstalk

By default, Elastic Beanstalk looks for a file named application.py in our project. It uses that file to run our application, but we don't have that file in our project. Do we? We need to tell Elastic Beanstalk to use the wsgi.py file in our project to run our application instead. To do that, take the following step:

Create a folder named .ebextensions in your project directory. In that folder create a config file. You can name it anything you want. I named mine eb.config. Add the content below to your config file:

option_settings:
  aws:elasticbeanstalk:container:python:
    WSGIPath: django_github_actions_aws.wsgi:application

After completing the step above, your project directory should now look similar to the one in the image below:

demo project structure

One last thing you need to do in this section is to go to your settings.py file and update the ALLOWED_HOSTS setting to all:

ALLOWED_HOSTS = ['*']

Note that using the wildcard has major security drawbacks. We are only using it here for demo purposes.

Now that we are done configuring our project for Elastic Beanstalk, it's time to update our workflow file.

Step 4: Update your Workflow File

There are five important pieces of information we need to complete this step: application name, environment name, access key id, secret access key, and the server region (after login, you can grab the region from the right-most section of the navbar).

Because the access key id and the secret access key are sensitive data, we'll hide them somewhere in our repository and access them in our workflow file.

To do that, head over to the settings tab of your repo, and then click on secrets as shown in the image below. There, you can create your secrets as key-value pairs.

embedding secret data in your repo

Next, add the deployment job to the end of your existing workflow file:

deploy:
    needs: [test]
    runs-on: ubuntu-latest

    steps:
    - name: Checkout source code
      uses: actions/checkout@v2

    - name: Generate deployment package
      run: zip -r deploy.zip . -x '*.git*'

    - name: Deploy to EB
      uses: einaregilsson/beanstalk-deploy@v20
      with:

          // Remember the secrets we embedded? this is how we access them
        aws_access_key: ${{ secrets.AWS_ACCESS_KEY_ID }}
        aws_secret_key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

        // Replace the values here with your names you submitted in one of 
        // The previous sections
        application_name: django-github-actions-aws
        environment_name: django-github-actions-aws

        // The version number could be anything. You can find a dynamic way 
        // Of doing this.
        version_label: 12348
        region: "us-east-2"
        deployment_package: deploy.zip

needs simply tells GitHub Actions to only start executing the deployment job after the test job has been completed with a passing status.

The step Deploy to EB uses an existing action, einaregilsson/beanstalk-deploy@v20. Remember how we said actions are some reusable applications that takes care of some frequently repeated tasks for us? einaregilsson/beanstalk-deploy@v20 is one of those actions.

To reinforce the above, remember that our deployment was suppose to go through the following steps: GitHub -> Amazon S3 -> Elastic Beanstalk.

However, throughout this tutorial, we didn't do any Amazon s3 set up. Furthermore, in our workflow file we didn't upload to an s3 bucket nor did we pull from an s3 bucket to our Elastic Beanstalk environment.

Normally, we are supposed to do all that, but we didn't here – because under the hood, the einaregilsson/beanstalk-deploy@v20 action does all the heavy lifting for us. You can also create your own action that takes care of some repetitive tasks and make it available to other developers through the GitHub Marketplace.

Now that you've updated your workflow file locally, you can then commit and push this change to your remote. Your jobs will run and your code will be deployed to the Elastic Beanstalk instance you created. And that's it. We're done >>>

Wrapping Up

Wow! This was a really long one, wasn't it? In summary I explained what the terms GitHub Actions, CI/CD Pipeline, and AWS mean. In addition, we saw how we could configure GitHub Actions to automatically deploy our code to an Elastic Beanstalk instance on AWS.

If you love this work, and would like to stay up to date on future articles I will be putting out, let's connect on Twitter, Linkedin, or GitHub. I use those channels to share what I work on, immediately after I put them out.

Credits:

Cover image: www.freepik.com

A monorepo is a single repository that holds all the code and multiple projects in a single Git repository.

This setup is quite nice to work with because of its flexibility and ability to manage various services and frontends in one single repository. It also eliminates the hassle of tracking changes in multiple repositories and updating dependencies as projects change.

On the other hand, monorepos also come with their challenges, specifically as relates to Continuous Integration. As individual sub-projects within the monorepo change, we need to identify which sub-projects changed to build and deploy them.

This post will serve as a step by step guide to:

1. Configure Continuous Integration for monorepos in Bulidkite.
2. Deploy Buildkite Agents to AWS EC2 instances with autoscaling.
3. Configure GitHub to trigger Bulidkite CI pipelines.
4. Configure Buildkite to trigger appropriate pipelines when sub-projects within a monorepo change.
5. Automate all of the above using bash scripts.

Pre-requisites

1. AWS account to deploy the Buildkite agents.
2. Configure AWS CLI to talk to AWS Account.
3. Buildkite account to create continuous integration pipelines.
4. GitHub account to host the monorepo sourcecode.

The complete source code is available in the buildkite-monorepo in GitHub.

Project Setup

The Buildkite workflow consists of Pipelines and Steps. The top-level containers for modeling and defining workflows are called Pipelines. Steps run individual tasks or commands.

The following diagram lists the pipelines we are setting up, their associated triggers, and each step that the pipeline runs.

Pull Request Workflow

The above diagram visualizes the workflow for the Pull Request pipeline.

Creating a new Pull Request in GitHub triggers the pull-request pipeline in Buildkite. This pipeline then runs git diff to identify which folders (projects) within the monorepo have changed.

If it detects changes, then it will dynamically trigger the appropriate Pull Request pipeline defined for that project. Buildkite reports the status of each pipeline back to GitHub status check.

Merge Workflow

The Pull Request is merged when all status checks in Github pass. Merging Pull Request triggers the merge pipeline in Buildkite.

Similar to the previous pipeline, the merge pipeline identifies the projects that have changed and triggers the corresponding deploy pipeline for it. The Deploy pipeline initially deploys changes to the staging environment.

Once the deployment to staging is complete, production deployment is manually released.

Final project structure

├── .buildkite
│ ├── diff
│ ├── merge.yml
│ ├── pipelines
│ │ ├── deploy.json
│ │ ├── merge.json
│ │ └── pull-request.json
│ └── pull-request.yml
├── bar-service
│ ├── .buildkite
│ │ ├── deploy.yml
│ │ ├── merge.yml
│ │ └── pull-request.yml
│ └── bin
│ └── deploy
├── bin
│ ├── create-pipeline
│ ├── create-secrets-bucket
│ ├── deploy-ci-stack
│ └── stack-config
└── foo-service
├── .buildkite
│ ├── deploy.yml
│ ├── merge.yml
│ └── pull-request.yml
└── bin
└── deploy

Set Up the Project

Create a new Git project and push it to GitHub. Run the following commands in the CLI.

mkdir buildkite-monorepo-example
cd buildkite-monorepo-example
git init
echo node_modules/ > .gitignore
git add .
git commit -m "initialize repository"
git remote add origin <YOUR_GITHUB_REPO_URL>
git push origin master

Set up the Buildkite infrastructure

1. Create a bin directory with some executable scripts inside it.

mkdir bin 
cd bin
touch create-pipeline create-secrets-bucket deploy-ci-stack
chmod +x ./*

2. Copy the following contents into create-secrets-bucket.

#!/bin/bash

set -eou pipefail

CURRENT_DIR=$(pwd)
ROOT_DIR="$( dirname "${BASH_SOURCE[0]}" )"/..

BUCKET_NAME="buildkite-secrets-adikari"
KEY="id_rsa_buildkite"

echo "creating bucket $BUCKET_NAME.."
aws s3 mb s3://$BUCKET_NAME

# Generate SSH Key
ssh-keygen -t rsa -b 4096 -f $KEY -N ''

# Copy SSH Keys to S3 bucket
aws s3 cp --acl private --sse aws:kms $KEY "s3://$BUCKET_NAME/private_ssh_key"
aws s3 cp --acl private --sse aws:kms $KEY.pub "s3://$BUCKET_NAME/public_key.pub"


if [[ "$OSTYPE" == "darwin"* ]]; then
  pbcopy < id_rsa_buildkite.pub
  echo "public key contents copied in clipboard."
else
  cat id_rsa_buildkite.pub
fi

# Move SSH Keys to ~/.ssh directory
mv ./$KEY* ~/.ssh
chmod 600 ~/.ssh/$KEY
chmod 644 ~/.ssh/$KEY.pub

cd $CURRENT_DIR

The above script creates an S3 bucket that is used to store the ssh keys. Buildkite uses this key to connect to the Github repo. The script also generates an ssh key and sets its permission correctly.

Run the script

The script copies the generated public and private keys to the ~/.ssh folder. These keys can be used later to ssh into the EC2 instance, running the Buildkite agent for debugging.

Next, verify that the bucket exists and the keys are present in the new S3 bucket.

Navigate to https://github.com/settings/keys, add a new SSH key, then paste in the contents of id_rsa_buildkite.pub .

Deploy AWS Elastic CI Cloudformation Stack

The folks at Buildkite have created the Elastic CI Stack for AWS, which creates a private, autoscaling Buildkite Agent cluster in AWS. Let's deploy the infrastructure to our AWS Account.

Create a new file bin/deploy-ci-stack and copy the contents of the following script in it.

#!/bin/bash

set -euo pipefail

[ -z $BUILDKITE_AGENT_TOKEN ] && { echo "BUILDKITE_AGENT_TOKEN is not set."; exit 1;}

CURRENT_DIR=$(pwd)
ROOT_DIR="$( dirname "${BASH_SOURCE[0]}" )"/..
PARAMETERS=$(cat ./bin/stack-config | envsubst)

cd $ROOT_DIR

echo "downloading elastic ci stack template.."
curl -s https://s3.amazonaws.com/buildkite-aws-stack/latest/aws-stack.yml -O

aws cloudformation deploy \
  --capabilities CAPABILITY_NAMED_IAM \
  --template-file ./aws-stack.yml \
  --stack-name "buildkite-elastic-ci" \
  --parameter-overrides $PARAMETERS

rm -f aws-stack.yml

cd $CURRENT_DIR

You can get the BUILDKITE_AGENT_TOKEN from the Agents tab in Buildkite's Console.

Next, create a new file called bin/stack-config. Configuration in this file overrides the Cloudformation parameters. The complete list of parameters is available in the Cloudformation template used by Elastic CI.

On line 2, replace the bucket name with the bucket created earlier.

BuildkiteAgentToken=$BUILDKITE_AGENT_TOKEN
SecretsBucket=buildkite-secrets-adikari
InstanceType=t2.micro
MinSize=0
MaxSize=3
ScaleUpAdjustment=2
ScaleDownAdjustment=-1

Next, run the script in the CLI to deploy the Cloudformation stack.

./bin/deploy-ci-stack

The script will take some time to finish. Open up the AWS Cloudformation console to view the progress.

The Cloudformation stack would have created an Autoscaling Group that Buildkite will use to spawn up EC2 instances. The Buildkite Agents and the builds run inside those EC2 instances.

Create build pipelines in Bulidkite

At this point, we have the infrastructure ready that's required to run Buildkite. Next, we configure Buildkite and create some Pipelines.

Create an nAPI Access Token at https://buildkite.com/user/api-access-tokens and set the scope to write_builds, read_pipelines, and write_pipelines. More information about agent tokens is in this document.

Ensure the BUILDKITE_API_TOKEN is set on the environment. Either use dotenv or export it to the environment before running the script.

Copy the contents of the following script to bin/create-pipeline. Pipelines can be created manually in the Buildkite Console, but it is always better to automate and create reproducible infrastructure.

#!/bin/bash

set -euo pipefail

export SERVICE="."
export PIPELINE_TYPE=""
export REPOSITORY=git@github.com:adikari/buildkite-docker-example.git

CURRENT_DIR=$(pwd)
ROOT_DIR="$( dirname "${BASH_SOURCE[0]}" )"/..
STATUS_CHECK=false
BUILDKITE_ORG_SLUG=adikari # update to your buildkite org slug

USAGE="USAGE: $(basename "$0") [-s|--service] service_name [-t|--type] pipeline_type
Eg: create-pipeline --type pull-request
    create-pipeline --type merge --service foo-service
    create-pipeline --type merge --status-checks
NOTE: BUILDKITE_API_TOKEN must be set in environment
ARGUMENTS:
    -t | --type           buildkite pipeline type <merge|pull-request|deploy> (required)
    -s | --service        service name (optional, default: deploy root pipeline)
    -r | --repository     github repository url (optional, default: buildkite-docker-example)
    -c | --status-checks      enable github status checks (optional, default: true)
    -h | --help           show this help text"

[ -z $BUILDKITE_API_TOKEN ] && { echo "BUILDKITE_API_TOKEN is not set."; exit 1;}

while [ $# -gt 0 ]; do
    if [[ $1 =~ "--"* ]]; then
        case $1 in
            --help|-h) echo "$USAGE"; exit; ;;
            --service|-s) SERVICE=$2;;
            --type|-t) PIPELINE_TYPE=$2;;
            --repository|-r) REPOSITORY=$2;;
            --status-check|-c) STATUS_CHECK=${2:-true};;
        esac
    fi
    shift
done

[ -z "$PIPELINE_TYPE" ] && { echo "$USAGE"; exit 1; }

export PIPELINE_NAME=$([ $SERVICE == "." ] && echo "" || echo "$SERVICE-")$PIPELINE_TYPE

BUILDKITE_CONFIG_FILE=.buildkite/pipelines/$PIPELINE_TYPE.json
[ ! -f "$BUILDKITE_CONFIG_FILE" ] && { echo "Invalid pipeline type: File not found $BUILDKITE_CONFIG_FILE"; exit; }

BUILDKITE_CONFIG=$(cat $BUILDKITE_CONFIG_FILE | envsubst)

if [ $STATUS_CHECK == "false" ]; then
  pipeline_settings='{ "provider_settings": { "trigger_mode": "none" } }'
  BUILDKITE_CONFIG=$((echo $BUILDKITE_CONFIG; echo $pipeline_settings) | jq -s add)
fi
cd $ROOT_DIR
echo "Creating $PIPELINE_TYPE pipeline.."
RESPONSE=$(curl -s POST "https://api.buildkite.com/v2/organizations/$BUILDKITE_ORG_SLUG/pipelines" \
  -H "Authorization: Bearer $BUILDKITE_API_TOKEN" \
  -d "$BUILDKITE_CONFIG"
)
[[ "$RESPONSE" == *errors* ]] && { echo $RESPONSE | jq; exit 1; }
echo $RESPONSE | jq
WEB_URL=$(echo $RESPONSE | jq -r '.web_url')
WEBHOOK_URL=$(echo $RESPONSE | jq -r '.provider.webhook_url')
echo "Pipeline url: $WEB_URL"
echo "Webhook url: $WEBHOOK_URL"
echo "$PIPELINE_NAME pipeline created."
cd $CURRENT_DIR
unset REPOSITORY
unset PIPELINE_TYPE
unset SERVICE
unset PIPELINE_NAME

Make the script executable by setting the correct permission (chmod +x). Run ./bin/create-pipeline -h in the CLI for help.

The script uses Buildkite REST API to create the pipelines with the given configuration. The script uses a pipeline configuration defined as a json document and posts it to the REST API. Pipeline configurations live in the .bulidkite/pipelines folder.

To define the configuration for the pull-request pipeline, create .buildkite/pipelines/pull-request.json with the following content:

{
  "name": "$PIPELINE_NAME",
  "description": "Pipeline for $PIPELINE_NAME pull requests",
  "repository": "$REPOSITORY",
  "default_branch": "",
  "steps": [
    {
      "type": "script",
      "name": ":buildkite: $PIPELINE_TYPE",
      "command": "buildkite-agent pipeline upload $SERVICE/.buildkite/$PIPELINE_TYPE.yml"
    }
  ],
  "cancel_running_branch_builds": true,
  "skip_queued_branch_builds": true,
  "branch_configuration": "!master",
  "provider_settings": {
    "trigger_mode": "code",
    "publish_commit_status_per_step": true,
    "publish_blocked_as_pending": true,
    "pull_request_branch_filter_enabled": true,
    "pull_request_branch_filter_configuration": "!master",
    "separate_pull_request_statuses": true
  }
}

Next, create ./buildkite/pipelines/merge.json with the following content:

{
  "name": "$PIPELINE_NAME",
  "description": "Pipeline for $PIPELINE_NAME merge",
  "repository": "$REPOSITORY",
  "default_branch": "master",
  "steps": [
    {
      "type": "script",
      "name": ":buildkite: $PIPELINE_TYPE",
      "command": "buildkite-agent pipeline upload $SERVICE/.buildkite/$PIPELINE_TYPE.yml"
    }
  ],
  "cancel_running_branch_builds": true,
  "skip_queued_branch_builds": true,
  "branch_configuration": "master",
  "provider_settings": {
    "trigger_mode": "code",
    "build_pull_requests": false,
    "publish_blocked_as_pending": true,
    "publish_commit_status_per_step": true
  }
}

Finally, create .buildkite/pipelines/deploy.yml with the following content:

{
  "name": "$PIPELINE_NAME",
  "description": "Pipeline for $PIPELINE_NAME deploy",
  "repository": "$REPOSITORY",
  "default_branch": "master",
  "steps": [
    {
      "type": "script",
      "name": ":buildkite: $PIPELINE_TYPE",
      "command": "buildkite-agent pipeline upload $SERVICE/.buildkite/$PIPELINE_TYPE.yml"
    }
  ],
  "provider_settings": {
    "trigger_mode": "none"
  }
}

Now, run the ./bin/create-pipeline command to create a pull-request pipeline.

./bin/create-pipeline --type pull-request --status-checks
./bin/create-pipeline --type merge --status-checks

Copy the Webhook url from the console output and create a webhook integration in GitHub. The webhook URL is available in the pipeline settings in the Buildkite console if needed in the future.

We need to configure the webhook only for the pull-request and merge pipelines. All other pipelines are triggered dynamically.

Navigate to the GitHub repository Settings > Webhooks and add a webhook. Select Just the push event, then add the webhook. Repeat this for both pipelines.

Now in the Buildkite Console, there should be two newly created pipelines. 🎉

Next, add GitHub integration to allow Buildkite to send status updates to GitHub. You only need to set up this integration once per account. It is available at Setting > Integrations > Github in the Buildkite Console.

Next, create the remaining pipelines. These pipelines will be dynamically triggered by the pull-request and merge pipelines, so we do not need to create GitHub integration.

# foo service pipelines
./bin/create-pipeline --type pull-request --service foo-service
./bin/create-pipeline --type merge --service foo-service
./bin/create-pipeline --type deploy --service foo-service

# bar service pipelines
./bin/create-pipeline --type pull-request --service bar-service
./bin/create-pipeline --type merge --service bar-service
./bin/create-pipeline --type deploy --service bar-service

The Buildkite Console should now have all the pipelines listed. 🥳

Set up Buildkite Steps

Now that the pipelines are ready, let's configure steps to run for each pipeline.

Add the following script in .buildkite/diff. This script diffs between all the files changed in a commit against the master branch. The output of the script is used to trigger respective pipelines dynamically.

#!/bin/bash

[ $# -lt 1 ] && { echo "argument is missing."; exit 1; }

COMMIT=$1

BRANCH_POINT_COMMIT=$(git merge-base master $COMMIT)

echo "diff between $COMMIT and $BRANCH_POINT_COMMIT"
git --no-pager diff --name-only $COMMIT..$BRANCH_POINT_COMMIT

Change the permission of the script to make it executable.

chmod +x .buildkite/diff

Create a new file .buildkite/pullrequest.yml and add the following step configuration. We use the buildkite-monorepo-diff plugin to run the diff script and automatically upload and trigger the respective pipelines.

steps:
  - label: "Triggering pull request pipeline"
    plugins:
      chronotc/monorepo-diff#v1.1.1:
        diff: ".buildkite/diff ${BUILDKITE_COMMIT}"
        wait: false
        watch:
          - path: "foo-service"
            config:
              trigger: "foo-service-pull-request"
          - path: "bar-service"
            config:
              trigger: "bar-service-pull-request"

Now create the configuration for the merge pipeline by adding the following content in .buildkite/merge.yml.

steps:
  - label: "Triggering merge pipeline"
    plugins:
      chronotc/monorepo-diff#v1.1.1:
        diff: "git diff --name-only HEAD~1"
        wait: false
        watch:
          - path: "foo-service"
            config:
              trigger: "foo-service-merge"
          - path: "bar-service"
            config:
              trigger: "bar-service-merge"

At this point, we have configured the topmost level pull-request and merge pipelines. Now we need to configure individual pipelines for each service.

We'll configure pipelines for foo-service first. Create foo-service/.buildkite/pull-request.yml with the following content. When the pull-request pipeline for foo service runs, specify that the lint and test commands should run. The command option can also trigger other scripts.

steps:
  - label: "Foo service pull request"
    command:
      - "echo linting"
      - "echo testing"

Next, setup a merge pipeline for the foo service by adding the following content in foo-service/.buildkite/merge.yml:

steps:
  - label: "Run sanity checks"
    command:
      - "echo linting"
      - "echo testing"

  - label: "Deploy to staging"
    trigger: "foo-deploy"
    build:
      env:
        STAGE: "staging"

  - wait

  - block: ":rocket: Release to Production"

  - label: "Deploy to production"
    trigger: "foo-deploy"
    build:
      env:
        STAGE: "production"

When the foo-service-merge pipeline runs, here is what happens:

1. The pipeline runs the sanity check.
2. Then foo-deploy pipeline is dynamically triggered. We pass the STAGE environment to identify which environment to run the deployment against.
3. Once the deployment to staging is complete, the pipeline is blocked and the following pipeline is not triggered automatically. The pipeline can be resumed by pressing the “Release to Production” button.
4. Unblocking the pipeline triggers foo-deploy pipeline again, but this time with production stage.

Finally, add configuration for the foo-deploy pipeline by adding foo-service/.buildkite/deploy.yml. In the deploy configuration, we trigger a bash script and pass the STAGE variable which was received from the foo-service-merge pipeline.

steps:
  - label: "Deploying foo service to ${STAGE}"
    command: "./foo-service/bin/deploy ${STAGE}"

Now, create the deploy script foo-service/bin/deploy and add the following content:

#!/bin/bash

set -euo pipefail

STAGE=$1

echo "Deploying foo service to $STAGE"

Make the deploy script executable like this:

chmod +x ./foo-service/bin/deploy

The pipeline and steps configuration for foo-service are now complete. Repeat all the above steps above to configure pipelines for bar service.

Test the overall workflow

We have configured Buildkite and GitHub and we've set up the appropriate infrastructure to run the builds. Next, test the entire workflow and see it in action.

To test the workflow, start by creating a new branch and modifying some file in foo-service. Push the changes to GitHub and create a Pull Request.

git checkout -b change-foo-service
cd foo-service && touch test.txt
echo testing >> test.txt
git add .
git commit -m 'making some change'
git push origin master

Pushing changes to GitHub should trigger the pull-request pipeline in Buildkite, which then triggers the foo-service-pull-request pipeline.

GitHub should report the status in GitHub checks. You can enable GitHub's branch protection to require the checks to pass before merging the Pull Request.

Once all the checks have passed in GitHub, merge the Pull Request. This merge will trigger the merge pipeline in Buildkite.

The changes in the foo service are detected, and foo-service-merge pipeline is triggered. The pipeline will eventually be blocked when the foo-service-deploy runs against the staging environment.

Unblock the pipeline by manually clicking the Release to Production button to run deployment against production.

Summary

In this post, we set up a continuous integration pipeline for a monorepo using Buildkite, Github, and AWS.

The pipeline gets our code from the development machine to staging, then to production. The build agents and steps run in autoscaled AWS EC2 instances.

We also created a bunch of bash scripts to create easily reproducible versions of this setup.

As an improvement to the current design, consider using the buildkite-docker-compose-plugin to isolate the builds in Docker containers.

Follow me on Twitter or check out my projects on Github.

These days, applications have to evolve as the needs of their target users grow and change. This is why engineering teams often adopt Agile software development principles (or any iterative variation).

Agile principles involve continuous integration and continuous delivery (CI/CD). This means that developers will frequently make code updates for new features to the existing codebase of the application.

So then how can you verify that a recent code addition doesn't break a part of the application? The answer is continuous testing.

What is Continuous Testing?

Continuous testing is a critical part of the CI/CD pipeline. It helps development teams discover if a particular code commit will break the application build and if it should be integrated or not.

In other words, continuous testing is the practice of integrating automated tests into a software delivery pipeline to determine the risks associated with each code release or addition. These automated tests are usually triggered during or after builds and are carried out using automation testing frameworks or tools.

Let me now introduce you to four recommended automation tools you can use for continuous testing.

Tools for Continuous Testing

1. TestSigma

TestSigma is a cloud-based automation testing tool for continuous testing. It has a low learning curve, as automated tests can be written in plain English, and requires no coding skills. Tests can also be extended with Selenium and JS-based custom functions for more advanced use cases.

TestSigma can be used for web applications, native mobile apps, regression, cross-browser and data-driven testing. It also features inbuilt seamless integrations with test management, bug reporting, CI/CD, and communication tools such as GitHub, Slack, Jira, BrowserStack, Jenkins, AWS, Bamboo, Azure DevOps, Circle CI, and so on.

TestSigma also uses AI to reduce maintenance effort and increase productivity by identifying affected tests and potential failures upfront to save execution time & cost, alongside other features.

The platform has a free tier, but to use all the features mentioned above, you’ll need to commit to a paid plan.

2. Tricentis Tosca

Tosca is another no-code continuous testing tool which makes it easy to learn. QA engineers with zero scripting knowledge can easily set up automated tests using a GUI.

Tosca is suitable for enterprise-level applications and is versatile because it supports and integrates seamlessly with over 160 technologies/languages. With Tosca, you can run tests on the web, mobile, and desktops with Windows OS (Mac and Linux are only possible with virtualization tools).

Tosca automatically creates and provisions on-demand test data to reduce the time it takes to provision and make reliable test data for test automation.

The platform offers free trials for a limited amount of time and custom pricing, which the sales team decides on based on your specific needs.

3. Katalon Studio

Katalon is another comprehensive continuous testing tool built on top of the popular open-source Selenium and Appium. It can be used for testing web, API, mobile, and desktop applications across Windows, macOS, and Linux operating systems.

In fact, with Katalon, you can execute tests on all OSs, browsers, and devices, as well as on cloud, on-premise, and hybrid environments.

Katalon also provides other useful features like recording test steps, executing test cases, providing infrastructure, analytics reporting, and CI/CD integration with the most popular CI tools (like Jenkins, Bamboo, Azure, and CircleCI).

Katalon Studio is easy to get started with because it offers codeless test creation for beginners. For advanced usage, experts can extend automation capabilities using the plugins in the Katalon Store.

It also has extensive documentation featuring a well-organized library of tutorials alongside images and videos to help you out if you ever get stuck on something. It has a robust free tier and an enterprise tier for advanced usage.

4. Watir

Watir is another continuous testing automation tool powered by the Selenium framework, and it is open-source. Watir can only run tests for web applications on Windows and can only execute simple and easily maintainable tests.

It is not codeless, as scripts have to be written in Ruby, Java, .NET or Python using its sister software: Watij, WatiN, and Nerodia. Regardless, it's easy to get started with if you're already familiar with Ruby because it features extensive documentation.

Watir can also be integrated with a couple of CI tools such as Jenkins and GitHub.

Even though Watir seems limited, most teams find its simplicity appealing. It is prevalent within the Ruby community and is even used by large companies like Slack and Oracle.

How to choose a continuous testing tool

There are other excellent continuous testing tools available aside from the four that I have mentioned above. I favour no-code testing tools because it lets teams set up and maintain automated tests much faster.

Regardless, here are a few things to consider before choosing a continuous testing tool:

1. Application types supported: Does the tool support your intended application type (for example, mobile, web, desktop)?
2. Learning curve: How easy/difficult is it to use? Will you need to learn a new scripting language? Ideally, you should go for something with a low learning curve that you and your team can get started with in the shortest amount of time.
3. Costs: Is the cost of the tool a feasible addition to your budget in the long run?
4. Integration capabilities: Can it integrate seamlessly with your existing CI/CD pipeline?
5. Scalability and reusability: Does the tool support scalability and reusability of test cases across multiple projects?
6. Documentation and Community: How concise and rich is the documentation for the tool? You’re going to run into some mental blocks in the future, and you may not be able to get through without proper documentation and community support.

Conclusion

With the right tools, continuous testing eliminates the risks associated with frequent code releases by ensuring that only quality code is delivered to the end-user.

As I previously mentioned, the tools I have listed above are not an exhaustive list of all the continuous testing tool options. They're just the ones I recommend, and they may or may not be the right choice for you.

Do some further research, check out different tools, and settle for one that will integrate seamlessly into your current setup and meet your needs.

I recently got a pull request merged into the popular Phoenix Framework, and I did it without writing any Elixir code. I didn't write any documentation either. What I did was help improve the build process.

In this post, I'd like to share the improvements I made to their build process. These improvements are not Phoenix Framework-specific and they might change the way you approach continuous integration.

But first, some background.

What is the Phoenix Framework?

Phoenix is a web framework with some very interesting properties. With Phoenix, you can build rich interactive web applications without writing client-side code.

You can do this using a feature called LiveView which sends real-time updates from the server to update the client browser's HTML.

We can create a page that shows the latest tweets on a topic, in real-time, quite easily.

Here is an example:

defmodule TimelineLive do
  use Phoenix.LiveView

  def render(assigns) do
    render("timeline.html", assigns)
  end

  def mount(_, socket) do
    Twitter.subscribe("elixirphoenix")
    {:ok, assign(socket, :tweets, [])}
  end

  def handle_info({:new, tweet}, socket) do
    {:noreply,
     update(socket, :tweets, fn tweets ->
       Enum.take([tweet | tweets], 10)
     end)}
  end
end

Real-Time Twitter Results with No Javascript Written

The framework is written in the programming language Elixir

It was created by José Valim. It looks a lot like Ruby but has very different semantics. Elixir runs on the Erlang VM, and it powers projects like Discord and is used at companies like Heroku.

How to Reproduce the Builds

The Phoenix Framework uses GitHub Actions for their build pipeline. Like many great projects, they have a suite of unit tests that they need to run on every user contribution.

This isn't where their testing efforts stop though. They also have a suite of integration tests. Phoenix uses an ORM to talk to various databases and the integration tests ensure that no changes break the integration with any of the 3 supported databases.

This is a common pattern. Having a large number of unit tests that are easy to run and well as a handful of slower but more comprehensive integration tests is a great way to prevent bugs from being introduced into the project.

The Phoenix Framework takes this even further, though, as they also need to support several versions of the Elixir language and a handful of versions of Open Telecom Platform (OTP).

This is starting to sound complex. We have to test each change with all combinations of the following:

• Databases (Postgres, MySQL MSSQL)
• Elixir (Current and Previous Version)
• OTP (Current and Previous Version)

It's relatively easy to set this up in GitHub Actions, but how would you run these tests locally?

Installing all these would be a lot to ask, so contributors tend to rely on GitHubActions to test these combinations. However, if everyone has to rely on pushing things to GitHub it see if the tests pass then development gets slower.

How do we fix this?

How to Unify the Test Runs

This is where I got involved. I work at Earthly Technologies as an open-source developer advocate. We have a pretty cool open-source build tool, and although I occasionally contribute directly to the project my job is to be the contact point between the community using the tool and the team working on it.

I had heard about this reproducibility problem the Phoenix team was having. I thought I could help write a build script that could be used both in GitHub Actions and for a local development workflow. So I set to work on a PR.

Running The Tests Locally

What I ended up creating, slightly simplified, is this:

setup:
   ARG ELIXIR=1.10.4
   ARG OTP=23.0.3
   # Pull a Docker Image to Run Build Inside Of
   FROM hexpm/elixir:$ELIXIR-erlang-$OTP-alpine-3.12.0
   ...

integration-test:
    FROM +setup
    COPY . .
    # Pull In Dependencies
    RUN mix deps.get 
    # Start Up Service Dependencies
    WITH DOCKER --compose docker-compose.yml 
        # Run Tests
        RUN mix test --include database 
    # Stop Service Dependencies
    END

This is an Earthfile. Its made up of several targets, like setup and integration-test. The targets can have dependencies between them. You can use the command-line tool earthly to run any target and each is run in a Docker container. Containerization is going allow us to run the build wherever we choose.

This example runs the integration-test inside of a the hexpm/elixir Docker container with the specified version of Elixir and OTP installed.

Before running the tests with mix test --include database, we use Docker compose to start up all the needed dependencies:

 WITH DOCKER --compose docker-compose.yml
        RUN mix test --include database
 END

The Docker compose file looks like this:

version: '3'
services:
  postgres:
    image: postgres
    ports:
      - "5432:5432"
    environment:
      POSTGRES_PASSWORD: postgres
  mysql:
    image: mysql
    ports:
      - "3306:3306"
    environment:
      MYSQL_ALLOW_EMPTY_PASSWORD: "yes"
  mssql:
    image: mcr.microsoft.com/mssql/server:2019-latest
    environment:
      ACCEPT_EULA: Y
      SA_PASSWORD: some!Password
    ports:
      - "1433:1433"

These are the databases we need for testing Phoenix.

Now we can run the integration tests at the command line like so:

>  earthly -P +integration-test

And if we want to test a different version of Elixir, we can specify the version as build arguments:

 > earthly -P --build-arg ELIXIR=1.11.0 --build-arg OTP=23.1.1 +integration-test

There are other ways to accomplish this. A combination of a makefile and dockerfiles would have worked as well. The key is to get the build logic out of a GHA specific format and into something that be run can anywhere.

How to Run it in GitHub Actions

To use this same process inside GitHub Actions, the only thing we need to do is adjust our GitHub Actions yaml to use Earthly for the build pipeline and we are all set.

  integration-test-elixir:
    runs-on: ubuntu-latest
    env:
      FORCE_COLOR: 1

    strategy:
      fail-fast: false
      matrix:
        include:
          - elixir: 1.11.1
            otp: 21.3.8.18
          - elixir: 1.11.1
            otp: 23.1.1
    steps:
      - uses: actions/checkout@v2
      - name: Download released earth
        run: "sudo /bin/sh -c 'wget https://github.com/earthly/earthly/releases/download/v0.4.1/earthly-linux-amd64 -O /usr/local/bin/earthly && chmod +x /usr/local/bin/earthly'"
      - name: Execute tests
        run: earthly -P --build-arg ELIXIR=${{ matrix.elixir }}  --build-arg OTP=${{ matrix.otp }} +integration-test

There we go, we are now able to run our build pipeline locally, without any complex environment setup. We can also run the same build process on our developer machine without needing to install anything except Earthly. This makes it easier for new contributors to approach the project.

The End Result

Eventually, with help from the Phoenix Team, I got this change approved and the Phoenix project now has an easy way to test and iterate on their build pipeline locally. And I didn't even write any Elixir code! You can find more details in the PR.

Thank you for reading this article. If you'd like to learn more about Earthly, you can find out a lot here. And if you'd like my help on your open source project's build, let me know.

In Software Development keeping up to date with technology updates is crucial. This is true both for developers as they learn and renew their skills, and also for the projects they work on and maintain.

When you start a project, you normally set it up with the latest stable versions of all libraries and tools.

Then time goes by, the project grows, and new features and libraries are added. But the versions of the libraries and packages remains the same the team never updates them.

After all, why would you update them if the project works perfectly with the current versions?

Why you should keep projects up to date

Here are some reasons why you should keep your dependencies updated:

• Solving problems from old versions.
• Adding vulnerability fixes.
• Increasing the overall performance.
• Adding new features.
• ...

When you keep your dependencies updated you are solving problems from older versions and improving performance with new optimizations. You are also able to use new features that other developers have added.

All of these improvements contribute to the maintainability of the code, and the overall project health.

We all have worked on projects where the dependencies have never (or rarely) been updated. And it's no fun.

So how, then, do we keep our projects up to date?

First of all, you can run npm outdated to see the latest releases of the packages you are currently using.

You can then run npm update to update them (it will not update them to the major versions). But how do you know which updates will break the project and which ones won't?

Then you have to think about when you should update everything. When should you check for updates – every day? every week? ...month?

What you'll learn in this tutorial

This is why I did this project: to learn about GitHub Actions and use it to have a safe way to automatically update dependencies without making the project fail.

In this tutorial you'll learn how to use the Renovate app to check for dependency updates and then submit Pull Requests to update them. This lets you abstract yourself away from checking for updates, so you can focus on more important things.

The point of using GitHub Actions is to set up a workflow and trigger it with every Pull Request. It will check that the build and tests pass with the updated dependencies before adding them to the project.

Table of Contents

• Getting Started
• Set up GitHub Actions Workflow
• Add Renovate
• Conclusion
• Useful Resources

Getting started

Although this approach can be applied to any project we will use a React project made with Create React App. This will give us a basic project with everything ready to work on.

By the way, if you do not have Node.js installed here is the link to do so.

If you want to check out the final result before you get started, here it is.

So let's begin by running

npx create-react-app my-app
cd my-app
npm start

If you use npm 5.1 or earlier, you can't use npx. Instead, install create-react-app globally:

npm install -g create-react-app

And then run:

create-react-app my-app

Set up Github Actions Workflow

Now we will proceed to define a GitHub Actions Workflow in our repository to automate the process.

GitHub Actions is a Github Feature that helps you automate your software development workflows. It can handle everything from simple tasks to custom end-to-end continuous integration (CI) and continuous deployment (CD) capabilities in your repositories.

In our root folder, we will create a new folder and name it .github. Inside that we'll create a workflows folder. This is how your project should look after these steps:

📁 my-app
├── 📁 .github
│   └── 📁 workflows
├── ...
...

Here is where we will create and add our Workflows. Github Actions Workflows are the Continuous Integration automated processes we want to run in our project.

Workflows are composed of jobs that contain a set of steps. To explain them in a clearer way let's create our own workflow and go through it step by step.

In the .github/workflows directory, add a .yml or .yaml file and name it main.yml. I chose that name to keep things simple, but you can give it any other name like build-test.yml or continuous-integration-workflow.yml.

📁 my-app
├── 📁 .github
│   └── 📁 workflows
│       └── 📄 main.yml
├── ...
...

Here is how the workflow will look in the end in case you just want to copy it and add it directly before the explanation.

name: Build and Test

on:
  push:
    branches: [master]
  pull_request:
    branches: [master]

jobs:
  build_and_test:
    runs-on: ubuntu-latest

    strategy:
      matrix:
        node: [10, 12]

    steps:
      - name: Checkout
        uses: actions/checkout@v2
      - name: Use Node.js ${{ matrix.node-version }}
        uses: actions/setup-node@v1
        with:
          node-version: ${{ matrix.node-version }}
      - name: Install project
        run: npm install
      - name: Build the project
        run: npm run build --if-present
      - name: Run tests
        run: npm test

The first param of our workflow will be its name.

name: Build and Test

The second param is the trigger.

We can choose if the workflow is triggered by an event like a push or pull request to a specific branch, or we can even schedule a cron to automatically trigger it every defined amount of time!.

In our project we will want to trigger it when pushing to the master branch, and when the Renovate app submits a Pull Request to update a dependency:

on:
  push:
    branches: [master]
  pull_request:
    branches: [master]

Next, we define the jobs.

In this example, there will only be one job: build and test the project, and chose the virtual machine where the job will be run.

jobs:
  build_and_test:
    runs-on: ubuntu-latest

Now comes the matrix where we will configure the combination of versions and systems we want to run our Workflow. In our case, we will run it on Node.js 10 and 12.

    strategy:
      matrix:
        node-version: [10, 12]

Finally, the Workflow's steps. First is the checkout action which is a standard action that you must include in your workflow when you need a copy of your repository to run the workflow.

Then you can run other actions and processes. In our app, we will use the setup-node action with the matrix we defined before. Then we will add steps to install the project, build it, and run the tests.

    steps:
      - name: Checkout
        uses: actions/checkout@v2
      - name: Use Node.js ${{ matrix.node-version }}
        uses: actions/setup-node@v1
        with:
          node-version: ${{ matrix.node-version }}
      - name: Install project
        run: npm install
      - name: Build the project
        run: npm run build --if-present
      - name: Run tests
        run: npm test

Now Create a GitHub Repository for the project, commit the local changes made, and push them to it.

Quick tip: if you want to create it faster, go to repo.new or github.new. You can use gist.new for gists too!

Once you push your changes the Workflow will run. Then you will be able to see how it went in the Actions tab from the GitHub Project.

Add Renovate

Renovate is a free, open-source, customizable app that helps you automatically update your dependencies in software projects by receiving pull requests.

It is used by software companies like Google, Mozilla, and Uber, and you can use it on GitHub, Gitlab, Bitbucket, Azure DevOps, and Gitea.

We will add a bot that will submit pull requests to our repository when there are updates in our project dependencies.

The cool thing, and the whole point of our project, is that we have previously defined in our workflow to run the tests with the pull requests. So when Renovate submits one, we will automatically check if the updates proposed will break the project or not before merging them to the master branch.

To add Renovate to our project we have to install its app into the project's repository.

Be careful when selecting the repository you want to add Renovate to and choose the one created before. If you made a mistake you want to reconfigure it, you can do it in the Personal Settings' Applications tab from your account.

After a few minutes you will have to accept and merge the onboarding Pull Request that you receive.

Once you have it integrated, you need to configure it by updating the renovate.json file on the project root. Remember to pull the changes after merging the Pull Request for it to appear.

You can use the default configuration where renovate will submit the pull requests whenever it finds updates and waits for you to merge them:

{
  "extends": ["config:base"]
}

Or you can adapt it to the requirements of your project like the one used by Renovate itself.

To avoid any issues, and to learn a little more about the tool, we will use a configuration with some of its most useful features.

If you want to learn more about its configuration here are the docs for it.

This will be our renovate.json file. Have a look at it, and I will explain it after.

{
  "extends": [
    "config:base"
  ],
  "packageRules": [
    {
      "updateTypes": [
        "minor",
        "patch"
      ],
      "automerge": true
    }
  ],
  "timezone": "Europe/Madrid",
  "schedule": [
    "after 10pm every weekday",
    "before 5am every weekday",
    "every weekend"
  ]
}

In the first part, we are telling renovate that our configuration will be an extension from the default one.

{
  "extends": [
    "config:base"
  ],

Then we have the packageRules. After some months using it I realized that going through checking the pull requests (from time to time) and accepting them if the tests passed was a major waste of time.

This is why the automerge is set to true, so Renovate automatically merges the pull request if the workflow passed successfully.

To restrict Renovate's freedom a bit, we define that it can only perform automerge when it is a minor or patch update.

This way, if it is a major or another kind of update, we will be the ones to check whether that update should be added or not.

Here you can find more information about the types of updates available.

  "packageRules": [
    {
      "updateTypes": [
        "minor",
        "patch"
      ],
      "automerge": true
    }
  ],

Lastly, we have the time schedule. If you work alone or in a team at certain hours it is nice to have updates done when you are not working to avoid unnecessary distractions.

We select our timezone and add a custom schedule for it. You can find the valid timezone names here.

  "timezone": "Europe/Madrid",
  "schedule": [
    "after 10pm every weekday",
    "before 5am every weekday",
    "every weekend"
  ],

Anyway, if you do not care about the time the pull requests will be submitted, or the people that contribute to the code are in different timezones, then you can remove this part.

Once we have updated the configuration we push the changes to GitHub to have the Renovate app adapted to the new configuration.

Now you finally have the project dependencies safely up-to-date without having to check for them. Here is the resulting project after following all the steps mentioned above.

Remember that if you added the time schedule part you will not get the pull request merged automatically until it complies with that configuration.

Conclusion

There are other ways to keep the dependencies updated in an automated way. But if you use GitHub to host your code, you should take advantage and make the most of its awesome free features.

If you are wondering what else you can do and automate with the GitHub apps and actions, just have a look at its Marketplace.

In addition, you can have a look at a project I made that I work on from time to time. It was the basis of this tutorial. It is a bit more complex and has more features than the one from this tutorial.

I hope you enjoyed this article and learned about GitHub Actions and its Apps. If you've got any questions, suggestions, or feedback in general, don't hesitate to reach out on any of the social networks from my site or by mail.

Useful Resources

Here is a collection of links and resources which I think can be useful to improve and learn more about GitHub Actions and Apps.

• Tutorial project. - The resulting project from this tutorial.
• GitHub Marketplace. - The place to find all GitHub Actions and Apps.
• GitHub Actions Workflow Configuration - The full documentation on how to set up a workflow on Github Actions.
• Renovate GitHub app - The Renovate App main page on the GitHub Marketplace.
• GitHub Actions project Workflow. - The Workflow used in this tutorial.
• Renovate App's configuration file. - Renovate App's custom configuration file from the tutorial.
• Up to Date React Template. - A Personal project that uses the approach described in this tutorial.

Developers are usually concerned with the quality of their code. There are different kinds of tests that help us avoid breaking code when a new feature is added in a project. But what can we do to ensure that components don't look different over time?

In this post, you will learn how to use Cypress to capture parts of pages of a website. After that, you will integrate the testing tool in CI to ensure that in the future no one will make unwanted changes to your project.

My motivation for creating this testing strategy came from work. At Thinkific we have an internal Design System and we added Cypress to avoid surprises when working in CSS/JS files.

By the end of this post we will have PRs with Cypress tests:

Before we start

I created a sample website to mimic a Component Library. It is a very simple website created with TailwindCSS and hosted in Vercel. It documents 2 components: badge and button.

You can check out the source code in GitHub. The website is static and it is inside the public folder. You can see the website locally by running npm run serve and checking in the browser http://localhost:8000.

Adding Cypress and Cypress Image Snapshot

Start by cloning the example repository. Next, create a new branch and install Cypress Image Snapshot, the package responsible for capturing/comparing screenshots.

git checkout -b add-cypress
npm install -D cypress cypress-image-snapshot

After adding the packages, a few extra steps are needed to add Cypress Image Snapshot in Cypress.

Create a cypress/plugins/index.js file with the following content:

const { addMatchImageSnapshotPlugin } = require('cypress-image-snapshot/plugin');

module.exports = (on, config) => {
  addMatchImageSnapshotPlugin(on, config);
};

Next, create a cypress/support/index.js file containing:

import { addMatchImageSnapshotCommand } from 'cypress-image-snapshot/command';

addMatchImageSnapshotCommand();

Creating the screenshot test

Time to create the screenshot test. Here is the plan:

1. Cypress will visit each page (badge and button) of the project.
2. Cypress will take a screenshot of each example in the page. The Badge page has 2 examples (Default and Pill), while the Button page has 3 examples (Default, Pill and Outline). All these examples are inside a <div> element with a cypress-wrapper. This class was added with the only intention being to identify what needs to be tested.

The first step is creating Cypress configuration file (cypress.json):

{
  "baseUrl": "http://localhost:8000/",
  "video": false
}

The baseUrl is the website running locally. As I mentioned before, npm run serve will serve the content of the public folder. The second option, video disables Cypress video recording, which we won't use in this project.

Time to create the test. In cypress/integration/screenshot.spec.js, add:

const routes = ['badge.html', 'button.html'];

describe('Component screenshot', () => {
  routes.forEach((route) => {
    const componentName = route.replace('.html', '');
    const testName = `${componentName} should match previous screenshot`;

    it(testName, () => {
      cy.visit(route);

      cy.get('.cypress-wrapper').each((element, index) => {
        const name = `${componentName}-${index}`;

        cy.wrap(element).matchImageSnapshot(name);
      });
    });
  });
});

In the code above, I am dynamically creating tests based in the routes array. The test will create one image per .cypress-wrapper element that the page has.

Last, inside the package.json let's create the command to trigger the tests:

{
  "test": "cypress"
}

From here, there are 2 options: run Cypress in headless mode with npm run cypress run or use the Cypress Test Runner with npm run cypress open.

Headless option

Using npm run cypress run, the output should be similar to the next image:

The tests will pass and 5 images will be created under the /snapshots/screenshot.spec.js folder.

Test Runner option

Using npm run cypress open, Cypress Test Runner will be opened and you can follow the tests step by step.

Our first milestone is done, so let's merge this branch to master. If you want to see the work done so far, jump in my Pull Request.

Using Cypress inside Docker

If you run the test above alternating between headless and Test Runner, you may notice that the screenshot will vary.

Using the Test Runner with a retina display computer, you may get retina images (2x), while the headless mode doesn't give you high-quality screenshots.

Also, it is important to say that the screenshots may vary according to your Operating System.

Linux and Windows, for instance, have apps with visible scrollbars, while macOS hides the scrollbar.

If the content captured in the screenshot doesn't fit a component, you may or may not have a scrollbar. If your project relies on OS default fonts, screenshots will also be different according to the environment.

In order to avoid these inconsistencies, tests will run inside Docker so the developer's computer won't affect the screenshot captures.

Let's start by creating a new branch:

git checkout -b add-docker

Cypress offers different Docker images - you can check out the details in their documentation and their blog.

For this example, I will use the cypress/included image, which includes Electron and is ready to be used.

We need to make two changes: change the baseUrl in the cypress.json file:

{
  "baseUrl": "http://host.docker.internal:8000/",
}

and the test command in the package.json file:

{
  "test": "docker run -it -e CYPRESS_updateSnapshots=$CYPRESS_updateSnapshots --ipc=host -v $PWD:/e2e -w /e2e cypress/included:4.11.0"
}

Running npm run test will bring us a problem:

The images are slightly different but why? Let's see what is inside the __diff_output__ folder:

As I mentioned earlier, typography inconsistencies! The Button component uses the OS default font. Since Docker is running inside Linux, the rendered font won't be the same that I have installed on macOS.

Since now we moved to Docker, these screenshots are outdated. Time to update the snapshots:

CYPRESS_updateSnapshots=true npm run test

Please notice that I am prefixing the test command with the environment variable CYPRESS_updateSnapshots.

The second milestone is done. In case you need help, check out my pull request.

Let's merge this branch and move forward.

Adding CI

Our next step is adding the tests in CI. There are different CI solutions in the market but for this tutorial, I will use Semaphore. I am not affiliated with them and I use their product at work, so it was for me a natural choice.

The configuration is straightforward and it can be adapted to other solutions like CircleCI or Github Actions.

Before we create our Semaphore configuration file, let's prepare our project to run in CI.

The first step is installing start-server-and-test. As the package name says, it will start a server, wait for the URL, and then run a test command:

npm install -D start-server-and-test

Second, edit the package.json file:

{
  "test": "docker run -it -e CYPRESS_baseUrl=$CYPRESS_baseUrl -e CYPRESS_updateSnapshots=$CYPRESS_updateSnapshots --ipc=host -v $PWD:/e2e -w /e2e cypress/included:4.11.0",
  "test:ci": "start-server-and-test serve http://localhost:8000 test"
}

In the test script, we are adding the CYPRESS_baseUrl environment variable. This will allow us to change the base URL used by Cypress dynamically. Also, we are adding the test:ci script, which will run the package we just installed.

We are ready for Semaphore. Create the .semaphore/semaphore.yml file with the following content:

 1 version: v1.0
 2 name: Cypress example
 3 agent:
 4   machine:
 5     type: e1-standard-2
 6     os_image: ubuntu1804
 7 blocks:
 8   - name: Build Dependencies
 9     dependencies: []
10     task:
11       jobs:
12         - name: NPM
13           commands:
14             - sem-version node 12
15             - checkout
16             - npm install
17   - name: Tests
18     dependencies: ['Build Dependencies']
19     task:
20       prologue:
21         commands:
22           - sem-version node 12
23           - checkout
24       jobs:
25         - name: Cypress
26           commands:
27             - export CYPRESS_baseUrl="http://$(ip route | grep -E '(default|docker0)' | grep -Eo '([0-9]+\.){3}[0-9]+' | tail -1):8000"
28             - npm run test:ci

Breaking the configuration down in detail:

• Lines 1-6 define which kind of instance we will use in their environment
• Lines 8 and 16 create 2 blocks: the first block, "Build Dependencies" will run npm install, downloading the dependencies we need. The second block, "Tests" will run Cypress, with a few differences.
• In line 27, we are dynamically setting the CYPRESS_baseUrl environment variable based on the IP Docker is using at the moment. This will replace http://host.docker.internal:8000/, which may not work in all environments.
• In line 28, we finally run the test using start-server-and-test: once the server is ready for connections, Cypress will run the test suite.

Another milestone is done, time to merge our branch! You can check out the Pull request that contains all the files from this section and check the build inside Semaphore.

Recording the tests in cypress.io

Reading the output of tests in CI is not very friendly. In this step, we will integrate our project with cypress.io.

The following steps are based on the Cypress documentation.

Let's start by getting a project ID and a record key. In the terminal, create a new branch and run:

git checkout -b add-cypress-recording
CYPRESS_baseUrl=http://localhost:8000 ./node_modules/.bin/cypress open

Earlier I mentioned that we would be using Cypress inside Docker. But here we are opening Cypress locally since this is the only way to integrate with the website dashboard.

Inside Cypress, let's go to the Runs tab, click in "Set up project to record", and choose a name and visibility. We will get a projectId that is automatically added in the cypress.json file and a private record key. Here is a video of the steps:

In Semaphore, I added the record key as an environment variable called CYPRESS_recordKey. Next let's update our test script to use the variable:

{
  "test:ci": "start-server-and-test 'serve' 8000 'npm run test -- run --record --key $CYPRESS_recordKey'"
}

That is pretty much all that needs to be done. In the Pull request we can see the cypress.io integration in the comments. There is even a deep link that takes us to their dashboard and shows all the screenshots. Check out the video below:

Time to merge our work, and that is the end of our integration.

Testing in real life

Imagine we are working on a change that affects the padding of the buttons: time to test if Cypress will capture the difference.

In the example website, let's double the horizontal padding from 16px to 32px. This change is quite simple since we are using Tailwind CSS: px-4 gets replaced by px-8. Here is that Pull request.

As we might expect, Cypress captured that the button doesn't match the screenshots. Visiting the page, we can check the screenshot of the broken test:

The diff file shows the original screenshot on the left, the current result on the right, and they are combined in the middle. We also have the option to download the image so we can see the issue better:

If this is not an issue, update the screenshots:

CYPRESS_updateSnapshots=true npm run test

The end

That's it for today. I hope you have learned how Cypress can be useful to ensure no one is adding unexpected changes to a project.

Also posted on my blog. If you like this content, follow me on Twitter and GitHub.

But automation can be tough and can sometimes prove to be costly. How can Github Actions help harden our code and give us more time to work on features instead of bugs?

• What are Github Actions?
• What is CI/CD?
• What are we going to build?
• Part 0: Setting up a project
• Part 1: Automating tests
• Part 2: Post new pull requests to Slack

What are Github Actions?

Actions are a relatively new feature to Github that allow you to set up CI/CD workflows using a configuration file right in your Github repo.

Previously, if you wanted to set up any kind of automation with tests, builds, or deployments, you would have to look to services like Circle CI and Travis or write your own scripts. But with Actions, you have first class support to powerful tooling to automate your workflow.

What is CI/CD?

CD/CD stands for Continuous Integration and Continuous Deployment (or can be Continuous Delivery). They're both practices in software development that allow teams to build projects together quickly, efficiently, and ideally with less errors.

Continuous Integration is the idea that as different members of the team work on code on different git branches, the code is merged to a single working branch which is then built and tested with automated workflows. This helps to constantly make sure everyone's code is working properly together and is well-tested.

Continuous Deployment takes this a step further and takes this automation to the deployment level. Where with the CI process, you automate the testing and the building, Continuous Deployment will automate deploying the project to an environment.

The idea is that the code, once through any building and testing processes, is in a deployable state, so it should be able to be deployed.

What are we going to build?

We're going to tackle two different workflows.

The first will be to simply run some automated tests that will prevent a pull request from being merged if it is failing. We won't walk through building the tests, but we'll walk through running tests that already exist.

In the second part, we'll set up a workflow that sends a message to slack with a link to a pull request whenever a new one is created. This can be super helpful when working on open source projects with a team and you need a way to keep track of requests.

Part 0: Setting up a project

For this guide, you can really work through any node-based project as long as it has tests you can run for Part 1.

If you'd like to follow along with a simpler example that I'll be using, I've set up a new project that you can clone with a single function that has two tests that are able to run and pass.

A function with two tests

If you'd like to check out this code to get started, you can run:

git clone --single-branch --branch start git@github.com:colbyfayock/my-github-actions.git

Once you have that cloned locally and have installed the dependencies, you should be able to run the tests and see them pass!

Passing tests

It should also be noted that you'll be required to have this project added as a new repository on Github in order to follow along.

Follow along with the commit!

Part 1: Automating tests

Tests are an important part of any project that allow us to make sure we're not breaking existing code while we work. While they're important, they're also easy to forget about.

We can remove the human nature out of the equation and automate running our tests to make sure we can't proceed without fixing what we broke.

Step 1: Creating a new action

The good news, is Github actually makes it really easy to get this workflow started as it comes as one of their pre-baked options.

We'll start by navigating to the Actions tab on our repository page.

Github Actions starting page

Once there, we'll immediately see some starter workflows that Github provides for us to dive in with. Since we're using a node project, we can go ahead and click Set up this workflow under the Node.js workflow.

Setting up a Node.js Github Action workflow

After the page loads, Github will land you on a new file editor that already has a bunch of configuration options added.

We're actually going to leave this "as is" for our first step. Optionally, you can change the name of the file to tests.yml or something you'll remember.

Adding a new Github Action workflow file

You can go ahead and click Start commit then either commit it directory to the master branch or add the change to a new branch. For this walkthrough, I'll be committing straight to master.

To see our new action run, we can again click on the Actions tab which will navigate us back to our new Actions dashboard.

Viewing Github Action workflow events

From there, you can click on Node.js CI and select the commit that you just made above and you'll land on our new action dashboard. You can then click one of the node versions in the sidebar via build (#.x), click the Run npm test dropdown, and we'll be able to see the output of our tests being run (which if you're following along with me, should pass!).

Viewing logs of a Github Action workflow

Follow along with the commit!

Step 2: Configuring our new action

So what did we just do above? We'll walk through the configuration file and what we can customize.

Github Action Node.js workflow file

Starting from the top, we specify our name:

name: Node.js CI

This can really be whatever you want. Whatever you pick should help you remember what it is. I'm going to customize this to "Tests" so I know exactly what's going on.

on:
  push:
    branches: [ master ]
  pull_request:
    branches: [ master ]

The on key is how we specify what events trigger our action. This can be a variety of things like based on time with cron. But here, we're saying that we want this action to run any time someone pushes commits to master or someone creates a pull request targeting the master branch. We're not going to make a change here.

jobs:
  build:
    runs-on: ubuntu-latest

This next bit creates a new job called build. Here we're saying that we want to use the latest version of Ubuntu to run our tests on. Ubuntu is common, so you'll only want to customize this if you want to run it on a specific environment.

    strategy:
      matrix:
        node-version: [10.x, 12.x, 14.x]

Inside of our job we specify a strategy matrix. This allows us to run the same tests on a few different variations.

In this instance, we're running the tests on 3 different versions of node to make sure it works on all of them. This is definitely helpful to make sure your code is flexible and future proof, but if you're building and running your code on a specific node version, you're safe to change this to only that version.

    steps:
    - uses: actions/checkout@v2
    - name: Use Node.js ${{ matrix.node-version }}
      uses: actions/setup-node@v1
      with:
        node-version: ${{ matrix.node-version }}
    - run: npm ci
    - run: npm run build --if-present
    - run: npm test

Finally, we specify the steps we want our job to run. Breaking this down:

• uses: actions/checkout@v2: In order for us to run our code, we need to have it available. This checks out our code on our job environment so we can use it to run tests.
• uses: actions/setup-node@v1: Since we're using node with our project, we'll need it set up on our environment. We're using this action to do that setup for us for each version we've specified in the matrix we configured above.
• run: npm ci: If you're not familiar with npm ci, it's similar to running npm install but uses the package-lock.json file without performing any patch upgrades. So essentially, this installs our dependencies.
• run: npm run build --if-present: npm run build runs the build script in our project. The --if-present flag performs what it sounds like and only runs this command if the build script is present. It doesn't hurt anything to leave this in as it won't run without the script, but feel free to remove this as we're not building the project here.
• run: npm test: Finally, we run npm test to run our tests. This uses the test npm script set up in our package.json file.

And with that, we've made a few tweaks, but our tests should run after we've committed those changes and pass like before!

Logs of passing tests in Github Action workflow

Follow along with the commit!

Step 3: Testing that our tests fail and prevent merges

Now that our tests are set up to automatically run, let's try to break it to see it work.

At this point, you can really do whatever you want to intentionally break the tests, but here's what I did:

Code diff - https://github.com/colbyfayock/my-github-actions/pull/1

I'm intentionally returning different expected output so that my tests will fail. And they do!

Failing status checks on pull request

In my new pull request, my new branch breaks the tests, so it tells me my checks have failed. If you noticed though, it's still green to merge, so how can we prevent merges?

We can prevent pull requests from being merged by setting up a Protected Branch in our project settings.

First, navigate to Settings, then Branches, and click Add rule.

Github branch protection rules

We'll then want to set the branch name pattern to *, which means all branches, check the Require status checks to pass before merging option, then select all of our different status checks that we'd like to require to pass before merging.

Setting up a branch protection rule in Github

Finally, hit Create at the bottom of the page.

And once you navigate back to the pull request, you'll notice that the messaging is a bit different and states that we need our statuses to pass before we can merge.

Failing tests preventing merge in pull request

Note: as an administrator of a repository, you'll still be able to merge, so this technically only prevents non-administrators from merging. But will give you increased messaging if the tests fail.

And with that, we have a new Github Action that runs our tests and prevents pull requests from merging if they fail.

Follow along with the pull request!

Note: we won't be merging that pull request before continuing to Part 2.

Part 2: Post new pull requests to Slack

Now that we're preventing merge requests if they're failing, we want to post a message to our Slack workspace whenever a new pull request is opened up. This will help us keep tabs on our repos right in Slack.

For this part of the guide, you'll need a Slack workspace that you have permissions to create a new developer app with and the ability to create a new channel for the bot user that will be associated with that app.

Step 1: Setting up Slack

There are a few things we're going to walk through as we set up Slack for our workflow:

• Create a new app for our workspace
• Assign our bot permissions
• Install our bot to our workspace
• Invite our new bot to our channel

To get started, we'll create a new app. Head over to the Slack API Apps dashboard. If you already haven't, log in to your Slack account with the Workspace you'd like to set this up with.

Creating a new Slack app

Now, click Create New App where you'll be prompted to put in a name and select a workspace you want this app to be created for. I'm going to call my app "Gitbot" as the name, but you can choose whatever makes sense for you. Then click Create App.

Configuring a new Slack app

Once created, navigate to the App Home link in the left sidebar. In order to use our bot, we need to assign it OAuth scopes so it has permissions to work in our channel, so select Review Scopes to Add on that page.

Reviewing Slack app scopes

Scroll own and you'll see a Scopes section and under that a Bot Token section. Here, click Add an OAuth Scope. For our bot, we don't need a ton of permissions, so add the channels:join and chat:write scopes and we should be good to go.

Adding scopes for a Slack app Bot Token

Now that we have our scopes, let's add our bot to our workspace. Scroll up on that same page to the top and you'll see a button that says Install App to Workspace.

Installing Slack app to a workspace

Once you click this, you'll be redirected to an authorization page. Here, you can see the scopes we selected for our bot. Next, click Allow.

Allowing permission for Slack app to be installed to workspace

At this point, our Slack bot is ready to go. At the top of the OAuth & Permissions page, you'll see a Bot User OAuth Access Token. This is what we'll use when setting up our workflow, so either copy and save this token or remember this location so you know how to find it later.

Note: this token is private - don't give this out, show it in a screencast, or let anyone see it!

Copying OAuth Access Token for Slack bot user

Finally, we need to invite our Slack bot to our channel. If you open up your workspace, you can either use an existing channel or create a new channel for these notifications, but you'll want to enter the command /invite @[botname] which will invite our bot to our channel.

Inviting Slack bot user to channel

And once added, we're done with setting up Slack!

Slack bot was added to channel

Create a Github Action to notify Slack

Our next step will be somewhat similar to when we created our first Github Action. We'll create a workflow file which we'll configure to send our notifications.

While we can use our code editors to do this by creating a file in the .github directory, I'm going to use the Github UI.

First, let's navigate back to our Actions tab in our repository. Once there, select New workflow.

Setting up a new Github Action workflow

This time, we're going to start the workflow manually instead of using a pre-made Action. Select set up a workflow yourself at the top.

Setting up a Github Action workflow manually

Once the new page loads, you'll be dropped in to a new template where we can start working. Here's what our new workflow will look like:

name: Slack Notifications

on:
  pull_request:
    branches: [ master ]

jobs:
  notifySlack:

    runs-on: ubuntu-latest

    steps:
    - name: Notify slack
      env:
        SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }}
      uses: abinoda/slack-action@master
      with:
        args: '{\"channel\":\"[Channel ID]\",\"blocks\":[{\"type\":\"section\",\"text\":{\"type\":\"mrkdwn\",\"text\":\"*Pull Request:* ${{ github.event.pull_request.title }}\"}},{\"type\":\"section\",\"text\":{\"type\":\"mrkdwn\",\"text\":\"*Who?:* ${{ github.event.pull_request.user.login }}\n*Request State:* ${{ github.event.pull_request.state }}\"}},{\"type\":\"section\",\"text\":{\"type\":\"mrkdwn\",\"text\":\"<${{ github.event.pull_request.html_url }}|View Pull Request>\"}}]}'

So what's happening in the above?

• name: we're setting a friendly name for our workflow
• on: we want our workflow to trigger when there's a pull request is created that targets our master branch
• jobs: we're creating a new job called notifySlack
• jobs.notifySlack.runs-on: we want our job to run on a basic setup of the latest Unbuntu
• jobs.notifySlack.steps: we really only have one step here - we're using a pre-existing Github Action called Slack Action and we're configuring it to publish a notification to our Slack

There are two points here we'll need to pay attention to, the env.SLACK_BOT_TOKEN and the with.args.

In order for Github to communicate with Slack, we'll need a token. This is what we're setting in env.SLACK_BOT_TOKEN. We generated this token in the first step. Now that we'll be using this in our workflow configuration, we'll need to add it as a Git Secret in our project.

_Github secrets including SLACK_BOTTOKEN

The with.args property is what we use to configure the payload to the Slack API that includes the channel ID (channel) and our actual message (blocks).

The payload in the arguments is stringified and escaped. For example, when expanded it looks like this:

{
  "channel": "[Channel ID]",
  "blocks": [{
    "type": "section",
    "text": {
      "type": "mrkdwn",
      "text": "*Pull Request:* ${{ github.event.pull_request.title }}"
    }
  }, {
    "type": "section",
    "text": {
      "type": "mrkdwn",
      "text": "*Who?:*n${{ github.event.pull_request.user.login }}n*State:*n${{ github.event.pull_request.state }}"
    }
  }, {
    "type": "section",
    "text": {
      "type": "mrkdwn",
      "text": "<${{ github.event.pull_request._links.html.href }}|View Pull Request>"
    }
  }]
}

Note: this is just to show what the content looks like, we need to use the original file with the stringified and escaped argument.

Back to our configuration file, the first thing we set is our channel ID. To find our channel ID, you'll need to use the Slack web interface. Once you open Slack in your browser, you want to find your channel ID in the URL:

https://app.slack.com/client/[workspace ID]/[channel ID]

Channel ID in Slack web app URL

With that channel ID, you can modify our workflow configuration and replace [Channel ID] with that ID:

with:
  args: '{\"channel\":\"C014RMKG6H2\",...

The rest of the arguments property is how we set up our message. It includes variables from the Github event that we use to customize our message.

We won't go into tweaking that here, as what we already have will send a basic pull request message, but you can test out and build your own payload with Slack's Block Kit Builder.

Follow along with the commit!

Test out our Slack workflow

So now we have our workflow configured with our Slack app, finally we're ready to use our bot!

For this part, all we need to do is create a new pull request with any change we want. To test this out, I simply created a new branch where I added a sentence to the README.md file.

Code diff - https://github.com/colbyfayock/my-github-actions/pull/2

Once you create that pull request, similar to our tests workflow, Github will run our Slack workflow! You can see this running in the Actions tab just like before.

As long as you set everything up correctly, once the workflow runs, you should now have a new message in Slack from your new bot.

Slack bot automated message about new pull request

Note: we won't be merging that pull request in.

What else can we do?

Customize your Slack notifications

The message I put together is simple. It tells us who created the pull request and gives us a link to it.

To customize the formatting and messaging, you can use the Github Block Kit Builder to create your own.

If you'd like to include additional details like the variables I used for the pull request, you can make use of Github's available contexts. This lets you pull information about the environment and the job to customize your message.

I couldn't seem to find any sample payloads, so here's an example of a sample github context payload you would expect in the event.

Sample github context

More Github actions

With our ability to create new custom workflows, that's not a lot we can't automate. Github even has a marketplace where you can browse around for one.

If you're feeling like taking it a step further, you can even create your own! This lets you set up scripts to configure a workflow to perform whatever tasks you need for your project.

Join in the conversation!

What do you use Github actions for?

Share with me on Twitter!

I recently published a package on npm: a data structures and algorithms library implemented in JavaScript.

The purpose of the project is to help others learn and understand data structures and algorithms from a JavaScript perspective.

Rather than containing only snippets of code with accompanying explanations, the project is meant to provide an eager learner with fully working code, good test cases, and a playground full of examples.

If you’re interested, the project can be found on npm here.

But, rather than talking about the project itself, what I want to write about today are all the neat tools I learned about and used while creating the project.

I’ve worked on tons of side projects and demos over the last six years, but each of them are very visibly just "pet projects". They in no way have the qualities that’d make them look professional or production-ready.

What I set out to create was something that could be considered a respectable open-source package. To do that, I decided my project would need proper documentation, tooling, linting, continuous integration, and unit tests.

Below are some of the tools I used. Each one serves a unique purpose. I’ve linked to the documentation for each package so you, too, can start utilizing these tools in projects of you own.

Note: This article assumes that you are already familiar with the process of creating a simple JavaScript package and publishing it on npm.

If not, the npm team has some great documentation on getting started that will walk you through the initialization of a project and the steps for publishing.

So let's get started.

Prettier

Prettier is an opinionated code formatter that automatically formats your code for you. Rather than simply using ESLint to enforce whatever formatting standards your team has agreed on, Prettier can take care of the formatting for you.

No more worrying about fixing your indentation and line widths! I’m using this specifically for my JavaScript, but it can handle many different languages.

Sample JavaScript before and after running Prettier

You can check out the Prettier docs here: https://github.com/prettier/prettier

stylelint

stylelint autoformats your CSS for you. Similar to Prettier, this tool helps you keep your CSS clean while taking care of the heavy lifting for you.

Sample output from running stylelint

You can check out the stylelint docs here: https://github.com/stylelint/stylelint

ESLint

ESLint handles all my other JavaScript linting for catching syntax errors and enforcing best practices.

Sample output from linting with ESLint in their playground environment

You can check out the ESLint docs here: https://eslint.org/

Commitizen

Commitizen is a CLI tool that walks you through writing your commit messages. It generates the commit message for you based on your input and ensures that the resulting commit message follows the Conventional Commits standard.

Commitizen command line interface when creating a new commit

You can check out the Commitizen docs here: https://github.com/commitizen/cz-cli

commitlint

commitlint verifies that your commit messages follow the Conventional Commits standard. As long as you use Commitizen to create your commit messages, you won’t run into any problems.

The real benefit of using commitlint is to catch commits that developers wrote on their own that don’t follow your formatting standards.

commitlint demo to show possible error messages

You can check out the commitlint docs here: https://github.com/conventional-changelog/commitlint

lint-staged

lint-staged runs linters against code that you’re trying to commit. This is where you can validate that your code is passing the standards being enforced by Prettier, stylelint, and ESLint.

lint-staged example that runs ESLint on checked-in code

You can check out the lint-staged docs here: https://github.com/okonet/lint-staged

Husky

Husky makes it easy to run Git hooks.

All the previously mentioned tools can be run through Husky on Git hooks like pre-commit or commit-msg, so this is where the magic happens.

For instance, I’m running lint-staged and my unit tests during the pre-commit hook, and I’m running commitlint during the commit-msg hook. That means when I’m trying to check in my code, Husky does all the validation to make sure I’m abiding by all the rules I’m enforcing in my project.

Sample Husky configuration that runs on the pre-commit and commit-msg Git hooks

You can check out the Husky docs here: https://github.com/typicode/husky

Rollup

Rollup is a module bundler for JavaScript. It takes all of your source code and bundles it into the files you actually want to distribute as part of your package.

The conventional wisdom seems to be if you’re building a web application, you should use webpack. And if you’re building a library, you should use Rollup.

In my case, I was building a data structures and algorithms library, so I chose to use Rollup. One benefit seems to be the output that Rollup generates is significantly smaller than what webpack outputs.

A very minimal Rollup config that creates an output bundle in the CommonJS format

You can check out the Rollup docs here: https://rollupjs.org/guide/en/

Standard Version

Standard Version helps automate your versioning and changelog generation.

Previously, I mentioned tools like Commitizen and commitlint for formatting your commits according to the Conventional Commits standard. Why, you may ask, is that helpful?

The answer, at least in part, is that by using a consistent commit message format, you can use tools that are able to understand what kind of changes your commits are making.

For example, are you fixing bugs? Adding new features? Making breaking changes people consuming your library should be aware of? Standard Version is able to understand your commit messages and then generate a changelog for you.

It’s also able to intelligently bump the version of your package according to the semantic versioning standard (major, minor, patch).

Sample Standard Version pre-release script that runs before version bumps

You can check out the Standard Version docs here: https://github.com/conventional-changelog/standard-version

Travis CI

Travis CI is a continuous-integration (CI) tool that can be integrated with GitHub, where my code happens to be hosted.

CI tools are important because they allow your commits to be tested yet again before you merge them into your master branch. You could argue using Travis CI and a tool like Husky duplicates functionality, but it’s important to keep in mind that even Husky can be bypassed by passing a --no-verify flag to your commit command.

Through GitHub, you can specify that your Travis CI jobs must be passing before code can be merged, so this adds one more layer of protection and verifies that only passing code makes it into your repo.

Travis CI output from a passing build

You can check out the Travis CI docs here: https://docs.travis-ci.com/

Codecov

Codecov is another CI tool that looks at your project’s code coverage.

I’m writing JavaScript unit tests using Jest. Part of my Travis CI job runs my test suite and ensures they all pass. It also pipes the code coverage output to Codecov, which then can verify if my code coverage is slipping or staying high. It also can be used in conjunction with GitHub badges, which we’ll talk about next.

Codecov dashboard (look at that beautiful 100% code coverage!)

You can check out the Codecov docs here: https://docs.codecov.io/docs

Badges

Have you ever looked at a project in GitHub and seen little badges near the top of the README? Things like whether the build is passing, what the code coverage is, and what the latest version of the npm package is can all be shown using badges.

They’re relatively simple to add, but I think they add a really nice touch to any project. Shields.io is a great resource for finding lots of different badges that can be added to your project, and it helps you generate the markdown to include in your README.

GitHub badges for my js-data-structures-and-algorithms npm package

You can check out the Shields.io docs here: https://shields.io/

Documentation

A little documentation goes a long way. In my project, I’ve added a README, CHANGELOG, contributing guidelines, code of conduct, and a license.

These docs serve to help people know what your project is, how to use it, what changes have been made with each release, how to contribute if they want to get involved, how they’re expected to interact with other members of the community, and what the legal terms are.

The CHANGELOG for my js-data-structures-and-algorithms npm package

You can check out the documentation for my project here: https://github.com/thawkin3/js-data-structures-and-algorithms

GitHub Templates

Did you know you can create templates in GitHub for things like bug reports, feature requests, and pull requests? Creating these templates makes it crystal clear, for example, what information someone should be expected to provide when filing a bug.

GitHub templates for bug reports and feature requests

You can check out the GitHub templates docs here: https://help.github.com/en/github/building-a-strong-community/about-issue-and-pull-request-templates

Closing

That’s it. When I first showed this project to some friends, one of them commented, “Oh my build tool soup!” And he may be right. This is a lot. But I strongly believe that adding all the tooling above is worth it. It helps automate many things and helps keep your codebase clean and in working order.

My biggest takeaway from building this project is that setting up all of the tooling above isn’t as daunting as it may seem. Each of these tools has good documentation and helpful guides for getting started. It really wasn’t that bad, and you should feel confident adopting some (if not all) of these tools in your project, too.

Happy coding!