These terms might seem complex or interchangeable at first glance, leaving you wondering: What do they actually mean? How do they differ from one another? 🤔

In this handbook, I’ll break down these concepts in a clear and approachable way, drawing on relatable analogies to make each term easier to understand. 🧠💡 Beyond just theory, we’ll dive into a hands-on tutorial where you’ll learn how to set up a CI/CD workflow step by step.

Together, we’ll:

• Set up a Node.js project. ✨

• Implement automated tests using Jest and Supertest. 🛠️

• Set up a CI/CD workflow using GitHub Actions, triggered on push, and pull requests, or after a new release. ⚙️

• Build and publish a Docker image of your application to Docker Hub. 📦

• Deploy your application to a staging environment for testing. 🚀

• Finally, roll it out to a production environment, making it live! 🌐

By the end of this guide, not only will you understand the difference between CI/CD concepts, but you’ll also have practical experience in building your own automated pipeline. 😃

Table of Contents

1. What is Continuous Integration, Deployment, and Delivery?

2. Differences Between Continuous Integration, Continuous Delivery, and Continuous Deployment

3. How to Set Up a Node.js Project with a Web Server and Automated Tests

4. How to Create a GitHub Repository to Host Your Codebase

5. How to Set Up the CI and CD Workflows Within Your Project

6. Set Up a Docker Hub Repository for the Project's Image and Generate an Access Token for Publishing the Image

7. Create a Google Cloud Account, Project, and Billing Account

8. Create a Google Cloud Service Account to Enable Deployment of the Node.js Application to Google Cloud Run via the CD Pipeline

9. Create the Staging Branch and Merge the Feature Branch into It (Continuous Integration and Continuous Delivery)

10. Merge the Staging Branch into the Main Branch (Continuous Integration and Continuous Deployment)

11. Conclusion

What is Continuous Integration, Deployment, and Delivery? 🤔

Continuous Integration (CI)

Imagine you’re part of a team of six developers, all working on the same project. Without a proper system, chaos would ensue.

Let’s say Mr. A is building a new login feature, Mrs. B is fixing a bug in the search bar, and Mr. C is tweaking the dashboard UI—all at the same time. If everyone is editing the same "folder" or codebase directly, things could go horribly wrong: "Hey! Who just broke the app?!" 😱

To keep everything in order, teams use Version Control Systems (VCS) like GitHub, GitLab, or BitBucket. Think of it as a digital workspace where everyone can safely collaborate without stepping on each other’s toes. 🗂️✨

Here’s how Continuous Integration fits into this process step-by-step:

1. The Main Branch: The General Folder ✨

At the heart of every project is the main branch—the ultimate source of truth. It contains the stable codebase that powers your live app. It’s where every team member contributes their work, but with one important rule: only tested and approved code gets merged here. 🚀

2. Feature Branches: Personal Workspaces 🔨

When someone like Mr. A wants to work on a new feature, they create a feature branch. This branch is essentially a personal copy of the main branch where they can tinker, write code, and test without affecting others. Mrs. B and Mr. C are also working on their own branches. Everyone’s experiments stay neatly organized. 🧪💡

3. Merging Changes: The CI Workflow 🎉

When Mr. A is satisfied with his feature, he doesn’t just shove it into the main branch—CI ensures it’s done safely:

• Automated Tests: Before merging, CI tools automatically run tests on Mr. A’s code to check for bugs or errors. Think of it as a bouncer guarding the main branch, ensuring no bad code gets in. 🕵️‍♂️

• Build Verification: The feature branch code is also "built" (converted into a deployable version of the app) to confirm it works as intended.

Once these checks are passed, Mr. A’s feature branch is merged into the main branch. This frequent merging of changes is what we call Continuous Integration.

Continuous Delivery (CD)

Continuous Delivery (CD) often gets mixed up with Continuous Deployment, and while they share similarities, they serve distinct purposes in the development lifecycle. Let’s break it down! 🧐

The Need for a Staging Area 🌉

In the Continuous Integration (CI) process we discussed above, we primarily dealt with feature branches and the main branch. But directly merging changes from feature branches into the main branch (which powers the live product) can be risky. Why? 🛑

While automated tests and builds catch many errors, they’re not foolproof. Some edge cases or bugs might slip through unnoticed. This is where the staging branch and staging environment come into play! 🎭

Think of the staging branch as a “trial run.” Before unleashing changes to real customers, the codebase from feature branches is merged into the staging branch and deployed to a staging environment. This environment is an exact replica of the production environment, but it’s used exclusively by the Quality Assurance (QA) team for testing.

The QA team takes the role of a “test driver,” running the platform through its paces just as a real user would. They check for usability issues, edge cases, or bugs that automated tests might miss, and provide feedback to developers for fixes. 🚦 If everything passes, the codebase is cleared for deployment to production.

Continuous Delivery in Action 📦

The process of merging changes into the staging branch and deploying them to the staging environment is what we call Continuous Delivery. 🛠️ It ensures that the application is always in a deployable state, ready for the next step in the pipeline.

Unlike Continuous Deployment (which we’ll discuss later), Continuous Delivery doesn’t automatically push changes to production (live platform). Instead, it pauses to let humans—namely the QA team or stakeholders—decide when to proceed. This adds an extra layer of quality assurance, reducing the chances of errors making it to the live product. 🕵️‍♂️

Continuous Deployment (CD)

Continuous Deployment (CD) takes automation to its peak. While it shares similarities with Continuous Delivery, the key difference lies in the final step: there’s no manual approval required. The final process—merging the codebase and deploying it live for end users (the QA testers or the team lead could do this).

Let’s explore what makes Continuous Deployment so powerful (and a little scary)! 😅

The Last Mile of the CI/CD Pipeline 🛣️

Imagine you’ve gone through the rigorous process of Continuous Integration: teammates have merged their feature branches, automated tests were run, and the codebase was successfully deployed to the staging environment during Continuous Delivery.

Now, you’re confident that the application is free of bugs and ready to shine in the production environment—the live version of your platform used by real customers.

In Continuous Deployment, this final step of deploying changes to the live environment happens automatically. The pipeline triggers whenever specific events occur, such as:

• A Pull Request (PR) is merged into the main branch.

• A new release version is created.

• A commit is pushed directly to the production branch (though this is rare for most teams).

Once triggered, the pipeline springs into action, building, testing, and finally deploying the updated codebase to the production environment. 📡

Differences Between Continuous Integration, Continuous Delivery, and Continuous Deployment 🔍

Now that we’ve untangled the mysteries of Continuous Integration, Continuous Delivery, and Continuous Deployment, it’s time to roll up our sleeves and put theory into practice 😁.

How to Set Up a Node.js Project with a Web Server and Automated Tests ✨

In this hands-on section, we’ll build a Node.js web server with automated tests using Jest. From there, we’ll create a CI/CD pipeline with GitHub Actions that automates testing for every pull request to the staging and main branches. Finally, we’ll publish an Image of our application to DockerHub and deploy the image to Google Cloud Run, first to a staging environment for testing and later to the production environment for live use.

Ready to bring your project to life? Let’s get started! 🚀✨

Step 1: Install Node.js 📥

To get started, you’ll need to have Node.js installed on your machine. Node.js provides the JavaScript runtime we’ll use to create our web server.

1. Visit https://nodejs.org/en/download/package-manager

2. Choose your operating system (Windows, macOS, or Linux) and download the installer.

3. Follow the installation instructions to complete the setup.

To verify that Node.js was installed successfully, open your terminal and run node -v. This should display the installed version of Node.js

Step 2: Clone the Starter Repository 📂

The next step is to grab the starter code from GitHub. If you don’t have Git installed, you can download it at https://git-scm.com/downloads. Choose your OS and follow the instructions to install Git. Once you’re set, it’s time to clone the repository.

Run the following command in your terminal to clone the boilerplate code:

git clone --single-branch --branch initial https://github.com/onukwilip/ci-cd-tutorial

This will download the project files from the initial branch, which contains the starter template for our Node.js web server.

Navigate into the project directory:

cd ci-cd-tutorial

Step 3: Install Dependencies 📦

Once you’re in the project directory, install the required dependencies for the Node.js project. These are the packages that power the application:

npm install --force

This will download and set up all the libraries specified in the project. Alright, dependencies installed? You’re one step closer!

Step 4: Run Automated Tests ✅

Before diving into the code, let’s confirm that the automated tests are functioning correctly. Run:

npm test

You should see two successful test results in your terminal. This indicates that the starter project is correctly configured with working automated tests.

Step 5: Start the Web Server 🌐

Finally, let’s start the web server and see it in action. Run the following command:

npm start

Wait for the application to start running. Open your browser and visit http://localhost:5000. 🎉 You should see the starter web server up and running, ready for your CI/CD magic:

How to Create a GitHub Repository to Host Your Codebase 📂

Step 1: Sign In to GitHub

1. Go to GitHub: Open your browser and visit GitHub - https://github.com.

2. Sign In: Click on the Sign In button in the top-right corner and enter your username and password to log in, OR create an account if you don’t have one by clicking the Sign up button.

Step 2: Create a New Repository

Once you're signed in, on the main GitHub page, you’ll see a "+" sign in the top-right corner next to your profile picture. Click on it, and select “New repository” from the dropdown.

Now it’s time to set the repository details. You’ll include:

• Repository Name: Choose a name for your repository. For example, you can call it ci-cd-tutorial.

• Description (Optional): You can add a short description, like “A tutorial project for CI/CD with Docker and GitHub Actions.”

• Visibility: Choose whether you want your repository to be public (accessible by anyone) or private (only accessible by you and those you invite). For the sake of this tutorial, make it public.

• Do Not Check the Add a README File Box: Important: Make sure you do not check the option to Add a README file. This will automatically create a README.md file in your repository, which could cause conflicts later when you push your local files. We'll add the README file manually if needed later.

After filling out the details, click on “Create repository”.

Step 3: Change the Remote Destination and Push to Your New Repository

Update the Remote Repository URL:

Since you've already cloned the codebase from my repository, you need to update the remote destination to point to your newly created GitHub repository.

Copy your repository URL (the URL of the page you were redirected to after creating the repository). It should look similar to this: https://github.com/<username>/<repo-name>.

Open your terminal in the project directory and run the following commands:

git remote set-url origin <your-repo-url>

Replace <your-repo-url> with your GitHub repository URL which you copied earlier.

Rename the Current Branch to main:

If your branch is named something other than main, you can rename it to main using:

git branch -M main

Push to Your New Repository:

Finally, commit any changes you’ve made and push your local repository to the new remote GitHub repository by running:

git add .
git commit -m 'Created boilerplate'
git push -u origin main

Now your local codebase is linked to your new GitHub repository, and the files are successfully pushed there. You can verify by visiting your repository on GitHub.

How to Set Up the CI and CD Workflows Within Your Project ⚙️

Now it’s time to create the CI and CD workflows for our project! These workflows won’t run on your local PC but will be automatically triggered and executed in the cloud once you push your changes to the remote repository. GitHub Actions will detect these workflows and run them based on the triggers you define.

Step 1: Prepare the Workflow Directory 📂

Before adding the CI/CD pipelines, it's a good practice to first create a feature branch. This step mirrors the workflow commonly used in teams, where new features or changes are made in separate branches before they are merged into the main codebase.

To create and switch to a new branch, run the following command:

git checkout -b feature/ci-cd-pipeline

This will create a new branch called feature/ci-cd-pipeline and switch to it. Now, you can safely add and test the CI/CD workflows without affecting the main branch.

Once you finish, you’ll be able to merge this feature branch back into main or staging as part of the pull request process.

In the project’s root directory, create a folder named .github. Inside .github, create another folder called workflows.

Any YAML file placed in the .github/workflows directory is automatically recognized as a GitHub Actions workflow. These workflows will execute based on specific triggers, such as pull requests, pushes, or releases.

Step 2: Create the Continuous Integration Workflow 🚀

We’ll now create a CI workflow that automatically tests the application whenever a pull request is made to the main or staging branches.

First, inside the workflows directory, create a file named ci-pipeline.yml.

Paste the following code into the file:

name: CI Pipeline to staging/production environment
on:
  pull_request:
    branches:
      - staging
      - main
jobs:
  test:
    runs-on: ubuntu-latest
    name: Setup, test, and build project
    env:
      PORT: 5001
    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Install dependencies
        run: npm ci

      - name: Test application
        run: npm test

      - name: Build application
        run: |
          echo "Run command to build the application if present"
          npm run build --if-present

Explanation of the CI Workflow

Here’s a breakdown of each section in the workflow:

1. name: CI Pipeline to staging/production environment: This is the title of your workflow. It helps you identify this pipeline in GitHub Actions.

2. on: The on parameter is what determines the events that trigger your workflow. When the workflow YAML file is pushed to the remote GitHub repository, GitHub Actions automatically registers the workflow using the configured triggers in the on field. These triggers act as event listeners that tell GitHub when to execute the workflow

   For example:

   If we set pull_request as the value for the on parameter and specify the branches we want to monitor using the branches key, GitHub sets up event listeners for pull requests to those branches.

    on:
      pull_request:
        branches:
          - main
          - staging
   

   This configuration means that GitHub will trigger the workflow whenever a pull request is made to the main or staging branches.

   Multiple Triggers:
   You can define multiple event listeners in the on parameter. For instance, in addition to pull requests, you can add a listener for push events.

    on:
      pull_request:
        branches:
          - main
          - staging
      push:
        branches:
          - main
   

   This configuration ensures that the workflow is triggered when:

   • A pull request is made to either the main or staging branch.

   • A push is made directly to the main branch.

📘 Learn more about triggers: Check out the official GitHub documentation here.

3. jobs: The jobs section outlines the specific tasks (or jobs) that the workflow will execute. Each job is an independent unit of work that runs on a separate virtual machine (VM). This isolation ensures a clean, unique environment for every job, avoiding potential conflicts between tasks.

   Key Points About Jobs:

   1. Clean VM for Each Job: When GitHub Actions runs a workflow, it assigns a dedicated VM instance to each job. This means the environment is reset for every job, ensuring there’s no overlap or interference between tasks.

   2. Multiple Jobs: Workflows can have multiple jobs, each responsible for a specific task. For example:

      • A Test job to install dependencies and run automated tests.

      • A Build job to compile the application.

   3. Job Organization: Jobs can be organized to run:

      • Sequentially: Ensures one job is completed before the next starts, for example the Test job must finish before the Build job. This sequential flow mimics the "pipeline" structure.

      • Simultaneously: Multiple jobs can run in parallel to save time, especially if the jobs are independent of one another.

   4. Single Job in This Workflow: In our current workflow, there is only one job, test, which:

      • Installs dependencies.

      • Runs automated tests.

      • Builds the application.

📘 Learn more about jobs: Dive into the GitHub Actions jobs documentation here.

4. runs-on: ubuntu-latest: Specifies the operating system the job will run on. GitHub provides pre-configured virtual environments, and we’re using the latest Ubuntu image.

5. env: Sets environment variables for the job. Here, we define the PORT variable used by our application.

6. Steps: Steps define the individual actions to execute within a job:

   • Checkout: Uses the actions/checkout action to clone the repository containing the codebase in the feature branch into the virtual machine instance environment. This step ensures the pipeline has access to the project files.

   • Install dependencies: Runs npm ci to install the required Node.js packages.

   • Test application: Runs the automated tests using the npm test command. This validates the codebase for errors or failing test cases.

   • Build application: Builds the application if a build script is defined in the package.json. The --if-present flag ensures this step doesn’t fail if no build script is present.

Now that we’ve completed the CI pipeline, which runs on pull requests to the main or staging branches, let’s move on to setting up the Continuous Delivery (CD) and Continuous Deployment pipelines. 🚀

Step 3: The Continuous Delivery and Deployment Workflow

First, create the Pipeline File:
In the .github/workflows folder, create a new file called cd-pipeline.yml. This file will define the workflows for automating delivery and deployment.

Next, paste the configuration:
Copy and paste the following configuration into the cd-pipeline.yml file:

name: CD Pipeline to Google Cloud Run (staging and production)
on:
  push:
    branches:
      - staging
  workflow_dispatch: {}
  release:
    types: published

env:
  PORT: 5001
  IMAGE: ${{vars.IMAGE}}:${{github.sha}}
jobs:
  test:
    runs-on: ubuntu-latest
    name: Setup, test, and build project
    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Install dependencies
        run: npm ci

      - name: Test application
        run: npm test
  build:
    needs: test
    runs-on: ubuntu-latest
    name: Setup project, Authorize GitHub Actions to GCP and Docker Hub, and deploy
    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Authenticate for GCP
        id: gcp-auth
        uses: google-github-actions/auth@v0
        with:
          credentials_json: ${{ secrets.GCP_SERVICE_ACCOUNT }}

      - name: Set up Cloud SDK
        uses: google-github-actions/setup-gcloud@v0

      - name: Authenticate for Docker Hub
        id: docker-auth
        env:
          D_USER: ${{secrets.DOCKER_USER}}
          D_PASS: ${{secrets.DOCKER_PASSWORD}}
        run: |
          docker login -u $D_USER -p $D_PASS
      - name: Build and tag Image
        run: |
          docker build -t ${{env.IMAGE}} .
      - name: Push the image to Docker hub
        run: |
          docker push ${{env.IMAGE}}
      - name: Enable the Billing API
        run: |
          gcloud services enable cloudbilling.googleapis.com --project=${{secrets.GCP_PROJECT_ID}}
      - name: Deploy to GCP Run - Production environment (If a new release was published from the master branch)
        if: github.event_name == 'release' && github.event.action == 'published' && github.event.release.target_commitish == 'main'
        run: |
          gcloud run deploy ${{vars.GCR_PROJECT_NAME}} \
          --region ${{vars.GCR_REGION}} \
          --image ${{env.IMAGE}} \
          --platform "managed" \
          --allow-unauthenticated \
          --tag production \
      - name: Deploy to GCP Run - Staging environment
        if: github.ref != 'refs/heads/main'
        run: |
          echo "Deploying to staging environment"
          # Deploy service with to staging environment
          gcloud run deploy ${{vars.GCR_STAGING_PROJECT_NAME}} \
          --region ${{vars.GCR_REGION}} \
          --image ${{env.IMAGE}} \
          --platform "managed" \
          --allow-unauthenticated \
          --tag staging \

The CD pipeline configuration combines Continuous Delivery and Continuous Deployment workflows into a single file for simplicity. It builds on the concepts of CI/CD we discussed earlier, automating testing, building, and deploying the application to Google Cloud Run.

Explanation of the CD pipeline:

1. Workflow Triggers (on)

• push: Workflow triggers on pushes to the staging branch.

• workflow_dispatch: Enables manual execution of the workflow via the GitHub Actions interface.

• release: Triggers when a new release is published.
  Example: When a release is published from the main branch, the app deploys to the production environment.

2. Job 1 – Testing the Codebase: The first job in the pipeline, Test, ensures the codebase is functional and error-free before proceeding with delivery or deployment

3. Job 2 – Building and Deploying the Application: Aha! Moment ✨: These jobs run sequentially. 😃 The Build job begins only after the Test job is completed successfully. It prepares the application for deployment and manages the actual deployment process.

   Here's what happens:

   • Authorization for GCP and Docker Hub: The workflow authenticates with both Google Cloud Platform (GCP) and Docker Hub. For GCP, it uses the google-github-actions/auth@v0 action to handle service account credentials stored as secrets. Similarly, it logs into Docker Hub with stored credentials to enable image uploads.

   • Build and Push Docker Image: The application is built into a Docker image and tagged with a unique identifier (${{env.IMAGE}}). This image is then pushed to Docker Hub, making it accessible for deployment.

   • Deploy to Google Cloud Run: Based on the event that triggered the workflow, the application is deployed to either the staging or production environment in Google Cloud Run. A push to the staging branch deploys to the staging environment (Continuous Delivery), while a release from the main branch deploys to production (Continuous Deployment).

To ensure the security and flexibility of our pipeline, we rely on external variables and secrets rather than hardcoding sensitive information directly into the workflow file.

Why? Workflow configuration files are part of your repository and accessible to anyone with access to the codebase. If sensitive data, like API keys or passwords, is exposed here, it can be easily compromised. 😨

Instead, we use GitHub’s Secrets to securely store and access this information. Secrets allow us to define variables that are encrypted and only accessible by our workflows. For example:

• DockerHub Credentials: We’ll add a Docker username and access token to the repository’s secrets. These are essential for authenticating with DockerHub to upload the built Docker images.

• Google Cloud Service Account Key: This key will grant the pipeline the necessary permissions to deploy the application on Google Cloud Run securely.

We'll set up these variables and secrets incrementally as we proceed, ensuring each step is fully secure and functional. 🎯

Set Up a Docker Hub Repository for the Project's Image and Generate an Access Token for Publishing the Image 📦

Before we dive into the steps, let’s quickly go over what we’re about to do. In this section, you’ll learn how to create a Docker Hub repository, which acts like an online storage space for your application’s container image.

Think of a container image as a snapshot of your application, ready to be deployed anywhere. To ensure smooth and secure access, we’ll also generate a special access token, kind of like a revokable password that our CI/CD pipeline can use to upload your app’s image to Docker Hub. Let’s get started! 🚀

Step 1: Sign Up for Docker Hub

Here are the steps to follow to sign up for Docker Hub:

1. Go to the Docker Hub website: Open your web browser and visit Docker Hub - https://hub.docker.com/.

2. Create an account: On the Docker Hub homepage, you’ll see a button labelled "Sign Up" in the top-right corner. Click on it.

3. Fill in your details: You'll be asked to provide a few details like your username, email address, and password. Choose a strong password that you can remember.

4. Agree to the terms: You’ll need to check a box to agree to Docker’s terms of service. After that, click “Sign Up” to create your account.

5. Verify your email: Docker Hub will send you an email to verify your account. Open that email and click on the verification link to complete your account creation.

Step 2: Sign In to Docker Hub

After verifying your email, go back to Docker Hub, and click on "Sign In" at the top right. Then you can use the credentials you just created to log in.

Step 3: Generate an Access Token (for the CI/CD pipeline)

Now that you have an account, you can create an access token. This token will allow your GitHub Actions workflow to securely sign into Docker Hub and upload Docker images.

Once you’re logged into Docker Hub, click on your profile picture (or avatar) in the top right corner. This will open a menu. From the menu, click “Account Settings”.

Then in the left-hand menu of your account settings, scroll to the "Security" tab. This section is where you manage your tokens and passwords.

Now you’ll need to create a new access token. In the Security tab, you’ll see a link labelled “Personal access tokens” – click on it. Click the button labelled “Generate new token”.

You’ll be asked to give your token a description. You can name it something like "GitHub Actions CI/CD" so that you know what it's for.

After giving it a description, click on the “Access permissions dropdown“ and select “Read & Write“, or “Read, Write, Delete“. Click “Generate“

Now, you need to copy the credentials. After clicking the generate button, Docker Hub will create an access token. Immediately copy this token along with your username and save it somewhere safe, like in a file (don’t worry, we’ll add it to our GitHub secrets). You won’t be able to see this token again, so make sure you save it!

Step 4: Add the Token to GitHub as a Secret

To do this, open your GitHub repository where the codebase is hosted. In the GitHub repo, click on the Settings tab (located near the top of your repo page).

Then on the left sidebar, scroll down and click on “Secrets and Variables”, then choose “Actions”.

Here are the steps to create and manage your new secret:

1. Add a new secret: Click on the “New repository secret” button.

2. Set up the secret:

   • In the Name field, type DOCKER_PASSWORD.

   • In the Value field, paste the access token you copied earlier.

3. Save the secret: Finally, click Add secret to save your Docker access token securely in GitHub.

Then you’ll repeat the process for your Docker username. Create a new secret called DOCKER_USER and add your Docker username that you copied earlier.

And that’s it! Now your CI/CD pipeline can use this token to securely log in to Docker Hub and upload images automatically when triggered. 🎉

Step 5: Creating the Dockerfile for the Project

Before you can build and publish the Docker image to Docker Hub, you need to create a Dockerfile that contains the necessary instructions to build your application.

Follow the steps below to create the Dockerfile in the root folder of your project:

1. Navigate to your project’s root folder.

2. Create a new file named Dockerfile.

3. Open the Dockerfile in a text editor and paste the following content into it:

FROM node:18-slim

WORKDIR /app

COPY package.json .

RUN npm install -f

COPY . .

# EXPOSE 5001
EXPOSE 5001

CMD ["npm", "start"]

Explanation of the Dockerfile:

• FROM node:18-slim: This sets the base image for the Docker container, which is a slim version of the official Node.js image based on version 18.

• WORKDIR /app: Sets the working directory for the application inside the container to /app.

• COPY package.json .: Copies the package.json file into the working directory.

• RUN npm install -f: Installs the project dependencies using npm.

• COPY . .: Copies the rest of the project files into the container.

• EXPOSE 5001: This tells Docker to expose port 5001, which is the port our app will run on inside the container.

• CMD ["npm", "start"]: This sets the default command to start the application when the container is run, using npm start.

Create a Google Cloud Account, Project, and Billing Account ☁️

In this section, we’re laying the foundation for deploying our application to Google Cloud. First, we’ll set up a Google Cloud account (don’t worry, it’s free to get started!). Then, we’ll create a new project where all the resources for your app will live.

Finally, we’ll enable billing so you can unlock the cloud services needed for deployment. Think of this as setting up your workspace in the cloud—organized, ready, and secure! Let’s dive in! ☁️

Step 1: Create or Sign in to a Google Cloud Account 🌐

First, go to Google Cloud Console. If you don’t have a Google Cloud account, you’ll need to create one.

To do this, click on Get Started for Free and follow the steps to set up your account (you’ll need to provide payment information, but Google offers $300 in free credits to get started). If you already have a Google account, simply sign in using your credentials.

Once you’ve signed in, you’ll be taken to your Google Cloud dashboard. This is where you can manage all your cloud projects and resources.

Step 2: Create a New Google Cloud Project 🏗️

At the top left of the Google Cloud Console, you’ll see a drop-down menu beside the Google Cloud logo. Click on this drop-down to display your current projects.

Now it’s time to create a new project. In the top-left corner of the pop-up modal, click on the New Project button.

You’ll be redirected to a page where you’ll need to provide some basic details for your new project. So now enter the following information:

• Project Name: Enter a name of your choice for the project (for example, gcr-ci-cd-project).

• Location: Select a location for your project. You can leave it as the default "No organization" if you're just getting started.

Once you've entered the project name, click the Create button. Google Cloud will now start creating your new project. It may take a few seconds.

Step 3: Access Your New Project 🛠️

After a few seconds, you’ll be redirected to your Google Cloud dashboard.

Click on the drop-down menu beside the Google Cloud logo again, and you should now see your newly created project listed in the modal where you can select it.

Then click on the project name (for example, gcr-ci-cd-project) to enter your project’s dashboard.

Step 4: Link A Billing Account To Your Project 💳

To access the billing page, in the Google Cloud Console, find the Navigation Menu (the three horizontal lines) at the top left of the screen. Click on it to open a list of options. Scroll down and click on Billing. This will take you to the billing section of your Google Cloud account.

If you haven't set up a billing account yet, you'll be prompted to do so. Click on the "Link a billing account" button to start the process.

Now you can create a new billing account (if you don’t have one). You’ll be redirected to a page where you can either select an existing billing account or create a new one. If you don't already have a billing account, click on "Create a billing account".

Provide the necessary details, including:

• Account name (for example, "Personal Billing Account" or your business name).

• Country: Choose the country where your business or account is based.

• Currency: Choose the currency in which you want to be billed.

  

Next, enter your payment information (credit card or bank account details). Google Cloud will verify your payment method, so make sure the information is correct.

Read and agree to the Google Cloud Terms of Service and Billing Account Terms. Once you’ve done this, click "Start billing" to finish setting up your billing account

After setting up your billing account, you’ll be taken to a page that asks you to link it to your project. Select the billing account you just created or an existing billing account you want to use. Click Set Account to link the billing account to your project.

After you’ve linked your billing account to your project, you should see a confirmation message indicating that billing has been successfully enabled for your project.

You can always verify this by returning to the Billing section in the Google Cloud Console, where you’ll see your billing account listed.

Create a Google Cloud Service Account to Enable Deployment of the Node.js Application to Google Cloud Run via the CD Pipeline 🚀

Why Do We Need a Service Account and Key? 🤔

A service account allows our CI/CD pipeline to authenticate and interact with Google Cloud services programmatically. By assigning specific roles (permissions), we ensure the service account can only perform tasks related to deployment, such as managing Google Cloud Run.

The service account key is a JSON file containing the credentials used for authentication. We securely store this key as a GitHub secret to protect sensitive information.

Step 1: Open the Service Accounts Page

Here are the steps you can follow to set up your service account and get your key:

First, visit the Google Cloud Console at https://console.cloud.google.com/. Ensure you’ve selected the correct project (e.g. gcr-ci-cd-project). To change projects, click the drop-down menu next to the Google Cloud logo at the top-left corner and select your project.

Then navigate to the Navigation Menu (three horizontal lines in the top-left corner) and click on IAM & Admin > Service Accounts.

Step 2: Create a New Service Account

Click on the "Create Service Account" button. This will open a form where you’ll define your service account details.

Next, enter the Service Account details:

• Name: Enter a descriptive name (for example, ci-cd-sa).

• ID: This will auto-fill based on the name.

• Description: Add a description to help identify its purpose, such as “Used for deploying Node.js app to Cloud Run.”

• Click Create and Continue to proceed.

Step 3: Assign Necessary Roles (Permissions)

On the next screen, you’ll assign roles to the service account. Add the following roles one by one:

• Cloud Run Admin: Allows management of Cloud Run services.

• Service Account User: Grants the ability to use service accounts.

• Service Usage Admin: Enables control over enabling APIs.

• Viewer: Provides read-only access to view resources.

To add a role:

• Click on "Select a Role".

• Use the search bar to type the role name (for example, "Cloud Run Admin") and select it.

• Repeat for all four roles.

Your screen should look similar to this:

After assigning the roles, click Continue.

Step 4: Skip Granting Users Access to the Service Account

On the next screen, you’ll see an option to grant additional users access to this service account. Click Done to complete the creation process.

Step 5: Generate a Service Account Key 🔑

You should now see your newly created service account in the list. Find the row for your service account (for example, ci-cd-sa) and click the three vertical dots under the “Actions” column. Select "Manage Keys" from the drop-down menu.

To add a new key:

• Click on "Add Key" > "Create New Key".

• In the pop-up dialog, select JSON as the key type.

• Click Create.

  

Now, download the key file. A JSON file will automatically be downloaded to your computer. This file contains the credentials needed to authenticate with Google Cloud.

Make sure you keep the key secure and store it in a safe location. Don’t share it – treat it as sensitive information.

Step 6: Add the Service Account Key to GitHub Secrets 🔒

Start by opening the downloaded JSON file using a text editor (like Notepad or VS Code). Then select and copy the entire contents of the file.

Then navigate to the repository you created for this project on GitHub. Click on the Settings tab at the top of the repository. Scroll down and find the Secrets and variables > Actions section.

Now you need to add a new secret. Click the "New repository secret" button. In the Name field, enter GCP_SERVICE_ACCOUNT. In the Value field, paste the JSON content you copied earlier. Click Add secret to save it.

Do the same for the GCP_PROJECT_ID secret, but now add your Google Project ID as the value. To get your project ID, follow these steps:

1. Navigate to the Google Cloud Console: Open Google Cloud Console at https://console.cloud.google.com/.

2. Locate the Project Dropdown: At the top-left of the screen, next to the Google Cloud logo, you will see a drop-down that shows the name of your current project.

3. View the Project ID: Click the drop-down, and you'll see a list of all your projects. Your Project ID will be displayed next to the project name. It is a unique identifier used by Google Cloud.

4. Copy the Project ID: Copy the Project ID that is displayed, and add it as the value of the GCP_PROJECT_ID secret.

Step 7: Adding External Variables to the GitHub Repository 🔧

Before proceeding with deployment, we need to define some external variables that were referenced in the CD workflow. These variables ensure that the pipeline knows critical details about your Google Cloud Run services and Docker container registry.

Here are the steps you’ll need to follow to do this:

1. First, go to your repository on GitHub.

2. Click the Settings tab at the top of the repository. Scroll down to Secrets and variables > Actions.

3. Click on the Variables tab next to Secrets. Click "New repository variable" for each variable. Then you’ll need to define these variables:

   • GCR_PROJECT_NAME: Set this to the name of your Cloud Run service for the production/live environment. For example, gcr-ci-cd-app.

   • GCR_STAGING_PROJECT_NAME: Set this to the name of your Cloud Run service for the staging/test environment. For example, gcr-ci-cd-staging.

   • GCR_REGION: Enter the region where you’d like to deploy the services. For this tutorial, set it to us-central1.

   • IMAGE: Specify the name of the Docker image/container registry where the published image will be uploaded. For example, <dockerhub-username>/ci-cd-tutorial-app.

4. After entering each variable name and value, click Add variable.

Enabling the Service Usage API on the Google Cloud Project 🌐

To deploy your application, the Service Usage API must be enabled in your Google Cloud project. This API allows you to manage Google Cloud services programmatically, including enabling/disabling APIs and monitoring their usage.

Follow these steps to enable it:

1. First, visit the Google Cloud Console at https://console.cloud.google.com/.

2. Then make sure you’re in the correct project. Click the project drop-down menu near the Google Cloud logo at the top-left corner. Select gcr-ci-cd-project , or the name you gave your project from the list of projects.

3. Next you’ll need to access the API library. Open the Navigation Menu (three horizontal lines in the top-left corner). Select APIs & Services > Library from the menu.

4. In the API Library, use the search bar to search for "Service Usage API".

5. Click on the Service Usage API from the search results. On the API’s details page, click Enable.

6. To verify, go to APIs & Services > Enabled APIs & Services in the Google Cloud Console. Confirm that the Service Usage API appears in the list of enabled APIs.

   

Create the Staging Branch and Merge the Feature Branch into It (Continuous Integration and Continuous Delivery) 🌟

When changes from the feature/ci-cd-pipeline branch are merged into the staging branch, we complete the Continuous Integration (CI) process, and the workflow ci-pipeline.yml will run. This ensures that the changes made in the feature branch are tested and integrated into a shared branch.

Once the pull request (PR) is merged into staging, the Continuous Delivery (CD) pipeline automatically triggers, deploying the application to the staging environment. This simulates how updates are tested in a safe environment before being pushed to production.

Create the staging Branch on the Remote Repository

To enable the CI/CD pipeline, we’ll first create a staging branch on the remote GitHub repository. This branch will serve as the test environment where changes are deployed before they reach the production environment.

To create the staging branch directly on GitHub, follow these steps:

1. First, navigate to your repository on GitHub. Open your web browser and go to the GitHub repository where you want to create the new staging branch.

2. Then, switch to the main branch. On the top of the repository page, locate the Branch dropdown (usually labelled as main or the current branch name). Click on the dropdown and make sure you are on the main branch.

3. Next, create the staging branch. In the same dropdown where you see the main branch, type staging into the text box. Once you start typing, GitHub will offer you the option to create a new branch called staging. Select the Create branch: staging option from the dropdown.

4. Finally, verify the branch**.** After creating the staging branch, GitHub will automatically switch to it. You should now see staging in the branch dropdown, confirming the new branch was created.

   

Merge Your Feature Branch into the Staging Branch via a Pull Request (PR)

This process combines both Continuous Integration (CI) and Continuous Delivery (CD). You will commit changes from your feature branch, push them to the remote feature branch, and then open a PR to merge those changes into the staging branch. Here's how to do it:

Step 1: Commit Local Changes on Your Feature Branch

First, you’ll want to make sure that you are on the correct branch (the feature branch) by running:

git status

If you are not on the feature/ci-cd-pipeline branch, switch to it by running:

git checkout feature/ci-cd-pipeline

Now, it’s time to add your changes you made for the commit:

git add .

This stages all changes, including new files, modified files, and deleted files.

Next, commit your changes with a clear and descriptive message:

git commit -m "Set up CI/CD pipelines for the project"

Then you can verify your commit by running:

git log

This will display your most recent commits, and you should see the commit message you just added.

Step 2: Push Your Feature Branch Changes to the Remote Repository

After committing your changes, push them to the remote repository:

git push origin feature/ci-cd-pipeline

This pushes your local changes on the feature/ci-cd-pipeline branch to the remote GitHub repository.

Once the push is successful, visit your GitHub repository in a web browser, and confirm that the feature/ci-cd-pipeline branch is updated with your new commit.

Step 3: Create a Pull Request to Merge the Feature Branch into Staging

Go to your repository on GitHub and ensure that you are on the main page of the repository.

You should see an alert at the top of the page suggesting you create a pull request for the recently pushed branch (feature/ci-cd-pipeline). Click the Compare & Pull Request button next to the alert.

Now, it’s time to choose the base and compare branches. On the PR creation page, make sure the base branch is set to staging (this is the branch you want to merge your changes into). The compare branch should already be set to feature/ci-cd-pipeline (the branch you just pushed). If they’re not selected correctly, use the dropdowns to change them.

You’ll want to come up with a good PR description for this. Write a clear title and description for the pull request, explaining what changes you're merging and why. For example:

• Title: "Merge CI/CD setup changes from feature branch"

• Description: "This pull request adds the CI/CD pipelines for GitHub Actions and Docker Hub integration to the project. It includes the configurations for both CI and CD workflows."

Now GitHub will show a list of all the changes that will be merged. Take a moment to review them and ensure everything looks correct.

If all looks good after reviewing, click on the Create pull request button. This will create the PR and notify team members (if any) that changes are ready to be reviewed and merged.

Wait a few seconds, and you should see a message indicating that all the checks have passed. Click on the link with the description "CI Pipeline to staging/production environment...". This should direct you to the Continuous Integration workflow, where you can view the steps that ran

The Continuous Integration (CI) Process

The CI process begins when a Pull Request is made to the staging branch. It triggers the GitHub Actions workflow defined in the .github/workflows/ci-pipeline.yml file. The workflow runs the necessary steps to set up the environment, install dependencies, and build the Node.js application.

It then runs automated tests (using npm test) to ensure that the changes do not break any functionality in the codebase. If all these steps are completed successfully, the CI pipeline confirms that the feature branch is stable and ready to be merged into the staging branch for further testing and deployment.

Step 4: Merge the Pull Request

If your team or collaborators are part of the project, they may review your PR. This step may involve discussing any changes or improvements. If everything looks good, a reviewer will merge the PR.

Once the PR has been reviewed and approved, you can merge the PR. To do this, just click on the Merge pull request button. Choose Confirm merge when prompted.

After merging, you can go to the staging branch to verify that the changes were successfully merged.

Navigating to the Actions Page After Merging the PR

Once you have successfully merged your pull request from the feature/ci-cd-pipeline branch into the staging branch, the Continuous Delivery (CD) pipeline will be triggered. To view the progress of the CD pipeline, navigate to the Actions tab in your GitHub repository. Here's how to do it:

1. Go to your GitHub repository.

2. At the top of the page, you will see the Actions tab next to the Code tab. Click on it.

3. On the Actions page, you will see a list of workflows that have been triggered. Look for the one labelled CD Pipeline to Google Cloud Run (staging and production). It should appear as a new run after the PR merge.

4. Click on the workflow run to view its progress and see the detailed logs for each step.

This will allow you to monitor the status of the CD pipeline and check if there are any issues during deployment.

If you look at the CD steps and workflow, you'll see that the step to deploy the application to the production environment was skipped, while the step to deploy to the staging environment was executed.

Continuous Delivery (CD) pipeline – what’s going on:

The Continuous Delivery (CD) Pipeline automates the process of deploying the application to Google Cloud Run (testing environment). This workflow is triggered by a push to the staging branch, which happens after the changes from the feature branch are merged into staging. It can also be manually triggered via workflow_dispatch or upon a new release being published.

The pipeline consists of multiple stages:

1. Test Job: The pipeline begins by setting up the environment and running tests using the npm test command. If the tests pass, the process moves forward.

2. Build Job: The next step builds the Docker image of the Node.js application, tags it, and then pushes it to Docker Hub.

3. Deployment to GCP: After the image is pushed, the workflow authenticates to Google Cloud and deploys the application. If the event is a release (that is, a push to the main branch), the application is deployed to the production environment. If the event is a push to staging, the app is deployed to the staging environment.

The CD process ensures that any changes made to the staging branch are automatically tested, built, and deployed to the staging environment, ready for further validation. When a release is published, it will trigger deployment to production, ensuring your app is always up to date.

Accessing the Deployed Application in the Staging Environment on Google Cloud Run 🌐

Once the deployment to Google Cloud Run is successfully completed, you'll want to access your application running in the staging environment. Follow these steps to find and visit your deployed application:

1. Navigate to the Google Cloud Console

Open the Google Cloud Console in your browser by visiting https://console.cloud.google.com. If you're not already signed in, make sure you log in with your Google account.

2. Go to the Cloud Run Dashboard

In the Google Cloud Console, use the Search bar at the top or navigate through the left-hand menu: Go to Cloud Run (you can type this into the search bar, or find it under Products & services > Compute > Cloud Run). Click on Cloud Run to open the Cloud Run dashboard.

3. Select Your Staging Service

In the Cloud Run dashboard, you should see a list of all your services deployed across various environments. Find the service associated with the staging environment. The name should be similar to what you defined in your workflow (for example, gcr-ci-cd-staging).

4. Access the Service URL

Once you've selected your staging service, you’ll be taken to the Service details page. This page provides all the important information about your deployed service.
On this page, look for the URL section under the Service URL heading. The URL will look something like: https://gcr-ci-cd-staging-<unique-id>.run.app.

5. Visit the Application

Click on the Service URL, and it will open your staging environment in a new tab in your browser. You can now interact with your application as if it were live, but in the staging environment.

Merge the Staging Branch into the Main Branch (Continuous Integration and Continuous Deployment) 🌐

In this section, we'll take the updates in the staging branch, merge them into the main branch, and trigger the CI/CD pipeline. This process not only ensures your changes are production-ready but also deploys them to the production/live environment. 🚀

Step 1: Push Local Changes and Open a Pull Request

Why? The first step involves merging the staging branch into the main branch. Just like in the previous Continuous Delivery process, this ensures the integration of thoroughly tested updates.

Here’s how to do it:

First, visit the GitHub repository where your project is hosted.

Then go to the Pull Requests tab. Click New Pull Request. Choose staging as the source branch (base branch) and main as the target branch. Add a clear title and description for the Pull Request, explaining why these updates are ready for production deployment.

Step 2: Continuous Integration (CI) Pipeline Execution

After merging the pull request, the Continuous Integration (CI) pipeline will automatically execute to validate that the changes are still stable when integrated into the main branch.

Pipeline Steps:

• Code Checkout: The workflow fetches the latest code from the main branch.

• Dependency Installation: The pipeline installs all required dependencies.

• Testing: Automated tests are run to validate the application's stability.

Step 3: Create a New Release

The Continuous Deployment (CD) workflow to deploy to the production environment is triggered by the creation of a new release from the main branch.

Let’s walk through the steps to create a release.

On your GitHub repository page, click on the Releases section (located under the Code tab).

Next, click Draft a new release. Set the Target branch to main. Enter a Tag version (for example, v1.0.0) following semantic versioning. Add a Release title and an optional description of the changes.

Then, click Publish Release to finalize.

Why run the Continuous Deployment pipeline on release instead of on push? 🤔

In our setup, we decided not to trigger the Continuous Deployment (CD) pipeline every time changes are pushed to the main branch. Instead, we trigger it only when a new release is created. This gives the team more control over when updates are deployed to the production environment.

Imagine a scenario where developers are working on new features—they may push changes to the main branch as part of their regular workflow, but these features might not be complete or ready for users yet. Automatically deploying every push could accidentally expose unfinished features to your users, which can be confusing or disruptive.

By requiring a release to trigger the deployment, the team gets a chance to finalize and polish all changes before they go live.

For example, developers can test new features in the staging environment, fix any issues, and merge those changes into the main branch without worrying about them immediately appearing in production. This workflow ensures that only well-tested and complete features make their way to your end users.

Ultimately, this approach helps maintain a smooth user experience. Instead of seeing half-built features or unexpected changes, users only see updates that are ready and functional. It also gives the team the flexibility to push changes to the main branch frequently—preventing merge conflicts and making collaboration easier—while keeping control over what gets deployed live. 🚀

Step 4: Navigate to the Actions Page

After the release is published, the CD pipeline for the production environment is triggered. To monitor this repeat the process taken for the Continuous Delivery workflow, follow these steps:

1. Go to the GitHub Actions tab: In your GitHub repository, click on the Actions tab.

2. Locate the deployment workflow: Look for the CD Pipeline to Google Cloud Run (staging and production) workflow. You’ll notice that the workflow has been triggered on the main branch due to the push event.

3. Open the workflow details: Click on the workflow to view detailed steps, logs, and statuses for each part of the deployment process.

This time, the Continuous delivery workflow deploys the application to the production/live environment.

Step 5: Access the Live Application

Once the deployment is complete, go to Google Cloud Console at https://console.cloud.google.com.

Navigate to Cloud Run from the menu. Select the service corresponding to the production environment (for example, gcr-ci-cd-app).

Locate the Service URL in the service details page. Open the URL in your browser to access the live application.

And now, congratulations – you’re done!

Conclusion 🌟

In this article, we explored how to build and automate a CI/CD pipeline for a Node.js application, using GitHub Actions, Docker Hub, and Google Cloud Run.

We set up workflows to handle Continuous Integration by testing and integrating code changes and Continuous Delivery to deploy those changes to a staging environment. We also containerized our app using Docker and deployed it seamlessly to Google Cloud Run.

Finally, we implemented Continuous Deployment, ensuring updates to the production environment happen only when a release is created from the main branch.

This approach gives teams the flexibility to push and test incomplete features without impacting end users. By following these steps, you've built a robust pipeline that makes deploying your application smoother, faster, and more reliable.

Study Further 📚

If you would like to learn more about Continuous Integration, Delivery, and Deployment you can check out the courses below:

• Continuous Integration and Continuous Delivery (CI/CD) (from IBM Coursera)

• GitHub Actions - The Complete Guide (from Udemy)

• Learn CI/CD by buliding a project (freeCodeCamp tutorial)

About the Author 👨‍💻

Hi, I’m Prince! I’m a software engineer passionate about building scalable applications and sharing knowledge with the tech community.

If you enjoyed this article, you can learn more about me by exploring more of my blogs and projects on my LinkedIn profile. You can find my LinkedIn articles here. And you can visit my website to read more of my articles as well. Let’s connect and grow together! 😊

We're going to review what this practice is about, how it compares to the previous approach in the software development industry, and finally see a practical example of how we can implement it in our projects.

Let's go!

Table of Contents

• Intro

• How CI/CD works

• The key benefits of CI/CD

• Tools for CI/CD

• How to set up a CI/CD pipeline with GitHub Actions

  • Initializing the project

  • What are GitHub Actions and how do they work?

  • Setting up our workflow

  • The magic

• Wrapping up

Intro

Continuous Integration and Continuous Delivery (CI/CD) is a software development approach that aims to improve the speed, efficiency, and reliability of software delivery. This approach involves frequent code integration, automated testing, and continuous deployment of software changes to production.

Before the adoption of CI/CD in the software development industry, the common approach was a traditional, waterfall model of software development.

In this approach, developers worked in silos, with each stage of the software development life cycle completed in sequence. The process typically involved gathering requirements, designing the software, coding, testing, and deployment.

The disadvantages of this traditional approach include:

1. Slow Release Cycles: Since each stage of the software development life cycle was completed in sequence, the release cycle was slow, which made it difficult to respond quickly to changing customer needs.

2. High Failure Rates: Software projects were prone to failure due to a lack of automated testing, which meant that developers had to rely on manual testing, leading to errors and bugs in the code.

3. Limited Collaboration: The traditional approach did not encourage collaboration between developers, testers, and other stakeholders, which made it difficult to identify and fix issues.

4. High Cost: The manual nature of software development meant that it was expensive, with high costs associated with testing, debugging, and fixing errors.

5. Limited Agility: Since the traditional approach was linear, it was not possible to make changes to the software quickly or respond to customer needs in real-time.

CI/CD emerged as a solution to these disadvantages, by introducing a more agile and collaborative approach to software development. CI/CD enables teams to work together, integrating their code changes frequently, and automating the testing and deployment process.

How CI/CD Works

CI/CD is an automated process that involves frequent code integration, automated testing, and continuous deployment of software changes to production.

Let's explain each step in a little more detail:

Code Integration

The first step in the CI/CD pipeline is code integration. In this step, developers commit their code changes to a remote repository (like GitHub, GitLab or BitBucket), where the code is integrated with the main codebase.

This step aims to ensure that the code changes are compatible with the rest of the codebase and do not break the build.

Automated Testing

Once the code is integrated, the next step is automated testing. Automated testing involves running a suite of tests to ensure that the code changes are functional, meet the expected quality standards, and are free of defects.

This step helps identify issues early in the development process, allowing developers to fix them quickly and efficiently.

If you're not familiar with the topic of testing, you can refer to this article I wrote a while ago.

Continuous Deployment

After the code changes pass the automated testing step, the next step is continuous deployment. In this step, the code changes are automatically deployed to a staging environment for further testing.

This step aims to ensure that the software is continuously updated with the latest code changes, delivering new features and functionality to users quickly and efficiently.

Production Deployment

The final step in the CI/CD pipeline is production deployment. In this step, the software changes are released to end-users. This step involves monitoring the production environment, ensuring that the software is running smoothly, and identifying and fixing any issues that arise.

The four steps of a CI/CD pipeline work together to ensure that software changes are tested, integrated, and deployed to production automatically. This automation helps to reduce errors, increase efficiency, and improve the overall quality of the software.

By adopting a CI/CD pipeline, development teams can achieve faster release cycles, reduce the risk of software defects, and improve the user experience.

Keep in mind that the pipeline stages might look different given the specific project or company we're talking about. Meaning, some teams might or might not use automated testing, some teams might or might not have a "staging" environment, and so on.

The key parts that make up the CI/CD practice are integration and deployment. This means that the code has to be continually integrated in a remote repository, and that this code has to be continually deployed to a given environment after each integration.

The Key Benefits of CI/CD

The key benefits of CI/CD include:

1. Faster Release Cycles: By automating the testing and deployment process, CI/CD enables teams to release software more frequently, responding quickly to customer needs.

2. Improved Quality: Automated testing ensures that software changes do not introduce new bugs or issues, improving the overall quality of the software.

3. Increased Collaboration: Frequent code integration and testing require developers to work closely together, leading to better collaboration and communication.

4. Reduced Risk: Continuous deployment allows developers to identify and fix issues quickly, reducing the risk of major failures and downtime.

5. Cost-Effective: CI/CD reduces the amount of manual work required to deploy software changes, saving time and reducing costs.

In summary, CI/CD emerged as a solution to the limitations of the traditional, linear approach to software development. By introducing a more agile and collaborative approach to software development, CI/CD enables teams to work together, release software more frequently, and respond quickly to customer needs.

Tools for CI/CD

There are several tools available for implementing CI/CD pipelines in software development. Each tool has its unique features, pros, and cons. Here are some of the most commonly used tools in CI/CD pipelines today:

Jenkins

Jenkins is an open-source automation server that is widely used in CI/CD pipelines. It is highly customizable and supports a wide range of plugins, making it suitable for various development environments. Some of its key features include:

Pros:

• Highly customizable with a wide range of plugins

• Supports integration with various tools and technologies

• Provides detailed reporting and analytics

Cons:

• Requires some technical expertise to set up and maintain

• Can be resource-intensive, especially for large projects

• Lack of a centralized dashboard for managing multiple projects

If you want to learn more about Jenkins, here's a full course for you.

Travis CI

Travis CI is a cloud-based CI/CD platform that provides automated testing and deployment for software projects. It supports several programming languages and frameworks, making it suitable for various development environments. Some of its key features include:

Pros:

• Easy to set up and use

• Cloud-based, so there's no need to set up and maintain infrastructure

• Supports a wide range of programming languages and frameworks

• Provides detailed reporting and analytics

Cons:

• Limited customization options

• Not suitable for large projects with complex requirements

• Limited support for on-premise installations

Here's a helpful tutorial about how to automate deployment on GitHub Pages with Travis CI.

GitHub actions

GitHub Actions is a powerful CI/CD tool that allows developers to automate workflows, run tests, and deploy code directly from their GitHub repositories.

Pros:

• Integrated with GitHub

• Easy to use

• Provides large ecosystem and good documentation

Cons:

• Limited build minutes

• Complex YAML syntax

Side comment: I mention GitHub actions here because it's a popular tool – but keep in mind that other online repository providers like GitLab and BitBucket also provide very similar options to GitHub actions.

Built-in CI/CD features by hosts

Popular hosts such as Vercel or Netlify have built-in in CI/CD features that allow you to link an online repository to a given site, and deploy to that site after a given event occurs in that repo.

Pros:

• Very simple to set up and use

Cons:

• Limited customization options

Each of these tools has its unique features, pros, and cons. The choice of tool will depend on the specific requirements of your project, your team's technical expertise, and your budget.

Here's a tutorial about how to deploy a front-end app with Netlify. And in this tutorial, you'll learn how to use Vercel to deploy a Next.js app.

How to Set Up a CI/CD Pipeline with GitHub Actions

Cool, so now that we have a clear idea of what CI/CD is, let's see how we can implement a simple example with an actual project using GitHub actions.

Initializing the Project

We'll start with a very basic React app built with Vite. You can do that by running yarn create vite in your console.

We'll focus on the CI/CD pipeline here, so there will be no complexity in the actual app code. But just to have an idea, the app.jsx component will have this code:

import './App.css';

function App() {

    return (
        <div className='App'>
            <h1>Vite + Reactooooo</h1>
        </div>
    );
}

export default App;

And then we'll have a test file that will check for that text to render:

import { describe, expect, it } from 'vitest';
import { render, screen } from './utils/test-utils/test-utils.jsx';

import App from 'src/App.jsx';

describe('App', async () => {
    it('should render while authenticating', () => {
        render(<App />);

        expect(screen.getByText('Vite + Reactooooo')).toBeInTheDocument();
    });
});

This test will run each time we run the yarn test command.

Next step should be to push our code to a GitHub repo. Then, let's talk a bit more about what GitHub actions are and how they work.

What are GitHub Actions and How Do They Work?

GitHub Actions is a CI/CD (Continuous Integration/Continuous Deployment) service provided by GitHub. It allows developers to automate workflows by defining custom scripts, known as "actions", that can be triggered by events such as pushes to a repository, pull requests, or issues.

Actions are defined in a YAML file, also known as a "workflow", which specifies the steps required to complete a task. GitHub Actions workflows can run on Linux, Windows, and macOS environments and support a wide range of programming languages and frameworks.

When an event triggers a GitHub Actions workflow, the service creates a fresh environment, installs dependencies, and runs the defined steps in the order specified. This can include tasks such as building, testing, packaging, and deploying code.

GitHub Actions also provides several built-in actions that can be used to simplify common tasks, such as checking out code, building and testing applications, publishing releases, and deploying to popular cloud providers like AWS, Azure, and Google Cloud.

GitHub Actions workflows can be run on a schedule, manually, or automatically when a specific event occurs, such as a pull request being opened or a new commit being pushed to a branch.

Setting Up Our Workflow

Great, so as we've seen, basically GitHub actions are a feature that allows us to define workflows four our projects. These workflows are nothing but a series of tasks or steps that will execute on GitHub's cloud after a given event we declare.

The way GitHub reads and executes these workflows is by automatically reading files within the .github/workflows directory in the root of our project. These workflow files should have the .yaml extension and use the YAML syntax.

To create a new workflow we just have to create a new YAML file within that directory. We'll call ours prod.yaml since we'll use it to deploy the production branch of our project.

Keep in mind a single project can have many different workflows that run different tasks on different occasions. For example, we could have a workflow for dev and staging branches as well, as those environments could require different tasks to execute and will probably deploy on different sites.

After creating this file, let's drop the following code in it:

# Name of our workflow
name: Production deploy

# Trigger the workflow on push to the main branch
on:
  push:
    branches:
      - main

# List of jobs
# A "job" is a set of steps that are executed on the same runner
jobs:
  # Name of the job
  test-and-deploy-to-netlify:
    # Operating system to run on
    runs-on: ubuntu-latest

    # List of steps that make up the job
    steps:
    # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
    - name: Checkout code
      uses: actions/checkout@v2

    # Setup Node.js environment
    - name: Use Node.js 16.x
      uses: actions/setup-node@v2
      with:
        node-version: '16.x'

    # Install dependencies
    - name: Install dependencies
      run: yarn install

    # Run tests
    - name: Run tests
      run: yarn test

    # Deploy to Netlify
    - name: Netlify Deploy
      uses: jsmrcaga/action-netlify-deploy@v2.0.0
      with:
        # Auth token to use with netlify
        NETLIFY_AUTH_TOKEN: ${{ secrets.NETLIFY_AUTH_TOKEN }}
        # Your Netlify site id
        NETLIFY_SITE_ID:  ${{ secrets.NETLIFY_SITE_ID }}
        # Directory where built files are stored
        build_directory: './dist'
        # Command to install dependencies
        install_command: yarn install
        # Command to build static website
        build_command: yarn build

So our workflow has the following tasks declared:

1. "Checkout code" step that checks out the latest commit on the current branch.

2. "Use Node.js 16.x" step that sets up the Node.js environment to version 16.x.

3. "Install dependencies" step that installs the project dependencies using the Yarn package manager.

4. "Run tests" step that runs the project's tests using the Yarn package manager.

5. "Netlify Deploy" step that deploys the project to Netlify using the jsmrcaga/action-netlify-deploy action. This step uses the Netlify authentication token and site ID secrets stored in the GitHub repository's secrets. The build directory, install command, and build command are also specified.

You probably noticed the first and last steps have the keyword uses. This keyword allows you to use actions or workflows developed by other GitHub users, and it's one of the best features of GitHub actions.

What's great is that by using this third party actions we can execute complex tasks such as deploying to an external host or building complex cloud infrastructure without the need to write every single line of necessary code.

As these tasks tend to be repetitive and frequently executed in many projects, we can just use a workflow developed by an official company account (such as Azure or AWS for example) or an independent open-source developer. Think about it as using a third party library. It's the same idea but taken to CI/CD workflows. Very convenient.

Another important thing to mention here is that in GitHub actions workflows tasks run sequentially, one after the other. And if a given task fails or throws and error, the next one won't execute. This is important because if we have a problem when installing our dependencies or a test fails, we don't want that code to be deployed.

Before we can push this code and see how the magic works, we first need to create a site on Netlify and get the NETLIFY_AUTH_TOKEN and NETLIFY_SITE_ID. This is quite straight forward even if you don't have previous experience with Netlify, so give it a try and Google a bit if you can't figure it out. ;)

Once you have these two tokens, you need to declare them as repository secrets in your GitHub repo. You can do this from the "settings" tab:

Configure both Netlify secret tokens in your repo

With this in place, now our prod.yaml file will be able to read these two tokens and execute the Netlify deploy action.

The Magic

Now that we have everything in place, let's push our code and see how it goes.

After pushing, if we go to the "actions" tabs of our repo, on the left we'll see a list of all the workflows we have in our repo. And on the right we'll see a list of each execution of the selected workflow. Since our workflow executes after each push, we should see a new execution each time we push.

A workflow execution

When the execution has a yellow light to the left of it, it means it's still running (executing tasks). If it has a green light it means it finished executing successfully and if the light is red, you know something went wrong, haha...

After clicking on the execution we can see a list of the workflow's jobs (we only had a single one).

The workflow's jobs

And after clicking on the job we can see a list of the job's tasks.

The tasks of the job

Each task is expansible and within it we can see logs corresponding to that task's execution. This is quite useful for debugging purposes. ;)

Now if we go to our previously set up Netlify site, we should see our app up and running!

And now that we have our CI/CD pipeline in place, we can deploy our app after each push to the main branch, all without lifting another finger. =D

Wrapping Up

CI/CD is a software development approach that provides several benefits to software development teams, including faster time-to-market, improved quality, increased collaboration, reduced risk, and cost-effectiveness.

By automating the software delivery pipeline, teams can quickly deploy new features and bug fixes, while reducing the risk of major failures and downtime.

With the availability of several CI/CD tools, it has become easier for teams to implement this approach and improve their software delivery process.

Well everyone, as always, I hope you enjoyed the article and learned something new.

If you want, you can also follow me on LinkedIn or Twitter. See you in the next one!

But I recently discovered a way to automate the deployment process for Next.js apps using AWS CodeDeploy and CodePipeline. It made my life so much easier, and I'm excited to share it with you.

In this tutorial, I'll guide you through the process of setting up auto-deployment for your Next.js app using the AWS services CodePipeline and CodeDeploy. By the end of it, you'll be able to save a lot of time by deploying your app automatically every time you push the code.

Let's get started!

Table of Contents:

1. Prerequisites
2. How to Deploy the Next.js App to AWS EC2
3. How to Run the Next.js App in Production Mode
4. How to Run a Next.js App Forever When the Console is Closed
5. What is CodeDeploy?
6. How to Setup Auto-Deployment using CodePipeline and CodeDeploy
7. How to Attach the IAM Role to EC2
8. How to Create the CodePipeline
9. Conclusion

Prerequisites

1. EC2 machine running Ubuntu
2. Very basic knowledge of EC2 and IAM AWS Services

How to Deploy the Next.js App to AWS EC2

To start simple, let's manually deploy the sample Next.js boilerplate app "hello-world" to EC2. The steps are almost the same for all Next.js applications.

Login to EC2

Login to the EC2 machine which you've created using the below command:

ssh -i /path/key-pair-name.pem instance-user-name@instance-IP-address

When you try to log into EC2, this is the common error that most people will encounter (I got it, too):

Permissions 0664 for .pem file is too open error

This error describes that the .pem file should be read-protected. Only the root user should be able to read it. So, you have to set the file permission to 400. Run the following command to achieve that:

chmod 400 key-pair-name.pem

EC2 by default comes with no software installed. Once you've logged into EC2, install NodeJS. There's an excellent article published by Digital Ocean and I use it every time I have to install Node on the server.

I have uploaded the boilerplate repo to Github. You can clone the repo by running the following command:

git clone https://github.com/5minslearn/deploy_nextjs_app.git

Navigate to the project and install the dependencies by running the below commands.

A quick note here. I'm a big fan of yarn for its lightning-fast dependency management. But I see most people use npm to manage their dependencies. If you like to use npm, you can replace yarn install with npm install in the below commands.

If you like to go with yarn, install yarn by following this tutorial first.

cd deploy_nextjs_app
yarn install

Let's run the application:

yarn dev

Hit "http://ec2-public-ip-address:3000/" on your browser and you should be able to see the following page:

Next.js Hello World App

There's another common issue that most people face here which we'll look at next.

How to fix the timeout error (EC2)

"Oh, My God! My site is loading for a long time and finally, it's throwing a timeout error. What could be the issue? Where did I made mistake?"

If this happens to you, then you can follow the below steps to fix it.

This issue basically occurs if your server does not expose port 3000. Remember, by default Next apps will be running on port 3000. But, you have to allow port 3000 from the Security Group of your EC2 console to access from your browser.

Login to your AWS console, select your EC2 instance, and then select the Security Group option. Click on the "Edit inbound rules" button. Add port 3000 to the list as shown in the below screenshot. Then hit the "Save rules" button.

Adding port 3000 to a security group

Visit the link "http://ec2-public-ip-address:3000/", and you'll be amazed to see that your page loads like magic.

So far, we've just run our app in development mode and verified that it's working.

How to Run the Next.js App in Production Mode

To deploy the app in Production Mode, you have to build your app first. Run yarn build to build the app and yarn start to start the app in production mode.

yarn build
yarn start

Hit "http://ec2-public-ip-address:3000/" again and this time you'll see that your app loads faster than before.

Apps running in Production mode will always be faster when compared to the ones running in Development mode. This is because Production apps will be optimized for performance.

How to Run a Next.js App Forever When the Console is Closed

So, you have your app running now. But you might notice that it's blocking you from closing your terminal and exiting from server connection. If you do so, your site will be down. That's where PM2 comes to play.

Basically, PM2 is a process manager that helps keep Node applications alive all the time. It runs in the background managing Node applications for you.

Install PM2 using the following command:

sudo yarn global add pm2

After PM2 installation, run the below command to run and manage your app in the background:

pm2 start yarn --name [name-of-your-app] -- start -p [port-number]

Replace [name-of-your-app] with your app name and [port-number] with 3000. Here's an example command,

pm2 start yarn --name next_hello_world_app -- start -p 3000

Hit "http://ec2-public-ip-address:3000/" and you'll again be amazed to see your app up and running.

It's always a best practice to save the PM2 process. When you reboot your instance, your PM2 instances will be lost. In order to restore it to its old state, you have to save the PM2 process. Here's the command for that:

pm2 save

Here's the command to restore your PM2 instances on reboot (don't execute this now, we'll come back to this shortly):

pm2 resurrect

We have successfully deployed the Next.js app manually. But remember, every time you make a code change and want to see the changes on your site, you have to login into EC2, pull the latest changes, build the app, and restart the app.

This will consume a lot of time and I'm too lazy to do it. So let's automate this in the next step!

Before setting up automatic deployment you have to know how CodeDeploy works.

What is CodeDeploy?

CodeDeploy lets you deploy your application automatically to any number of EC2 instances. We need to prepare two items before beginning this process:

1. CodeDeploy Agent must be installed in the EC2 instance. We use this to continuously poll CodeDeploy and deploy if any new changes are available.
2. A file called appspec.yml must be present in the root folder. This file describes the steps to be followed for the deployment.

There is awesome documentation by AWS to help you install CodeDeploy Agent. Please follow each and every step to install CodeDeploy Agent on your EC2 machine.

To verify that CodeDeploy agent is installed, run the below command. If you see active (running), Kudos to you! CodeDeploy was installed successfully.

sudo service codedeploy-agent status

CodeDeploy Agent running status

Now let's create the appspec.yml file. I've written the deployment instructions in the deploy.sh file. It's enough to run this file in the appspec.yml file. If you want to learn more about appspec.yml, check out the AWS official documentation.

Create a file called appspec.yml and add the following contents:

version: 0.0
os: linux
hooks:
  ApplicationStart:
    - location: deploy.sh
      timeout: 300
      runas: ubuntu

I hope you understand the instructions in the above file. If not, here's a super simple explanation. I'm advising the CodeDeploy Agent that I'm running a Linux OS in my instance and instructing it to run the deploy.sh file as ubuntu user with the timeout set to 300 seconds.

Here's my deploy.sh file:

#!/bin/bash
cd /path/to/project/on/EC2 
git pull origin master
yarn install &&
yarn build &&
pm2 restart [name]

This file contains instructions to navigate to the project folder on EC2, pull the latest code from source control, install dependencies, build the project, and restart the project instance.

This file is already available in the repo. No action for you here. Now it's time to set up automatic deployment.

How to Setup Auto-Deployment using CodePipeline and CodeDeploy

Two IAM roles have to be created to set up auto-deployment. Some complications will begin from here. To make things simple, I've attached screenshots with the appropriate items highlighted with red boxes.

Create an IAM Role for CodeDeploy

You have to create this role to deploy the code every time you push.

Navigate to IAM in the AWS Console by searching for "IAM" in the search bar at the top. Click Roles on the left pane and Click the "Create role" button at the top right.

Create IAM role

Choose AWS service in Trusted entity types and choose CodeDeploy in the Use cases section and proceed to the next step.

IAM role for CodeDeploy

Now, you can see that the AWSCodeDeployRole policy is the only policy available, and it'll be chosen by default in this (Permissions) step. Let's proceed to the next section. No action for you here.

AWSCodeDeploy Permission

Enter a name for your IAM role. You should choose a meaningful name to identify this in the future. I'm calling it service-role-for-code-deploy. Review the permission in the JSON and click the Create role button at the bottom.

AWSCodeDeploy Permission Review

Create an IAM role for EC2

Let's create the next role. This role is for EC2. Choose AWS service in the Trusted entity type, EC2 in the Common use cases section, and choose CodeDeploy in Use cases for other AWS services. Click Next to proceed to the next section.

IAM role for EC2

There are a lot of policies available for EC2 and CodeDeploy. In the Add permissions section, search for codedeploy (No space between code and deploy) and select "AmazonEC2RoleForCodeDeploy" and proceed to the next step.

Adding AmazonEC2RoleForCodeDeploy permission

No change in this step. Review and give a meaningful name (I'm naming it as "code-deploy-role-for-ec2") for your role and click "Create role" button.

How to Attach the IAM Role to EC2

Once the IAM role for EC2 is created, we have to attach it to the EC2 instance.

EC2 instance before attaching the IAM role

To attach the IAM role to the EC2 instance, open your EC2 instance, click on the "Actions" button on the top right, and select "Security" in the drop-down. Then select "Modify IAM role".

Modify IAM role for EC2 instance

Select the IAM role which you created last (code-deploy-role-for-ec2) and click the "Update IAM role" button. Reboot the EC2 for the changes to take effect.

Update IAM role for EC2 instance

EC2 instance after attaching IAM role

After rebooting the EC2, login to EC2 with SSH and run the pm2 resurrect command to restore the PM2 processes. Failing to do this may land you at "PM2 Process or Namespace not found error" while running automatic deployment.

PM2 process restore

PM2 process or namespace not found an error

How to Create the CodeDeploy Application

In the AWS Console, search "CodeDeploy" in the search bar at the top. Select "Applications" in the left pane. Click on the "Create application" button on the top right.

Navigate to CodeDeploy in AWS Console

Enter the Application name, choose the "EC2/On-premises" compute platform, and click the "Create application" button.

Create CodeDeploy application

Once it's done, you'll automatically be redirected to the Deployment groups section. We have to create a deployment group. Click on the "Create deployment group" button.

Create CodeDeploy Deployment group

Enter the deployment group name, select the service role (1st created role) you created, and select the deployment type as in-place:

Create CodeDeploy Deployment group

In the Environment configuration section, select "Amazon EC2 instances" and select the key as Name. Enter your EC2 instance name in the value.

Code Deployment Group Environment configuration

In the Agent configuration section, select Never, as we installed CodeDeployAgent already. Select "CodeDeployDefault.AllAtOnce" in the Deployment settings section. Leave the "Enable load balancing" checkbox unchecked. Finally, click the "Create a deployment group" button.

CodeDeploy deployment group configurations

How to Create the CodePipeline

AWS CodePipeline helps you to automate your release pipelines for fast and reliable application and infrastructure updates. Now it's time to create our CodePipeline. In the AWS Console, search for "CodePipeline" in the search bar.

Select "Pipelines" in the left pane and click on "Create pipeline" button.

Create CodePipeline

Enter the Pipeline name, and Role name. Remember, we created roles for EC2 and CodeDeploy, but not for CodePipeline. AWS by default creates it from here.

CodePipeline settings

Add Source Stage

In this step, we have to connect our repo with CodePipeline to deploy the changes immediately after the code is pushed.

We'll be using GitHub as our source. Choose GitHub (version 2) in the source provider, and click on the "Connect to GitHub" button. This will open up a new pop-up window. Click on the "Connect to GitHub" button.

CodePipeline adding source stage

This will take you to the GitHub authorization page where you have to sign into your GitHub account. Click on the "Install a new app" button.

CodePipeline Github Authorization

Choose "Only select repositories" and choose your repository below that.

Installing AWS connector for GitHub

Once installed, it will prompt you for the password. Click the "Connect" button once you're done with your authentication.

Connecting GitHub to AWS

After connecting to GitHub, select the Repository name and branch name. To start the CodePipline on code change, it's important to select the check box "Start the pipeline on source code change" – otherwise auto deployment will not happen. For "Output and artifact format", select "CodePipeline default" and click the "Next" button.

CodePipeline - Select source code repo

The next step is to add the build stage, but since we're deploying a simple app we don't need a build stage. Enterprise companies prefer to build their app using AWS CodeBuild service. Let's keep things simple and simply skip the build stage.

If you want me to write about CodeBuild let me know, I will try to cover it in my upcoming tutorials.

Add Deploy Stage

In the deployment stage step, choose "AWS CodeDeploy" for the "Deploy provider" and select the region where you created the above CodeDeploy application. Then select the "Application name" and "Deployment group" that we created in the previous steps and click the "Next" button.

CodePipleine - Adding Deployment stage

The last step is "Review". Review everything carefully and click on the "Create pipeline" button. Once the pipeline is created it will start the deployment process. If you followed all the above steps, the pipeline should read "Succeeded" on your very first build.

Pipeline Succeeded

How to Verify Auto-Deployment

Now let's verify if the Auto-deployment works properly. This is the Home page of our project:

Next.js Hello World App

Let's change the text from "Hello World" to "Welcome to 5minslearn" and push the code to GitHub.

Git code diff

Here we go! The CodePipeline has triggered automatically and the changes were successfully deployed.

CodeDeploy getting triggered automatically on code changes in Git

Now head to "http://ec2-public-ip-address:3000/", you will see the below page:

Next.js App after auto deployment

Congrats! 🎉 We successfully completed setting up auto-deployment for a Next.js app.

Conclusion

In this article, we learned how to deploy Next.js manually on EC2 and set up auto-deployment using AWS services such as CodeDeploy and CodePipeline.

Hope you enjoyed reading this article! If you are stuck at any point feel free to drop your queries to my email. I'll be happy to help you!

If you wish to learn more about AWS, subscribe to my newsletter (https://5minslearn.gogosoon.com/) and follow me on social media.

As you continue to develop your software, you must also continue to integrate it with previous code and deploy it to servers.

Manually doing this is a time-consuming process that can occasionally result in errors. So we need to do this in a continuous and automated manner – which is what you will learn in this article.

We'll go over how you can improve your MERN (MongoDB, Express, React, and NodeJs) app development process by setting up a CI/CD pipeline with Jenkins. You'll see how to automate deployment for faster, more efficient releases.

Let's Get Started

Prerequisites

• Basic understanding of MERN stack technologies.
• Basic understanding of Docker.
• Get source code from GitHub

The Problem

Consider this productivity app – it's a MERN project that we are going to use in this article. There are numerous steps we must complete, from building the application to pushing it to the Docker hub.

First, we must run tests with a command to determine whether all tests pass or not. If all tests pass, we build the Docker images and then push those images to Docker Hub. If your application is extremely complex, you may need to take additional steps.

Now, imagine that we're doing everything manually, which takes time and can lead to mistakes.

Waiting for deployment without devops meme

The Solution

To address this problem, we can create a CI/CD Pipeline. So, whenever you add a feature or fix a bug, this pipeline gets triggered. This automatically performs all of the steps from testing to deploying.

What is CI/CD and Why is it Important?

Continuous Integration and Continuous Deployment is a series of steps performed to automate software integration and deployment. CI/CD is the heart of DevOps.

CI/CD steps

From development to deployment, our MERN app goes through four major stages: testing, building Docker images, pushing to a registry, and deploying to a cloud provider. All of this is done manually by running various commands. And we need to do this every time a new feature is added or a bug is fixed.

But this will significantly reduce developer productivity, which is why CI/CD can be so helpful in automating this process. In this article, we will cover the steps up until pushing to the registry.

CI/CD meme

The Project

The project we are going to use in this tutorial is a very simple full-stack MERN application.

Project demo

It contains two microservices.

1. Frontend
2. Backend

You can learn more about the project here.

Both of these applications contains a Dockerfile. You can learn how to dockerize a MERN application here.

What is Jenkins?

To run a CI/CD pipeline, we need a CI/CD server. This is where all of the steps written in a pipeline run.

There are numerous services available on the market, including GitHub Actions, Travis CI, Circle CI, GitLab CI/CD, AWS CodePipeline, Azure DevOps, and Google Cloud Build. Jenkins is one of the popular CI/CD tools, and it's what we'll use here.

How to Set Up Jenkins Server on Azure

Because Jenkins is open source and it doesn't provide a cloud solution, we must either run it locally or self-host on a cloud provider. Now, running locally can be difficult, particularly for Windows users. As a result, I've chosen to self-host it on Azure for this demo.

If you want to run locally or self-host somewhere other than Azure (follow these guides by Jenkins), skip this section and proceed to the How to Configure Jenkins section.

First, you'll need to sign in to your Azure account (Create one if you don't have one already). Open Azure Cloud Shell.

Opening Azure Cloud Shell

Then create a directory called jenkins to store all the Jenkins config, and switch to that directory:

mkdir jenkins
cd jenkins

Create a file called cloud-init-jenkins.txt. Open with nano or vim,

touch cloud-init-jenkins.txt
nano cloud-init-jenkins.txt

and paste this code into it:

#cloud-config
package_upgrade: true
runcmd:
  - sudo apt install openjdk-11-jre -y
  - wget -qO - https://pkg.jenkins.io/debian-stable/jenkins.io.key | sudo apt-key add -
  - sh -c 'echo deb https://pkg.jenkins.io/debian-stable binary/ > /etc/apt/sources.list.d/jenkins.list'
  - sudo apt-get update && sudo apt-get install jenkins -y
  - sudo service jenkins restart

Here, we'll use this file to install Jenkins after creating a virtual machine. First, we install openjdk, which is required for Jenkins to function. The Jenkins service is then restarted after we install it.

Next, create a resource group. (A resource group in Azure is like a container that holds all the related resources of a project in one group. Learn more about resource groups here.)

az group create --name jenkins-rg --location centralindia

Note: make sure to change the location to the one closest to you.

Now, create a virtual machine.

az vm create \
--resource-group jenkins-rg \
--name jenkins-vm \
--image UbuntuLTS \
--admin-username "azureuser" \
--generate-ssh-keys \
--public-ip-sku Standard \
--custom-data cloud-init-jenkins.txt

You can verify the VM installation with this command:

az vm list -d -o table --query "[?name=='jenkins-vm']"

Don't be confused. This command simply displays JSON data in a tabular format for easy verification.

Jenkins server runs on port 8080, so we need to expose this port on our VM. You can do that like this:

az vm open-port \
--resource-group jenkins-rg \
--name jenkins-vm  \
--port 8080 --priority 1010

Now we can access the Jenkins dashboard in the browser with the URL http://<your-vm-ip>:8080. Use this command to get the VM IP address:

az vm show \
--resource-group jenkins-rg \
--name jenkins-vm -d \
--query [publicIps] \
--output tsv

You can now see the Jenkins application in your browser.

Jenkins dashboard

As you'll notice, Jenkins is asking us to provide an admin password which is automatically generated during its installation.

But first let's SSH into our virtual machine where Jenkins is installed.

ssh azureuser@<ip_address>

Now, type in the below command to get the password:

sudo cat /var/lib/jenkins/secrets/initialAdminPassword

Copy and paste it. Then click Continue.

How to Configure Jenkins

First, you'll need to click Install suggested plugins. It will take some time to install all the plugins.

Installing suggested plugins

An admin user is needed to restrict access to Jenkins. So go ahead and create one. After finishing, click Save and continue.

Create an admin user

Now you will be presented with the Jenkins dashboard.

The first step is to install the "Blue Ocean" plugin. Jenkins has a very old interface, which may make it difficult for some people to use. This blue ocean plugin provides a modern interface for some Jenkins components (like creating a pipeline).

To install plugins, go to Manage Jenkins -> click Manage Plugins under "System Configuration" -> Available plugins. Search for "Blue Ocean" -> check the box and click Download now and install after restart.

Blue ocean

Great, we're all set. Now let's create a pipeline.

How to Write a Jenkinsfile

To create a pipeline, we need a Jenkinsfile. This file contains all the pipeline configurations – stages, steps, and so on. Jenkinsfile is to Jenkins as a Dockerfile is to Docker.

Jenkinsfile uses the Groovy syntax. The syntax is very simple. You can understand everything by just looking at it.

Let's start by writing:

pipeline {

}

The word 'agent' should be the first thing you mention in the pipeline. An agent is similar to a container or environment in which jobs run. You can use multiple agents to run jobs in parallel. You can find more information about Jenkins agents can here.

pipeline {
    agent any
}

Here we are telling Jenkins to use any available agent.

We have a total of 5 stages in our pipeline:

CI/CD pipeline stages

Stage 1: Checkout code

Different CI/CD tools use different naming conventions. In Jenkins, these are referred to as stages. In each stage we write various steps.

Our first stage is checking out code from a source code management system (in our case, GitHub).

pipeline {
    agent any

    stages {
        stage('Checkout') {
            steps {
                checkout scm
            }
        }
    }
}

Commit the changes and push to your GitHub repo.

Since we haven't created any pipelines yet, let's do that now.

Before we begin, we must ensure that Git is installed on our system. If you followed my previous steps to install Jenkins on an Azure VM, Git is already installed.

You can test it by running the following command (make you are still SSHed into the VM):

git --version

If it isn't already installed, you can do so with:

sudo apt install git

Open blue ocean. Click Create new pipeline.

Creating new pipeline

Then select your source code management system. If you chose GitHub, you must provide an access token for Jenkins to access your repository. I recommend clicking on Create an access token here because it is a template with all of the necessary permissions. Then click Connect.

Selecting scm

After that, a pipeline will be created. Since our repository already contains a Jenkinsfile, Jenkins automatically detects it and runs the stages and steps we mentioned in the pipeline.

If everything went well, the entire page will turn green. (Other colors: blue indicates that the pipeline is running, red indicates that something went wrong in the pipeline, and gray indicates that we stopped the pipeline.)

Stage one successful

Stage 2: Run frontend tests

In general, all the CI/CD pipelines contains some tests that needs to be run before deploying. So I added simple tests to both the frontend and backend. Let's start with the frontend tests.

stage('Client Tests') {
    steps {
        dir('client') {
            sh 'npm install'
            sh 'npm test'
        }
    }
}

We're changing the directory to client/ because that's where the frontend code is. And then install the dependencies with npm install and run the tests with npm test in a shell.

Again, before we restart the pipeline, we have to make sure node and npm are installed or not. Install node and npm with these commands in the virtual machine:

curl -sL https://deb.nodesource.com/setup_16.x | sudo -E bash -

After that, run the following:

sudo apt-get install -y nodejs

Now, commit the code and restart the pipeline.

Run client tests

Stage 3: Run backend tests

Now do the same thing for the backend tests.

But there is one thing we need to do before we proceed. If you take a look at the codebase and activity.test.js, we are using a few environment variables. So let's add these environment varibales in Jenkins.

How to add environment variables in Jenkins

To add environment variables, go to Manage Jenkins -> click Manage Credentials under "Security" -> System -> Global credentials (unrestricted) -> click + Add Credentials.

For Kind select "Secret text", leave Scope default, and for Secret write the secret value and ID. This is what we use when using these environment variables in the Jenkinsfile.

Add the following env variables:

Environment variables

Then in the Jenkinsfile, use these env variables:

environment {
    MONGODB_URI = credentials('mongodb-uri')
    TOKEN_KEY = credentials('token-key')
    EMAIL = credentials('email')
    PASSWORD = credentials('password')
}

Add a stage to install dependencies, set these variables in the Jenkins environment, and run the tests:

stage('Server Tests') {
    steps {
        dir('server') {
            sh 'npm install'
            sh 'export MONGODB_URI=$MONGODB_URI'
            sh 'export TOKEN_KEY=$TOKEN_KEY'
            sh 'export EMAIL=$EMAIL'
            sh 'export PASSWORD=$PASSWORD'
            sh 'npm test'
        }
    }
}

Again, commit the code and restart the pipeline.

Run server tests

Stage 4: Build Docker images

Now, we have to specify a step to build the Docker images from the Dockerfiles.

Before we proceed, install Docker in the VM (if you don't already have it installed).

To install Docker:

sudo apt install docker.io

Add the user jenkins to the docker group so that Jenkins can access the Docker daemon – otherwise you'll get a permission denied error.

sudo usermod -a -G docker jenkins

Then restart the jenkins service.

sudo systemctl restart jenkins

Add a stage in the Jenkinsfile.

stage('Build Images') {
    steps {
        sh 'docker build -t rakeshpotnuru/productivity-app:client-latest client'
        sh 'docker build -t rakeshpotnuru/productivity-app:server-latest server'
    }
}

Commit the code and restart the pipeline.

Build docker images

Stage 5: Push images to the registry

As a final stage, we will push the images to Docker hub.

Before that, add your docker hub username and password to the Jenkins credentials manager, but for Kind choose "Username with password".

Username with password type credential

Add the final stage where we login and push images to Docker hub.

stage('Push Images to DockerHub') {
    steps {
        withCredentials([usernamePassword(credentialsId: 'dockerhub', passwordVariable: 'DOCKER_PASSWORD', usernameVariable: 'DOCKER_USERNAME')]) {
            sh 'docker login -u $DOCKER_USERNAME -p $DOCKER_PASSWORD'
            sh 'docker push rakeshpotnuru/productivity-app:client-latest'
            sh 'docker push rakeshpotnuru/productivity-app:server-latest'
        }
    }
}

Push images to dockerhub

Here is the complete Jenkinsfile:

// This is a Jenkinsfile. It is a script that Jenkins will run when a build is triggered.
pipeline {
    // Telling Jenkins to run the pipeline on any available agent.
    agent any

    // Setting environment variables for the build.
    environment {
        MONGODB_URI = credentials('mongodb-uri')
        TOKEN_KEY = credentials('token-key')
        EMAIL = credentials('email')
        PASSWORD = credentials('password')
    }

    // This is the pipeline. It is a series of stages that Jenkins will run.
    stages {
        // This state is telling Jenkins to checkout the source code from the source control management system.
        stage('Checkout') {
            steps {
                checkout scm
            }
        }

        // This stage is telling Jenkins to run the tests in the client directory.
        stage('Client Tests') {
            steps {
                dir('client') {
                    sh 'npm install'
                    sh 'npm test'
                }
            }
        }

        // This stage is telling Jenkins to run the tests in the server directory.
        stage('Server Tests') {
            steps {
                dir('server') {
                    sh 'npm install'
                    sh 'export MONGODB_URI=$MONGODB_URI'
                    sh 'export TOKEN_KEY=$TOKEN_KEY'
                    sh 'export EMAIL=$EMAIL'
                    sh 'export PASSWORD=$PASSWORD'
                    sh 'npm test'
                }
            }
        }

        // This stage is telling Jenkins to build the images for the client and server.
        stage('Build Images') {
            steps {
                sh 'docker build -t rakeshpotnuru/productivity-app:client-latest client'
                sh 'docker build -t rakeshpotnuru/productivity-app:server-latest server'
            }
        }

        // This stage is telling Jenkins to push the images to DockerHub.
        stage('Push Images to DockerHub') {
            steps {
                withCredentials([usernamePassword(credentialsId: 'dockerhub', passwordVariable: 'DOCKER_PASSWORD', usernameVariable: 'DOCKER_USERNAME')]) {
                    sh 'docker login -u $DOCKER_USERNAME -p $DOCKER_PASSWORD'
                    sh 'docker push rakeshpotnuru/productivity-app:client-latest'
                    sh 'docker push rakeshpotnuru/productivity-app:server-latest'
                }
            }
        }
    }
}

Pipeline ran successfully

Conclusion

In summary, let's review what we've covered:

• We explored the significance of implementing Continuous Integration and Continuous Deployment (CI/CD) in software development.
• We delved into the fundamentals of Jenkins and acquired knowledge on how to deploy a Jenkins server on the Azure cloud platform.
• We customized Jenkins to meet our specific requirements.
• Lastly, we wrote a Jenkinsfile and built a pipeline utilizing the user-friendly interface of Jenkins Blue Ocean.

That's all for now! Thanks for reading 🙂.

Connect with me on twitter.

How can we use GitHub Actions to automate and continuously deploy our app to S3?

• What are GitHub Actions?
• What is Continuous Deployment?
• What are we going to build?
• Step 0: Setting up a new Next.js project on GitHub
• Step 1: Manually creating and deploying a Next.js project to a new S3 Bucket
• Step 2: Creating a new GitHub Action workflow to automatically build a Next.js project
• Step 3: Configuring a GitHub Action to deploy a static website to S3

What are GitHub Actions?

GitHub Actions is a free service from GitHub that allows us to automate code tasks.

I previously wrote about how we can use them to automate tasks like running tests on our code and sending notifications to Slack.

They provide a flexible way to automatically run code based on our existing workflows. This provides a lot of possibilities like even deploying our website!

What is AWS S3?

S3 (Simple Storage Service) is an object storage service from AWS. It allows you to store files in the cloud easily making them available around the world.

It also allows you to use these files as a website. Because we can upload an HTML file as an object, we can also configure S3 to access that file as an HTTP request. This means that we can host an entire website right in S3.

What is Continuous Deployment?

Continuous Deployment, often referred to by its acronym CD, is the practice of maintaining code in a releasable state and deploying that code automatically or in short cycles.

Particularly in our use case, we’re going to configure our project so that any time a new update is pushed or merged to the primary Git branch, our website will deploy.

What are we going to build?

We’re first going to bootstrap a simple Next.js app using the default Next.js starting template and configure it to compile to static files.

If you don’t want to create a Next.js project, you can follow along with even a simple HTML file and not run any of the build commands. But Next.js is a modern way to build dynamic web apps, so we’ll start there.

With our website files ready to go, we’ll create and configure an S3 bucket in AWS where we’ll host our website.

Finally, we’ll create a new GitHub Action workflow that will automatically update the website files in S3 any time a new change occurs on our primary branch (main).

Step 0: Setting up a new Next.js project on GitHub

We’re going to get started with the default template with Next.js.

After navigating to the directory you want to create your project in, run:

yarn create next-app my-static-website
# or
npx create-next-app my-static-website

Note: Feel free to replace my-static-website with the name of your choice. We’ll use that for the rest of this tutorial.

If you navigate to that directory and run the development command, you should be able to successfully start up your development server.

cd my-static-website
yarn dev
# or
npm run dev

New Next.js App

Next, let’s configure our project to statically compile.

Inside the package.json file, update the build script to:

"build": "next build && next export",

What this will do is tell Next to take the website and export it to static files, which we’ll use to host the site.

We can test this out by running the command:

yarn build
# or
npm run build

And once finished, we can look inside of the out directory and see all of the files of our new website.

Static output from Next.js

Finally, we want to host this on GitHub.

Inside of your GitHub account, create a new repository. This will then provide instructions on how you can add an existing project to that repo.

And once you push our your project to GitHub, we should be ready to set up our new website project!

New repo in GitHub

Follow along with the commits:

• Adding the initial Next.js project via Create Next App
• Configuring Next.js to export the project

Step 1: Manually creating and deploying a Next.js project to a new S3 Bucket

To get started with our new S3 Bucket, first log in to your AWS account and navigate to the S3 service.

No buckets in S3

We’ll want to create a new bucket, using the name of our choice, which will be used for the S3 endpoint where our website is hosted. We’ll also want to configure our S3 bucket to be able to host a website.

Note: this tutorial will not walk you through how to host a website on S3, but you can check out my other tutorial that will walk you through hosting a website on S3 step-by-step.

Static website hosting in AWS S3

Once we have our S3 bucket configure as a website, we can go back to our Next.js project folder, run our build command, and then upload all of our files from the out directory into our new S3 bucket.

S3 Bucket with Static App

And once those files are uploaded and we’ve configured our S3 bucket for website hosting, we should now be able to see our project live on the web!

AWS S3 hosted Next.js app

Step 2: Creating a new GitHub Action workflow to automatically build a Next.js project

To get started, we’re going to need to create a new workflow.

If you’re familiar with GitHub Actions, you could create one manually, but we’ll quickly walk through how to do this in the UI.

Navigate to the Actions tab of your GitHub repository and click on "set up a workflow yourself."

New GitHub Action Workflow

GitHub provides a starting template that we can use for our workflow, though we’ll want to make some changes.

Let’s do the following:

• Optional: rename the file to deploy.yml
• Optional: rename the workflow to CD (as it’s a bit different from CI)
• Optional: remove all of the comments to make it a bit easier to read
• Remove the pull_request definition in the on property
• Remove all steps except for uses: actions/checkout@v2

So at this point we should be left with:

name: CD

on:
  push:
    branches: [ main ]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2

This code alone will trigger a process that spins up a new instance of Ubuntu and simply checks out the code from GitHub any time there’s a new change pushed to the main branch.

Next, once we have our code checked out, we want to build it. This will allow us to take that output and sync it to S3.

This step will differ slightly depending on if you are using yarn or npm for your project.

If you’re using yarn, under the steps definition, add the following:

- uses: actions/setup-node@v1
  with:
    node-version: 12
- run: npm install -g yarn
- run: yarn install --frozen-lockfile
- run: yarn build

If you’re using npm, add the following:

- uses: actions/setup-node@v1
  with:
    node-version: 12
- run: npm ci
- run: npm run build

Between both of these sets of steps, what we’re doing is:

• Setting up node: this is so that we can use npm and node to install and run our scripts
• Install Yarn (Yarn Only): if we’re using yarn, we install it as a global dependency so that we can use it
• Install Dependencies: we install our dependencies and we use a specific command that makes sure we use the lock file available to avoid any unexpected package upgrades
• Build: finally, we run our build command which will compile our Next.js project into the out directory!

And now we can commit that file right to our main branch which will kick off a new run of our workflow that we can see in our Actions tab.

New workflow in GitHub Actions

To see that it works, we can navigate into that run, select our workflow, and see that all of our steps ran including building our project!

Successful build logs for a GitHub Action workflow

Follow along with the commit!

Step 3: Configuring a GitHub Action to deploy a static website to S3

Now that we’re building our project automatically, we want to automatically update our website in S3.

To do that, we’re going to use the GitHub Action aws-actions/configure-aws-credentials and the AWS CLI.

The GitHub Action that we’re using will take in our AWS credentials and configuration and make it available to use throughout the lifecycle of the workflow.

As of now, the Ubuntu instance that GitHub Actions provides allows us to use the AWS CLI as it comes included. So we’ll be able to use the CLI commands right in our workflow.

Alternatively, we could use the S3 Sync action. But by using the AWS CLI, we gain more flexibility to customize our setup, we can use it for additional CLI commands, and it’s also generally nice to get familiar with the AWS CLI.

So to get started, let’s add the following snippet as additional steps in our workflow:

- uses: aws-actions/configure-aws-credentials@v1
  with:
    aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
    aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
    aws-region: us-east-1

What the above will do is use the AWS credentials configuration action to set up our AWS Access Key, Secret Key, and region based on our settings.

The AWS Region can be customized to whatever region you typically use with your AWS account. I’m in the northeast United States, So I’ll keep us-east-1.

The Access Key and Secret Key are credentials that you’ll need to generate with your AWS account. The way our code is set up is that we’ll store those values inside of GitHub Secrets, which will prevent those keys from being leaked. When the action runs, Github changes those values to stars (***) so people can't access those keys.

So to set up those secrets, we first want to generate Access Keys in AWS.

Navigate to the AWS console. Under the user menu, select My Security Credentials, and then select Create access key.

Creating an Access Key in AWS

This will provide you with two values: the Access key ID and the Secret access key. Save these values, as you won’t be able to access the Secret key ID again.

Finding Secret and Access Key in AWS

Note: remember to NOT include the Access Key and Secret Key inside of your code. This could lead to someone compromising your AWS credentials.

Next, inside of the GitHub repo, navigate to Settings, Secrets, then select New secret.

Here we’ll want to add our AWS keys using the following secrets:

• AWS_ACCESS_KEY_ID: your AWS Access key ID
• AWS_SECRET_ACCESS_KEY: your AWS Secret key

And once saved you should have your two new secrets.

Creating Secrets in GitHub

Now that we have our credentials configured, we should be ready to run the command to sync our project to S3.

Inside of the GitHub Action, add the following step:

- run: aws s3 sync ./out s3://[bucket-name]

Note: be sure to replace [bucket-name] with the name of your S3 Bucket.

This command will trigger a sync with our specified S3 bucket, using the contents of the out directory, which is where our project builds to.

And now, if we commit our changes, we can see that our action is automatically triggered once committed to the main branch, where we build our project and sync it to S3!

Successful AWS S3 sync in GitHub Action workflow

Note: Make sure that before setting up this action you’ve configured the S3 bucket to host a website (including unblocking permissions on S3 bucket) – otherwise this action may fail.

At this point, our project probably looks the same, as we didn’t make any changes to the code.

Next.js app on AWS S3

But if you make a code change, such as changing the title of the homepage inside of pages/index.js and commit that change:

<h1 className={styles.title}>
  Colby's <a href="https://nextjs.org">Next.js!</a> Site
</h1>

We can see that our change triggers the workflow to kick off:

New GitHub Action workflow from code change

And once our workflow finishes, we can see that our content is now automatically updated on our website:

AWS S3 hosted app with updated code changes

Follow along with the commits:

• Adding AWS configuration and S3 sync command
• Title update to test workflow

What else can we do?

Setting up CloudFront

The goal of this post wasn’t to go through the entire process of configuring a website for AWS, but if you’re serving a website on S3, you might want to also include CloudFront in front of it.

You can check out my other guide here which walks you through setting up CloudFront as well as a step-by-step guide through creating the site in S3.

Invaliding CloudFront cache

If your S3 website is behind CloudFront, chances are, you’ll want to make sure CloudFront isn’t caching the new changes.

With the AWS CLI, we can also trigger a cache invalidation with CloudFront to make sure it’s grabbing the latest changes.

Check out the docs here to learn more.

Pull request deployments

If you’re constantly working on website changes in a pull request, sometimes it can be easier to see the changes live.

You can set up a new workflow that only runs on pull requests, where the workflow can dynamically create a new bucket based on the branch or environment and add a comment to the pull request with that URL.

You might be able to find a GitHub Action that exists to manage the comments on the pull request for you or you can check out the GitHub Actions docs.

But automation can be tough and can sometimes prove to be costly. How can Github Actions help harden our code and give us more time to work on features instead of bugs?

• What are Github Actions?
• What is CI/CD?
• What are we going to build?
• Part 0: Setting up a project
• Part 1: Automating tests
• Part 2: Post new pull requests to Slack

What are Github Actions?

Actions are a relatively new feature to Github that allow you to set up CI/CD workflows using a configuration file right in your Github repo.

Previously, if you wanted to set up any kind of automation with tests, builds, or deployments, you would have to look to services like Circle CI and Travis or write your own scripts. But with Actions, you have first class support to powerful tooling to automate your workflow.

What is CI/CD?

CD/CD stands for Continuous Integration and Continuous Deployment (or can be Continuous Delivery). They're both practices in software development that allow teams to build projects together quickly, efficiently, and ideally with less errors.

Continuous Integration is the idea that as different members of the team work on code on different git branches, the code is merged to a single working branch which is then built and tested with automated workflows. This helps to constantly make sure everyone's code is working properly together and is well-tested.

Continuous Deployment takes this a step further and takes this automation to the deployment level. Where with the CI process, you automate the testing and the building, Continuous Deployment will automate deploying the project to an environment.

The idea is that the code, once through any building and testing processes, is in a deployable state, so it should be able to be deployed.

What are we going to build?

We're going to tackle two different workflows.

The first will be to simply run some automated tests that will prevent a pull request from being merged if it is failing. We won't walk through building the tests, but we'll walk through running tests that already exist.

In the second part, we'll set up a workflow that sends a message to slack with a link to a pull request whenever a new one is created. This can be super helpful when working on open source projects with a team and you need a way to keep track of requests.

Part 0: Setting up a project

For this guide, you can really work through any node-based project as long as it has tests you can run for Part 1.

If you'd like to follow along with a simpler example that I'll be using, I've set up a new project that you can clone with a single function that has two tests that are able to run and pass.

A function with two tests

If you'd like to check out this code to get started, you can run:

git clone --single-branch --branch start git@github.com:colbyfayock/my-github-actions.git

Once you have that cloned locally and have installed the dependencies, you should be able to run the tests and see them pass!

Passing tests

It should also be noted that you'll be required to have this project added as a new repository on Github in order to follow along.

Follow along with the commit!

Part 1: Automating tests

Tests are an important part of any project that allow us to make sure we're not breaking existing code while we work. While they're important, they're also easy to forget about.

We can remove the human nature out of the equation and automate running our tests to make sure we can't proceed without fixing what we broke.

Step 1: Creating a new action

The good news, is Github actually makes it really easy to get this workflow started as it comes as one of their pre-baked options.

We'll start by navigating to the Actions tab on our repository page.

Github Actions starting page

Once there, we'll immediately see some starter workflows that Github provides for us to dive in with. Since we're using a node project, we can go ahead and click Set up this workflow under the Node.js workflow.

Setting up a Node.js Github Action workflow

After the page loads, Github will land you on a new file editor that already has a bunch of configuration options added.

We're actually going to leave this "as is" for our first step. Optionally, you can change the name of the file to tests.yml or something you'll remember.

Adding a new Github Action workflow file

You can go ahead and click Start commit then either commit it directory to the master branch or add the change to a new branch. For this walkthrough, I'll be committing straight to master.

To see our new action run, we can again click on the Actions tab which will navigate us back to our new Actions dashboard.

Viewing Github Action workflow events

From there, you can click on Node.js CI and select the commit that you just made above and you'll land on our new action dashboard. You can then click one of the node versions in the sidebar via build (#.x), click the Run npm test dropdown, and we'll be able to see the output of our tests being run (which if you're following along with me, should pass!).

Viewing logs of a Github Action workflow

Follow along with the commit!

Step 2: Configuring our new action

So what did we just do above? We'll walk through the configuration file and what we can customize.

Github Action Node.js workflow file

Starting from the top, we specify our name:

name: Node.js CI

This can really be whatever you want. Whatever you pick should help you remember what it is. I'm going to customize this to "Tests" so I know exactly what's going on.

on:
  push:
    branches: [ master ]
  pull_request:
    branches: [ master ]

The on key is how we specify what events trigger our action. This can be a variety of things like based on time with cron. But here, we're saying that we want this action to run any time someone pushes commits to master or someone creates a pull request targeting the master branch. We're not going to make a change here.

jobs:
  build:
    runs-on: ubuntu-latest

This next bit creates a new job called build. Here we're saying that we want to use the latest version of Ubuntu to run our tests on. Ubuntu is common, so you'll only want to customize this if you want to run it on a specific environment.

    strategy:
      matrix:
        node-version: [10.x, 12.x, 14.x]

Inside of our job we specify a strategy matrix. This allows us to run the same tests on a few different variations.

In this instance, we're running the tests on 3 different versions of node to make sure it works on all of them. This is definitely helpful to make sure your code is flexible and future proof, but if you're building and running your code on a specific node version, you're safe to change this to only that version.

    steps:
    - uses: actions/checkout@v2
    - name: Use Node.js ${{ matrix.node-version }}
      uses: actions/setup-node@v1
      with:
        node-version: ${{ matrix.node-version }}
    - run: npm ci
    - run: npm run build --if-present
    - run: npm test

Finally, we specify the steps we want our job to run. Breaking this down:

• uses: actions/checkout@v2: In order for us to run our code, we need to have it available. This checks out our code on our job environment so we can use it to run tests.
• uses: actions/setup-node@v1: Since we're using node with our project, we'll need it set up on our environment. We're using this action to do that setup for us for each version we've specified in the matrix we configured above.
• run: npm ci: If you're not familiar with npm ci, it's similar to running npm install but uses the package-lock.json file without performing any patch upgrades. So essentially, this installs our dependencies.
• run: npm run build --if-present: npm run build runs the build script in our project. The --if-present flag performs what it sounds like and only runs this command if the build script is present. It doesn't hurt anything to leave this in as it won't run without the script, but feel free to remove this as we're not building the project here.
• run: npm test: Finally, we run npm test to run our tests. This uses the test npm script set up in our package.json file.

And with that, we've made a few tweaks, but our tests should run after we've committed those changes and pass like before!

Logs of passing tests in Github Action workflow

Follow along with the commit!

Step 3: Testing that our tests fail and prevent merges

Now that our tests are set up to automatically run, let's try to break it to see it work.

At this point, you can really do whatever you want to intentionally break the tests, but here's what I did:

Code diff - https://github.com/colbyfayock/my-github-actions/pull/1

I'm intentionally returning different expected output so that my tests will fail. And they do!

Failing status checks on pull request

In my new pull request, my new branch breaks the tests, so it tells me my checks have failed. If you noticed though, it's still green to merge, so how can we prevent merges?

We can prevent pull requests from being merged by setting up a Protected Branch in our project settings.

First, navigate to Settings, then Branches, and click Add rule.

Github branch protection rules

We'll then want to set the branch name pattern to *, which means all branches, check the Require status checks to pass before merging option, then select all of our different status checks that we'd like to require to pass before merging.

Setting up a branch protection rule in Github

Finally, hit Create at the bottom of the page.

And once you navigate back to the pull request, you'll notice that the messaging is a bit different and states that we need our statuses to pass before we can merge.

Failing tests preventing merge in pull request

Note: as an administrator of a repository, you'll still be able to merge, so this technically only prevents non-administrators from merging. But will give you increased messaging if the tests fail.

And with that, we have a new Github Action that runs our tests and prevents pull requests from merging if they fail.

Follow along with the pull request!

Note: we won't be merging that pull request before continuing to Part 2.

Part 2: Post new pull requests to Slack

Now that we're preventing merge requests if they're failing, we want to post a message to our Slack workspace whenever a new pull request is opened up. This will help us keep tabs on our repos right in Slack.

For this part of the guide, you'll need a Slack workspace that you have permissions to create a new developer app with and the ability to create a new channel for the bot user that will be associated with that app.

Step 1: Setting up Slack

There are a few things we're going to walk through as we set up Slack for our workflow:

• Create a new app for our workspace
• Assign our bot permissions
• Install our bot to our workspace
• Invite our new bot to our channel

To get started, we'll create a new app. Head over to the Slack API Apps dashboard. If you already haven't, log in to your Slack account with the Workspace you'd like to set this up with.

Creating a new Slack app

Now, click Create New App where you'll be prompted to put in a name and select a workspace you want this app to be created for. I'm going to call my app "Gitbot" as the name, but you can choose whatever makes sense for you. Then click Create App.

Configuring a new Slack app

Once created, navigate to the App Home link in the left sidebar. In order to use our bot, we need to assign it OAuth scopes so it has permissions to work in our channel, so select Review Scopes to Add on that page.

Reviewing Slack app scopes

Scroll own and you'll see a Scopes section and under that a Bot Token section. Here, click Add an OAuth Scope. For our bot, we don't need a ton of permissions, so add the channels:join and chat:write scopes and we should be good to go.

Adding scopes for a Slack app Bot Token

Now that we have our scopes, let's add our bot to our workspace. Scroll up on that same page to the top and you'll see a button that says Install App to Workspace.

Installing Slack app to a workspace

Once you click this, you'll be redirected to an authorization page. Here, you can see the scopes we selected for our bot. Next, click Allow.

Allowing permission for Slack app to be installed to workspace

At this point, our Slack bot is ready to go. At the top of the OAuth & Permissions page, you'll see a Bot User OAuth Access Token. This is what we'll use when setting up our workflow, so either copy and save this token or remember this location so you know how to find it later.

Note: this token is private - don't give this out, show it in a screencast, or let anyone see it!

Copying OAuth Access Token for Slack bot user

Finally, we need to invite our Slack bot to our channel. If you open up your workspace, you can either use an existing channel or create a new channel for these notifications, but you'll want to enter the command /invite @[botname] which will invite our bot to our channel.

Inviting Slack bot user to channel

And once added, we're done with setting up Slack!

Slack bot was added to channel

Create a Github Action to notify Slack

Our next step will be somewhat similar to when we created our first Github Action. We'll create a workflow file which we'll configure to send our notifications.

While we can use our code editors to do this by creating a file in the .github directory, I'm going to use the Github UI.

First, let's navigate back to our Actions tab in our repository. Once there, select New workflow.

Setting up a new Github Action workflow

This time, we're going to start the workflow manually instead of using a pre-made Action. Select set up a workflow yourself at the top.

Setting up a Github Action workflow manually

Once the new page loads, you'll be dropped in to a new template where we can start working. Here's what our new workflow will look like:

name: Slack Notifications

on:
  pull_request:
    branches: [ master ]

jobs:
  notifySlack:

    runs-on: ubuntu-latest

    steps:
    - name: Notify slack
      env:
        SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }}
      uses: abinoda/slack-action@master
      with:
        args: '{\"channel\":\"[Channel ID]\",\"blocks\":[{\"type\":\"section\",\"text\":{\"type\":\"mrkdwn\",\"text\":\"*Pull Request:* ${{ github.event.pull_request.title }}\"}},{\"type\":\"section\",\"text\":{\"type\":\"mrkdwn\",\"text\":\"*Who?:* ${{ github.event.pull_request.user.login }}\n*Request State:* ${{ github.event.pull_request.state }}\"}},{\"type\":\"section\",\"text\":{\"type\":\"mrkdwn\",\"text\":\"<${{ github.event.pull_request.html_url }}|View Pull Request>\"}}]}'

So what's happening in the above?

• name: we're setting a friendly name for our workflow
• on: we want our workflow to trigger when there's a pull request is created that targets our master branch
• jobs: we're creating a new job called notifySlack
• jobs.notifySlack.runs-on: we want our job to run on a basic setup of the latest Unbuntu
• jobs.notifySlack.steps: we really only have one step here - we're using a pre-existing Github Action called Slack Action and we're configuring it to publish a notification to our Slack

There are two points here we'll need to pay attention to, the env.SLACK_BOT_TOKEN and the with.args.

In order for Github to communicate with Slack, we'll need a token. This is what we're setting in env.SLACK_BOT_TOKEN. We generated this token in the first step. Now that we'll be using this in our workflow configuration, we'll need to add it as a Git Secret in our project.

_Github secrets including SLACK_BOTTOKEN

The with.args property is what we use to configure the payload to the Slack API that includes the channel ID (channel) and our actual message (blocks).

The payload in the arguments is stringified and escaped. For example, when expanded it looks like this:

{
  "channel": "[Channel ID]",
  "blocks": [{
    "type": "section",
    "text": {
      "type": "mrkdwn",
      "text": "*Pull Request:* ${{ github.event.pull_request.title }}"
    }
  }, {
    "type": "section",
    "text": {
      "type": "mrkdwn",
      "text": "*Who?:*n${{ github.event.pull_request.user.login }}n*State:*n${{ github.event.pull_request.state }}"
    }
  }, {
    "type": "section",
    "text": {
      "type": "mrkdwn",
      "text": "<${{ github.event.pull_request._links.html.href }}|View Pull Request>"
    }
  }]
}

Note: this is just to show what the content looks like, we need to use the original file with the stringified and escaped argument.

Back to our configuration file, the first thing we set is our channel ID. To find our channel ID, you'll need to use the Slack web interface. Once you open Slack in your browser, you want to find your channel ID in the URL:

https://app.slack.com/client/[workspace ID]/[channel ID]

Channel ID in Slack web app URL

With that channel ID, you can modify our workflow configuration and replace [Channel ID] with that ID:

with:
  args: '{\"channel\":\"C014RMKG6H2\",...

The rest of the arguments property is how we set up our message. It includes variables from the Github event that we use to customize our message.

We won't go into tweaking that here, as what we already have will send a basic pull request message, but you can test out and build your own payload with Slack's Block Kit Builder.

Follow along with the commit!

Test out our Slack workflow

So now we have our workflow configured with our Slack app, finally we're ready to use our bot!

For this part, all we need to do is create a new pull request with any change we want. To test this out, I simply created a new branch where I added a sentence to the README.md file.

Code diff - https://github.com/colbyfayock/my-github-actions/pull/2

Once you create that pull request, similar to our tests workflow, Github will run our Slack workflow! You can see this running in the Actions tab just like before.

As long as you set everything up correctly, once the workflow runs, you should now have a new message in Slack from your new bot.

Slack bot automated message about new pull request

Note: we won't be merging that pull request in.

What else can we do?

Customize your Slack notifications

The message I put together is simple. It tells us who created the pull request and gives us a link to it.

To customize the formatting and messaging, you can use the Github Block Kit Builder to create your own.

If you'd like to include additional details like the variables I used for the pull request, you can make use of Github's available contexts. This lets you pull information about the environment and the job to customize your message.

I couldn't seem to find any sample payloads, so here's an example of a sample github context payload you would expect in the event.

Sample github context

More Github actions

With our ability to create new custom workflows, that's not a lot we can't automate. Github even has a marketplace where you can browse around for one.

If you're feeling like taking it a step further, you can even create your own! This lets you set up scripts to configure a workflow to perform whatever tasks you need for your project.

Join in the conversation!

What do you use Github actions for?

Share with me on Twitter!

How can we configure this so we don't have to directly edit our code to change our environment?

• What are environment variables?
• How can environment variables be useful?
• How can I keep these files secure?
• Gatsby and environment variables
• Netlify and environment variables
• Step 1: Creating a "Hello, world" website
• Step 2: Creating a local environment variable with Gatsby
• Step 3: Deploying the website to Netlify
• Where can you add or update more variables in Netlify?

What are environment variables?

Environment variables are predetermined values that are typically used to provide the ability to configure a value in your code from outside of your application.

_MY_SECRETKEY environment variable used for authorization

When developing locally, or sometimes even in a deployment pipeline, you'll oftentimes find these variables stored in a file named with some kind of variation of .env.

How can environment variables be useful?

Probably the most common use case for environment variables is being able to set up different configuration options for different environments. Often when developing against third party services, you want to have a development version or sandbox available to make test requests against, that way it doesn't impact real production data.

Environment variables are helpful because they allow you to change which of your environments use which third party service environment by changing an API key, endpoint, or whatever the service uses to distinguish between environments.

The code you deploy should be predictable, so by not having to change any code, just the configuration outside of the code, you can maintain that predictability.

How can I keep these files secure?

This is probably one of the more important points here – you need to ensure you're handling these files with care and not checking them into a git repository. By exposing these keys by inadvertently uploading them to a public location, the internet could easily find these keys and abuse them for their own gains.

For instance, AWS keys are a valuable source. People run bots with the sole purpose of trying to scan Github for keys. If someone finds an AWS key, they could use this key to access resources such as running a bitcoin operation at your expense. This isn't to scare you, its to make you aware so you avoid your keys getting compromised.

So how can we keep these secure? The easiest way is to add the environment file where you keep these keys to your .gitignore file.

To do this, simply open your existing .gitignore file or create a new one at the root of your repository and add the filename as a new line:

# Inside .gitignore
.env

If you want to get more advanced and make sure this never happens to a repository, you can check out some tools like git-secrets from AWS Labs or GitLeaks that even has a Github Action to make it easy to integrate with Github.

Gatsby and environment variables

Gatsby by default makes two files available as part of its environment variable workflow that makes these values available in the client: .env.development and .env.production. These correlate to the gatsby develop and gatsby build scripts to either develop or build your site.

_MY_SECRETKEY environment variable for development and production

To make use of these files within the Gatsby development and build process, Gatsby requires you to prefix these variables with GATSBY_. This also works if you'd like to have them available from an OS process level.

Though you could integrate dotenv if you have more advanced needs or don't want to use the GATSBY_ prefix, your path of least resistance is probably to just follow the Gatsby way when working in Gatsby.

Netlify and environment variables

Netlify provides the ability to add environment variables as part of its Build & deploy settings which gets picked up as part of the build processes.

Adding an environment variable in Netlify

Luckily, Netlify makes it easy to add whatever environment variable you'd like to the build process! To add one, you can simply navigate to the Environment section of your project's Build & deploy settings page and add a variable under Environment variables.

We'll walk you through this process a little later.

Step 1: Creating a "Hello, world" website

For our walkthrough, we're going to set up a really basic example of a Gatsby website just for the purposes of testing this out.

New website with Gatsby Sass Starter

Though this isn't really a common use case of environment variables, where normally you would use them for things like API keys and service configurations, this will give you a great idea of how it fundamentally works.

We're going to use this Gatsby Sass Starter I created which will give us a starting point and add "Hello, [Environment]" depending on where it's running.

To get started, let's create our local project by using the Gatsby CLI. Navigate to where you'd like to store this project and run:

gatsby new my-env-project https://github.com/colbyfayock/gatsby-starter-sass

You can change my-env-project to whatever directory you'd like this project created in, but once you run this command, you'll now have a project in that new directory.

New Gatsby project in the terminal

To get started, once inside that directory, run yarn develop to make changes locally or yarn build to compile your new site.

Once you're ready to go, you'll want to add this project to Github. If you're not familiar with how to do this, you can learn how to add an existing project to Github here.

Step 2: Creating a local environment variable with Gatsby

Our next step is to create a local environment and add a change that will let us see that it works.

To get started, let's first create a new file at the root of our project called .env.development. It might ask you if you really want to use the . prefix, make sure you say yes!

Inside that file, let's add:

# Inside .env.development
GATSBY_MY_ENVIRONMENT="Development"

Creating an .env.development file

Next, to make sure we don't forget to do this, let's also add this .env.development file to our .gitignore so we don't accidentally commit this to our git history. If you don't already have a .gitignore file, make sure you create it at the root of your project.

Adding .env.development to your .gitignore

Finally, to check that this works, let's open pages/index.js and let's replace our <h1> tag's content with a "Hello, world!" variation:

<h1>Hello, {process.env.GATSBY_MY_ENVIRONMENT}</h1>

And if we save that change and open it in our browser, we should see "Hello, Development"!

Using an environment variable for your Gatsby site

Follow along with the commit!

Step 3: Deploying the website to Netlify

So we have our website created using a simple environment variable. Next we'll want to actually deploy that site to Netlify. If you haven't already, we'll need to add our website to Github or another Git provider. Make sure to have that set up before continuing on.

After creating an account and logging in to Netlify, let's click the New site from Git button the main dashboard, follow the instructions for connecting your Github or other Git provider to Netlify, and then find your new repository.

Adding a new Github repository to Netlify

Once you select your repository, you'll be asked to configure your build process. Luckily, Netlify can detect that we're using a Gatsby site and has it pre-filled for us. Unless you've added something special, keep the basic configuration to use gatsby build to build your project and public/ for the output.

Configuring Netlify build settings

Now before we hit Deploy, there's one thing we want to add, and that's our environment variable!

Right above the Deploy site button there's an Advanced button. Click that and you'll see a new dropdown with an additional New variable button.

Configuring an environment variable in the Netlify setup

Click that New variable button, add our GATSBY_MY_ENVIRONMENT as a new variable and add Production as the value. And finally, hit Deploy site!

From here, you should be able to watch your website deploy and once finished, you'll see your new site with "Hello, Production"!

Deployed Gatsby site using Netlify environment variable

Where can you add or update more variables in Netlify?

With our example, we only added one variable during the setup. But Netlify lets you add or update any other variables you'd like.

If you'd ever like to change that variable or add more, you can navigate to the Environment section of the Build & deploy settings, where you can edit and add any other variables in the Environment variables section.

Environment variables settings in Netlify

Looking to learn more?

Here are a few other things to help you get started with development fundamentals!

• What is Gatsby and why it's time to get on the hype train?
• What is the JAMstack and how do I get started?
• How to Become a Full Stack Web Developer in 2020
• Put Down the Javascript - Learn HTML & CSS
• Set Future You Up for Success with Good Coding Habits

There is plenty of content out there describing what Continuous Integration, Continuous Delivery, and Continuous Deployment are. But what purposes do these processes serve in the first place?

It is crucial to understand the problems CI and CD solve to use them properly. This will allow your team to improve your process and avoid putting effort into chasing fancy metrics that do not bring any value to your process.

Continuous Integration is a team problem

If you work in a team, chances are there are several developers working on the same repository. There is a main branch in the repository carrying the latest version of the code. Developers work on different things on different branches. Once someone is done with their change, they'll push or merge it to the main branch. Eventually the whole team will pull this change.

The scenario we want to avoid is that a faulty commit makes it to the main branch. Faulty means the code does not compile or the app won't start or is unusable. Why? Not because the app is broken or because all tests must always be green. That is not a problem–you can decide not to deploy that version and wait for a fix.

The problem is that your entire team is stuck. All the developers who pulled the faulty commit will spend 5 minutes wondering why it doesn't work. Several will probably try to find the faulty commit. Some will try to fix the issue by themselves in parallel of the faulty code author.

This is a waste of time for your team. The worst part is that repeated incidents fuel a mistrust of the main branch and encourage developers to work apart.

Continuous Integration is all about preventing the main branch from breaking so your team is not stuck. That's it. It is not about having all your tests green all the time and the main branch deployable to production at every commit.

The process of Continuous Integration is independent of any tool. You could manually verify that the merge of your branch and the main branch works locally, and then only actually push the merge to the repository. But that would be very inefficient. That's why Continuous Integration is implemented using automated checks.

The checks ensure that, at the bare minimum:

• The app should build and start
• Most critical features should be functional at all times (user signup/login journey and key business features)
• Common layers of the application that all the developers rely on should be stable. This means unit tests on those parts.

In practice, this means you need to pull any unit test framework that works for you and secure the common layers of the application. Sometimes it is not that much code and can be done fairly quickly. Also you need to add a "smoke test" verifying that the code compiles and that the application starts. This is especially important in technologies with crazy dependency injections like Java Spring or .NET core. In large projects it is so easy to miswire your dependencies that verifying that the app always starts is a must.

If you have hundreds or thousands of tests you don't need to run them all for each merge. It will take a lot of time and most tests probably verify "non team blocker" features.

We'll see in the next sections how the process of Continuous Delivery will make good use of these many tests.

It's not about tools

Tools and automated checks are all fine. But if your developers only merge giant branches they work on for weeks, they won't help you. The team will spend a good amount of time merging the branches and fixing the code incompatibilities that will arise eventually. It is as much a waste of time as being blocked by a faulty commit.

Continuous Integration is not about tools. It is about working in small chunks and integrating your new code to the main branch and pulling frequently.

Frequently means at least daily. Split the task you are working on into smaller tasks. Merge your code very often and pull very often. This way nobody works apart for more than a day or two and problems do not have time to become snowballs.

A large task does not need to be all in one branch. It should never be. Techniques to merge work in progress to the main branch are called "branching by abstraction" and "feature toggles". See the blog post How to get started with Continuous Integration for more details.

Key points for a good CI build

It's very simple. Keep it short. 3-7 minutes should be the max. It's not about CPU and resources. It is about developers' productivity. The first rule of productivity is focus. Do one thing, finish it, then move to the next thing.

Context switching is costly. Studies show it takes ~23 minutes to deeply refocus on something when you get disturbed.

Imagine you push your branch to merge it. You start another task. You spend 15-20 minutes getting into it. The minute after you are in the zone you receive a "build failed" notification from your 20 minutes long CI build for the previous task. You get back to fix it. You push it again. You easily lost more than 20 minutes moving back and forth.

Multiply 20 minutes once or twice a day by the number of developers in your team... That's a lot of precious time wasted.

Now imagine if the feedback came within 3 minutes. You probably wouldn't have started the new task at all. You would have proof read your code one more time or reviewed a PR while waiting. The failed notification would come and you would fix it. Then you could move on to the next task. That is the kind of focus your process should enable.

Keeping your CI build short makes it a trade off. Tests that run longer or provide little value in the context of CI should be moved to the CD step. And yes, failures there also need to be fixed. But since they are not preventing anybody from doing their thing, you can take the fixes as a "next task" when you finish what you are doing. Just turn off the notifications while working and check every now and then. Keep the context switching to a minimum.

Continuous Delivery and Deployment are engineering problems

Let’s settle on the definitions to get that out of the way.

Continuous Delivery is about being able to deploy any version of your code at all times. In practice it means the last or pre-last version of your code. You don’t deploy automatically, usually because you don’t have to or are limited by your project lifecycle. But as soon as someone feels like it, a deployment can be done in a minimal amount of time. That someone can be the test/QA team that wants to test things out on a staging or pre-production environment. Or it can actually be time to roll out the code to production.

The idea of Continuous Delivery is to prepare artifacts as close as possible from what you want to run in your environment. These can be .jar or .war files if you are working with Java, or executables if you are working with .NET. These can also be folders of transpiled JS code or even Docker containers, whatever makes deployment shorter (i.e. you have pre-built as much as you can in advance).

By preparing artifacts, I don't mean turning code into artifacts. This is usually a few scripts and minutes of execution. Preparing means:

Run all the tests you can to ensure that, once deployed, the code will actually work. Run unit tests, integration tests, end to end tests, and even performance tests if you can automate that.

This way you can filter which versions of your main branch are actually production ready and which are not. The ideal test suite:

• Ensures that the application's key functionalities work. Ideally all functionalities
• Ensures that no performance deal breaker has been introduced so when your new version hits your many users, it has a chance to last
• Dry run any database updates your code needs to avoid surprises

It does not need to be very fast. 30 minutes or 1 hour is acceptable.

Continuous Deployment is the next step. You deploy the most up to date and production ready version of your code to some environment. Ideally production if you trust your CD test suite enough.

Note that, depending on the context, this is not always possible or worth the effort. Continuous Delivery is often enough to be productive, especially if you are working in a close network and have limited environments you can deploy to. It can also be that the release cycle of your software prevents unplanned deploys.

Continuous Delivery and Continuous Deployment (let’s call them CD from now on) are not team problems. They are about finding the right balance between execution time, maintenance efforts and relevance of your tests suite to be able to say "This version works as it should."

And it is a balance. If your tests last 30 hours that is a problem. See this epic post about what the Oracle database test suite looks like. But if you spend so much time keeping your tests up to date with the latest code that it impedes the team's progress, that is not good either. Also if your test suite ensures pretty much nothing... it is basically useless.

In an ideal world we want one set of deployable artifacts per commit to the main branch. You can see we have a vertical scalability problem: the faster we move from code to artifacts, the more ready we are to deploy the newest version of the code.

What’s the big difference?

Continuous Integration is a horizontal scalability problem. You want developers to merge their code often so the checks must be fast. Ideally within minutes to avoid developers switching context all the time with highly async feedback from the CI builds.

The more developers you have, the more computing power you need to run simple checks (build and test) on all the active branches.

A good CI build:

Ensures no code that breaks basic stuff and prevents other team members to work is introduced to the main branch, and

Is fast enough to provide feedback to developers within minutes to prevent context switching between tasks.

Continuous Delivery and Deployment are vertical scalability problems. You have one rather complex operation to perform.

A good CD build:

Ensures that as many features as possible are working properly.

The faster the better, but it is not a matter of speed. A 30-60 minutes build is OK.

A common misconception is to see CD as a horizontal scalability problem like CI: the faster you can move from code to artifacts, the more commits you can actually process, and the closer to the ideal scenario you can be.

But we don't need that. Producing artifacts for every commit and as fast as possible is usually overkill. You can very well approach CD on a best effort basis: have a single CD build that will just pick the latest commit to verify once a given build is finished.

Make no mistake about CD. It is really hard. Getting to sufficient test confidence to say your software is ready to be deployed automatically usually works on low surface applications like APIs or simple UIs. It is very difficult to achieve on a complex UI or a large monolith system.

Conclusion

Tools and principles used to execute CI and CD are often very similar. The goals are very different though.

Continuous Integration is a trade off between speed of feedback loop to developers and relevance of the checks your perform (build and test). No code that would impede the team progress should make it to the main branch.

Continuous Delivery of Deployment is about running as thorough checks as you can to catch issues on your code. Completeness of the checks is the most important factor. It is usually measured in terms code coverage or functional coverage of your tests. Catching errors early on prevents broken code to get deployed to any environment and saves the precious time of your test team.

Craft your CI and CD builds to achieve these goals and keep your team productive. No workflow is perfect. Problems will arise every now and then. Use them as lessons learned to strengthen your workflow every time they do.

Published on 27 Nov 2019 on the Fire CI Blog.

Disclaimer: This story is not sponsored by any of the tools that has been described into the article (Travis-CI, Github, Github-Pages)

You have created a project in React.js and deployed it on the GitHub-pages (not yet ?? — create your first project in React.js) But what if you are making frequent changes into the code base and also want to keep the deployed version updated to the latest ? … You will find yourself in the tedious process of running the deployment scripts again and again !!!

What if the deployment process can be automated ??

After some quick google search session, I found that it is possible and can be achieved by Travis CI — an open source tool can be used to automate the deployment of various types of projects.

What you will learn >

In this article, you will be able to learn how to implement the system which will trigger the react deployment scripts using the TRAVIS-CI to deploy the project onto the GitHub-pages whenever there are any changes found in the master branch of the code repository.

• Setup Automated deployment of ‘react-portfolio’ project
• Learn about some frequent errors encountered while the process
• Learn about some concepts related to ‘continuous deployment’

Let’s learn some fundamentals

Skip this section if you know you are not that type !!

Continuous Integration(CI) & Continuous Delivery(CD) >

“In software engineering, continuous integration (CI) is the practice of merging all developers’ working copies to a shared mainline several times a day” — wikipedia

In other words, the developers will try to merge their feature code into the master branch as frequent as possible. Following this practice enables the developers and product managers to release the product more frequently.

There are some extended versions of the CI pipelines in which these changes are also being tested automatically which makes the code deployable at any time, it’s called ‘Continuous Delivery’. A further extension of this pipeline is called ‘Continuous Deployment’ pipeline, where these tested code changes are pushed automatically into the production servers. ( We will be implementing the continuous deployment pipeline in our case)

Travis CI >

Travis CI is a hosted continuous integration service used to build and test software projects hosted at GitHub. Open source projects can be tested without any charges !!

Travis CI can be configured by adding a .travis.yml file to the repository. when Travis CI has been activated for a given repository, GitHub will notify whenever new commits are pushed to the repository or any pull request is submitted then according to the rules defined in the .travis.yml file, Travis CI will perform the steps which can be anything — from running tests, building the application or deployment scripts. Travis CI offers a wide range of options to build the software and of course, our beloved ❤️javascript is one of them.

NOTE: Github has student developer pack available with a bunch of premium features from different platforms (Travis CI is one of them) for free to students who wish to learn new things — get your student pack now !!

DevOps >

DevOps is a set of software development practices that combines software development (Dev) and information technology operations(Ops) to shorten the systems development life cycle while delivering features, fixes, and updates frequently. The concept of DevOps is founded on building a culture of collaboration between teams.

“DevOps is more than practice — it’s about culture”

Continuous Integration, Continuous Delivery, Continuous Deployment are some of the few key practices of the DevOps. Apart from these DevOps engineers heavily uses the power of the cloud infrastructure to make the deployment process seamless.

Enough talking !!! Let’s do some action

As you have already deployed on the GitHub pages using the gh-pages node module, there will be a branch called gh-pageson the repository which holds the files which are deployed onto the Github pages servers. After the integration of the Travis CI, we would ab able to implement the system where any changes made by the user on the master branch will automatically trigger a build. If the build is successful, then the build scripts will be triggered which will update the gh-pages branch. User will be notified about the status of the build via email notifications from the Travis CI

Create an account on Travis-CI >

• Go to Travis-ci.com and Sign up with GitHub.
• Accept the terms & conditions of Travis CI. You’ll be redirected to GitHub.
• Click the Activate button, and select the repositories you want to use with Travis CI.
• Add authorization token ( This will be done automatically when you sign-in with GitHub)

Add travis.yml file into the repository >

This file contains the instructions which tell Travis-CI — what?..how?..when?

NOTE: When you trigger a job in the Travis-CI, it will boot up a virtual machine with the appropriate deployment environment configured in the _.travis.yml_

Let’s break down the code —

language: node_js
node_js:
  - "stable"
cache:
  directories:
  - node_modules
script:
  - npm run build
deploy:
  provider: pages
  skip_cleanup: true
  github_token: $github_token
  local_dir: build
  on:
    branch: master

on : Travis-CI will automatically trigger a job whenever there are some changes made on the branch specified in this field.

deploy : In this filed we have declared that we will use the deployment provider for the GitHub pages provided by the Travis-CI which is nothing but the configuration instructions for setting up the environment for deployment.

script : This filed contains the build scripts which will be executed while running the job. For this case that is the build script, you can also add test scrips ( code-coverage, fusion test, etc.) before the build.

cache : Travis-CI provides an option to cache the library files and modules which will be the constant for all the builds. Cached files can be used again by the later build jobs which decreases the end-to-end running time of the job.

All set >

Ohkay everything is in the place, now onwards if you commit anything on the master branch it will trigger a Travis-CI build job which will look something like in the below screenshots. You can also trigger a build manually from the Travis-CI dashboard itself.

Travis-CI job(Running)

Travis-CI job (successful)

But …. (there is always a but !! huh!!)

I am pretty sure your build dashboard will not look like the above one same as life has not been smooth we were told it would be ?. There can be infinite reasons due to which your Travis-CI dashboard is full of failed builds ( I know..I have been through this)

This is the time when your most valuable “googling” skills will come handy. I will explain what all are the errors I have faced while I was trying to create a pipeline.

• Security errors
• Token errors
• Just random errors (You have to get dirty & find the solution!!)

Token errors >

If your builds are failing due permissions error then there are high chances that there is some problem with tokens. You need to go to the token URL https://github.com/settings/tokens and see when it was used lately, if it shows never then you have found your culprit.

Follow the below steps,

• Delete and create a new token
• Add it to the Travis environment variables ( Go to job settings )
• Re-try the build

Security errors >

There are plenty of security practices we ignore while coding & building web applications. When we run in local these security errors are not given much emphasis and often discarded as warning messages, but when we are trying to deploy the service using the Travis-CI these warnings will cause the build failure.

I will mention the errors I encountered while working on my project(I would encourage you to mention the errors you have encountered) The great thing is that most of them have their own dedicated web-pages which explain the underlying problem and offers the solutions/workarounds ( Workarounds — we all love it even knowing that we shouldn’t !! )

• Using target=_blank in HTML tag : This is more serious security flaw than it looks. You can learn more about it here.
• Redundancy in HTML code: There were many redundant tags/class names which were making the code look like junk.

Best way to prevent these errors is to install the es-lint plug-in in whichever text-editor you are using.

Built some project? — Share it

I am trying to build a community of developers where people can share their ideas, knowledge, work with others and find other people with similar ideology to build things together. So, if you built some project and want to share it, post it on the channel.

• Gitter channel: https://gitter.im/weekend-devs/community
• Github Organization: https://github.com/weekend-developers

Wrapping up

I would like to take a moment to acknowledge the work of the people who gave me the inspiration and knowledge to complete this article.

• Travis CI community: for providing awesome tools for free.
• My dearest friends: who helped me in correcting my mistakes.
• YOU: for sticking around, I hope you had a productive time. Keep exploring and building amazing things!

_Photo by [Unsplash](https://unsplash.com/@clemensvanlay?utm_source=medium&utm_medium=referral" data-href="https://unsplash.com/@clemensvanlay?utm_source=medium&utm_medium=referral" class="markup--anchor markup--figure-anchor" rel="photo-creator noopener noopener noopener" target="_blank">Clemens van Lay on <a href="https://unsplash.com?utm_source=medium&utm_medium=referral" data-href="https://unsplash.com?utm_source=medium&utm_medium=referral" class="markup--anchor markup--figure-anchor" rel="photo-source noopener noopener noopener" target="blank)

Continuous Deployment is a beautiful thing. Committing your project and seeing it being built and deployed without having to do anything is mesmerizing.

And in this article, I want to show you how to get it done in your home project with ease.

To clear it up, here is a flowchart showing the differences between Continuous Delivery and Continuous Deployment.

Continuous Delivery vs. Continuous Deployment

Since most of the time no one but you depends on your home project, we’re going for a workflow with Continuous Deployment since you want to see your changes immediately deployed. If that’s not the case, you can change the workflow later.

You will learn about the following:

• How to make a Dockerfile
• How to push your project to GitHub
• Automatically building the docker image on Docker Hub
• Automatically downloading and running the image with Watchtower

Prerequisites:

• Some knowledge about Docker and the Dockerfile, though I will explain some of it along the way
• Have git installed
• A Docker Hub account
• A (Linux) server (either physical or virtual) running Docker

For reference, this is the example GitHub repository, and this is the example docker hub repository that I’ll be using.

Thus this tutorial will only be useful if you intend to run your software with Docker (which I recommend as Docker is fantastic).

Why use Docker?

Docker enables you to have the same environment for development and production which eliminates Heisenbugs and the “it works on my machine” problem. Also, containers are isolated which gives us security benefits.
There’s more to it, but these two benefits make me always deliver my software in Docker containers.

Setting up your Dockerfile

First, we will make a Dockerfile for the project. This special file is always called “Dockerfile” without an extension and sits at the top directory of your project.

A Dockerfile starts with the FROM statement which tells Docker which base image you want to start with. You can imagine this as using a canvas with the background already drawn and only the central part (your program) missing.
Most of the time the image you want to pull is the base image of your programming language, which you can find at the before mentioned Docker Hub.

Next, we copy our project files into the docker container with the COPY.. command. What does this do?

It takes the files from the first directory (the dot refers to the current directory of the file, which includes all your project files) and puts it in the current directory of your Docker container (remember your docker container is its own OS). Your files are now at the base directory there, which you may want to change.

Next, we need to install dependencies, which I will use python pip for, but any equivalent package management system depending on your language of choice will do. The critical thing to learn here is how to execute commands in the container with RUN.

From python:3.7COPY . .RUN pip install -r requirements.txt

Easy, isn’t it? Now we have to start our program in the container.

CMD ["python", "./my_script.py"]

The CMD statement is unique. Every Dockerfile has to have it as its last line because it starts the primary process in the container.

You have finished your Dockerfile! You can now manually build your image and container, but we’re going to skip that for now.

Now, we’ll create our repository on GitHub, but remember to leave “Initialize this repository with a README” unticked.

Then you’d need to copy the remote URL.

Open a cmd/shell in the root directory of your project.

You need to initialize your git repository, add your files, configure the remote, commit the files and push your project to GitHub.

git initgit add *git remote add origin https://github.com/<user>/<repository>.gitgit commit -a -m "Make Dockerfile ready for CD"git push -u origin master

Now, your GitHub Repository should look like this:

Congratulations, you’re about halfway done!

The next step is to connect GitHub to Docker Hub. For this, you go to the account settings.

Scroll down and connect your git host.

Create your repository on docker hub now.

Give the repo a name and click the GitHub icon (or Bitbucket, if that’s your thing). Now choose your organization (usually your username) and your project’s name. If you want to use your master image for the build and always push to latest, you can now click “Create & Build” and watch your image being built for you. Otherwise, you have to edit the build settings.

Last steps! Now you need Watchtower on your target machine.
Watchtower is a program that pulls your running docker images and checks for updates. If there are any updates, it gracefully shuts down the original container and creates a container from the new image with the same settings.

The best thing is that we can also install Watchtower with Docker!

Enter the following into your terminal:

docker run -d --name watchtower -v /var/run/docker.sock:/var/run/docker.sock v2tec/watchtower

Then you need to run the Docker container for your project!

docker run -d --name <my-project> <username>/<my-project>

The “-d” option makes your program run in the background, so the program doesn’t shut down if you close the terminal.

So to summarize, if you push a commit to your GitHub repository, Docker hub will automatically build a Docker image for you. This image then gets pulled by WatchTower and is run with all original options.

If you need help at any point don’t be afraid to ask, I’m happy to help.
If it is a technical problem, an issue on the GitHub project would be awesome!

But what about tests?

Good question!
You can use Travis CI to run your tests at the same time.
You can read about this here, but the gist of it is, that you add another file to your repository which has instructions for an external server to execute unit tests or any other instructions.

But what if I only want my docker image to build if the tests pass?

This breaks our workflow a bit.
We now can’t rely on docker hub to build our images anymore. Instead, it’s also going to be Travis CI that produces the image and then pushes it to your Docker Hub repository. Read about this here.

I asked myself this question when I started to learn Kubernetes: how can we simplify the deployment of containerized apps? At first look, deploying containerized apps seems pretty simple. All you need is a bunch of YAML files, and by using [kubectl](https://kubernetes.io/docs/reference/generated/kubectl/kubectl/) (the Kubernetes command line utility), you’ll have your service up and running in your Kubernetes cluster.

But, Although deploying one app is an easy task, how do you deploy hundreds of apps? At Soluto, we have more than 100 live microservices and this number keeps growing. So as we started thinking about shifting workload to our Kubernetes cluster, we faced a few challenges:

First, Kubernetes deployment is actually more complex. There are many moving parts that you need to set up correctly: pod autoscaler, pod resources, ingress, and so on. Those parts require some experience with how Kubernetes works, and not setting it up correctly might cause issues in production. Ideally we’d have a way to simplify this, so developers can focus on writing their code and worry less about deployment.

Second, security is also a challenge. All services in production should have certain things, like Transport Layer Security (TLS). These are not necessarily complex things, but need to be taken care of nonetheless. We would like to pre-configure them so that any new deployment will be secured by default.

Finding a solution

To solve these challenges and speed up and ease the adoption process, we looked for a way to create a template for Kubernetes. Something that any developer would be able to use, and would require just a few parameters (for example, the Docker image of the service) to get the service up and running in production.

On the other hand, we needed to be careful not to hide too much — developers must be able to understand what’s going on so they can handle production issues. We had to find the right level of abstraction that makes it easier to deploy to Kubernetes, without hiding too much detail.

With that in mind, we started to look for a solution. After trying a few things, we found Helm. Helm is a package manager for Kubernetes.You can use it to install any app on your cluster, and Helm will take care of getting all the required configuration files and installing them on your cluster. Helm also support updating deployments, rollbacks, and many other cool features. Each Helm package is called a “Chart”, and the charts are stored in a repository. With helm, installing Mongo, for instance, is as easy as helm install stable/mongodb.

Sound like an excellent solution! We can define a chart for each type of service — like one for all our web APIs, which will handle things like load balancer and TLS — and the developer simply needs to specify the required parameters using Helm’s configuration files.

Me and my teammate, when we found Helm

Helm: Let’s see how it’s done

In order to use Helm, we first need to install it. Helm has two components: Helm client running on your computer, and Tiller, a server-side component running on your cluster. Then, we need to create the chart by using this simple Helm CLI command: helm create web-api

After running this command, you’ll notice the creation of a new folder named “web-api”. Within this folder, you’ll find all the familiar Kubernetes configuration files: deployment, service, ingress, and so on. Now it’s time to customize a bit: we can add a horizontal pod autoscaler, define the default resources the pod requires, and of course, enable TLS by default.

Everything is highly customizable based on the Go templating mechanism. So anything we add can be overridden later by the developer, in case the default configuration doesn’t work as needed.

So now we have a chart — but how can we consume it? The chart has to exist in a Helm repository, which is basically a server with a few Zip archives (which are all the charts in the repository) and one index file that is consumed by the CLI. You can manually set up your repo using any storage service like Azure Blob or AWS S3, but the simplest option is Chart Museum.

Chart museum is a Helm repository with a CRUD API to manage your charts. It supports basic authentication so you can restrict who can push new charts to your Helm repository. Helm doesn’t supply any museum-as-a-service solution, so you’re gonna have to roll your own. But it’s dead simple — simply use its docker image.

Now we can build a CI/CD pipeline for our web-api chart, to ease the process of modifying it:

• Run some kind of tests, to make sure that the new version is not broken. I’ll discuss how in the next paragraph.
• Pack the new chart, using Helm CLI.
• Push the new package to our Chart Museum instance, using Chart Museum’s API.

Testing it out

Now our chart is ready to be used by developers! But wait… how can we know that the chart actually works? And how can we make sure it will continue to work? This is why we need to write tests for our chart (like we write for anything else).

There are basically two things we want to test.

1. We want to test our template — for example, if an ingress is supposed to exist with a TLS and specific rules (defined by the developer), we should test the generated template and make sure the ingress was created correctly.
2. We want to test that the files are valid Kubernetes configurations and that they work as expected.

Testing the first thing is relatively simple — check out this sample repo to see just how simple it is.

This allows us to test the generated Kubernetes files, by using [kubetest](https://github.com/garethr/kubetest). This is great, but can be complex and messy, especially when having a lot of branching in your template files.

A better solution is required — one that will allow us to do unit testing for the templates, without generating Kubernetes files. This wasn’t a problem until recently when we started to have a lot of branching in our templates, and we’re now looking for options.

The second thing — testing that Kubernetes files are valid — is a bit more tricky. For now, at Soluto we’re using Helm’s version mechanism: each chart has a version, and all of our services will use the latest stable version. When a new chart version is pushed, we can test this version on a specific service. If it works correctly, update the rest of the services. Another option is to test that using minikube, but it was too complex for our needs.

Finally: Deploying!

So now we have a CI/CD pipeline for our Helm charts, and we have prepared a Helm chart that the developers can use. Now, when a new developer wants to deploy a new service to production, all they have to do is:

1. Add our repository to their local Helm using helm repo add chartmuseum http://<chart-museum-url>
2. Create a new Helm config file and specify the required parameters (for example, docker image of the service)
   3. Run helm upgrade —install <service-name> chartmusuem/web-api -f<path_to_config_file> and that’s it. The service is alive.

And to make it even easier to understand, I’ve created a sample repository. The repository contains all the things that I’ve discussed in this blog post: a generic chart for a web app that can be deployed with this chart and chart museum. Check it out to better understand what’s going on — and there is even a walkthrough to help you get started.

Thank you for reading along. If you have any questions, or you need help getting started with Helm, feel free to reach out either via the comments here or via Twitter.

Happy Helming!

Originally posted on Soluto’s Blog