In this tutorial we'll learn how to automatically publish machine learning models in a Gitlab package registry and make them available for your teammates to use. You can also use this technique to share a packaged version of your code as a binary.

If you are a beginner Gitlab user and are unfamiliar with CI/CD techniques, this tutorial is for you! A basic understanding of how machine-learning and deep learning is a plus, but it isn't a requirement to understand the CI/CD publishing part.

Here's what we'll cover:

• Gitlab Code Setup
• Deep Convolutional Neural Network Code
• Image Recognition Code
• Branching Methodology
• CI/CD Uploading
• Conclusion

First, Some Background

At some point during your machine learning engineer career you might need to share a model you've trained with other developers. There are multiple ways of doing this.

Give access to the repository

If you don't mind showing your whole code, this is a very viable option.

If you use a good branching methodology your colleagues will only need to look at the main branch in order to figure out what's the most up to date model they can use. Then they can check the README.md to learn how to use it.

However, giving full access to the repository might not be a viable option for you.

Share the latest model manually

Another way would be to extract the relevant code that you want to make public and send it to them manually.

This can become a bit of a mess if you are working with more than one person because the model you send might not be up to date. It also puts the burden on you to make sure that people are always using the latest version of your model.

Share the latest model automatically

A simpler solution, even in the case where the repository code is available, is to put the packaging burden on a CI/CD pipeline.

This is the topic of this tutorial, and our setup will look like this:

• The code repository, CI/CD tool set, and package registry will be on Gitlab
• The code we'll be packaging will be a simple trained PyTorch neural network on the MNIST dataset for digit recognition.
• All the instructions and the requirements will be available in the package.

🚨 Disclaimer 🚨: This isn't how you should deploy a PyTorch production-ready model! To learn how to do this, check out this tutorial on TorchScript.

Let's get started.

Gitlab Code Setup

For this tutorial we will bundle four files:

• model.pth: which is a pickled version of the latest version of the trained model.
• run_mnist.py: simple Python script to run the model to detect a digit from a png image.
• requirements.txt: text file containing all the dependencies required to run the model.
• INSTRUCTION.md: step by step instructions to use the package.

The package can then be used freely by anyone who has access to the package registry and will be automatically updated.

The package will then look like this on Gitlab Package Registry!

Let's jump into the neural network code, which is a modified version of this comprehensive article on digit recognition. The modified code can be found over at my public Gitlab repository.

Deep Convolutional Neural Network Code

In the section below, you will see quite a lot of terminology about deep neural networks. This isn't a tutorial on neural networks, so if you feel a bit overwhelmed by the specifics you can jump directly to the Branching Methodology section.

Just keep in mind that we've trained some sort of image recognition program that, given a .png file representing a digit, will be able to tell you what number it contains.

However, for those that want to get a better understanding about how Deep Neural Networks work under the hood, you can take a look at my tutorial where I build one from scratch or checkout the code directly in my Github.

Neural Network Definition

The network definition code is very straightforward since the network we will use is simple. It has the following characteristics:

• 2 convolutional layers.
• Dropout is applied on the second convolutional layer.
• Relu activation functions applied on all neurons.
• 2 fully connected layers at the end for inference.

import torch
import torchvision

import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim


# Define the network
# It's a 2 convolutional layer with dropout at the 2nd and finally 2 fully connected layer
# All layers use relu
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 10, kernel_size=5)
        self.conv2 = nn.Conv2d(10, 20, kernel_size=5)
        self.conv2_drop = nn.Dropout2d()
        self.fc1 = nn.Linear(320, 50)
        self.fc2 = nn.Linear(50, 10)

    def forward(self, x):
        x = F.relu(F.max_pool2d(self.conv1(x), 2))
        x = F.relu(F.max_pool2d(self.conv2_drop(self.conv2(x)), 2))
        x = x.view(-1, 320)
        x = F.relu(self.fc1(x))
        x = F.dropout(x, training=self.training)
        x = self.fc2(x)
        return F.log_softmax(x, dim=1)

Training Function

We then created a utility training function in order to iteratively improve our defined network using gradient descent. If you want to learn more about how gradient descent works check out my short tutorial on it.

This training regimen will do the following:

• Iterate on batches of training data representing 28 by 28 digits.
• Use the negative log likelihood cost function to calculate the loss.
• Calculate gradients.
• Optimize the weights of the network using gradient descent.
• Save the model at fixed intervals.

def train(network, optimizer, train_loader, epoch_id, log_interval=10):
  """Run the training regiment on the training set using train_loader

    Args:
        network: The instantiated network.
        optimizer: The optimizer used to change the weights.
        train_loader: the loader for the training set already setup
        epoch_id: the current id of the epoch used for cosmetic reason.
        log_interval: interval at which we print an output

    Returns:
        nothing, will save directly at root level the model and the optimizer state

  """

  # Set the network in training mode
  network.train()

  # Iterate over the full training set
  for batch_idx, (data, target) in enumerate(train_loader):

    # Calculate the gradients for this batch of data
    optimizer.zero_grad()
    output = network(data)
    loss = F.nll_loss(output, target)
    loss.backward()

    # Optimize the network
    optimizer.step()

    # Log and save every selected interval
    if batch_idx % log_interval == 0:

      print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
        epoch_id, batch_idx * len(data), len(train_loader.dataset),
        100. * batch_idx / len(train_loader), loss.item()))

      # This will save the state as a pickled object
      torch.save(network.state_dict(), './model.pth')
      torch.save(optimizer.state_dict(), './optimizer.pth')

The data for training can be found over here on the Yan LeCun website. Here we are using the datasets formatted as 28 by 28 PyTorch tensors for training.

Testing Function

The next function we create is a testing function to validate if our network has learned something without reusing the same training data. This function is simple in the sense that it will just tally the correct and incorrect predictions.

def test(network, test_loader):
  """Run the testing regiment on the test set using test_loader

    Args:
        network: The instantiated and trained network.
        test_loader: the loader for the testing set already setup

    Returns:
        nothing, will only print result

  """

  # Variable instantiation
  test_loss = 0
  correct = 0

  # Move the network to evaluate mode instead of training
  network.eval()

  # setup torch so to not track any  gradient
  with torch.no_grad():

    # Iterate on all the test data and accumulate the loss
    for data, target in test_loader:
      output = network(data)
      test_loss += F.nll_loss(output, target, size_average=False).item()
      pred = output.data.max(1, keepdim=True)[1]
      correct += pred.eq(target.data.view_as(pred)).sum()

  # Average loss calculation and printing   
  test_loss /= len(test_loader.dataset)
  print('\nTest set: Avg. loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\n'.format(
    test_loss, correct, len(test_loader.dataset),
    100. * correct / len(test_loader.dataset)))

This function will be useful to check how well our network has learned after each training iteration.

Training Regimen

Finally, we can tie all of the above together with the main body of the training script! A few things are happening, but the most important points are the following:

• We set our hyper parameters statically. A better way to define them would be to use a validation set to figure them out based on the data.
• We create our data loader which will ingest data and spit out tensors in the right shape for the network. These loader will transform the data by normalizing them with the global mean and standard deviation for the MNIST datasets.
• We use stochastic gradient descent with momentum as the optimization method, which is one of the many flavors of gradient descent we can use.
• We loop through the full training dataset's "epoch", the amount of time to train the network while testing on the held-out test datasets.

# Experimental Parameters that we can tweak
n_epochs = 3
batch_size_train = 64
batch_size_test = 1000
learning_rate = 0.01
momentum = 0.5

# Variable from the dataset that should stay as is
global_mean_mnist = 0.1307
global_std_mnist = 0.3081


# Random Seed for Reproducible Experimentation
random_seed = 42
torch.backends.cudnn.enabled = False
torch.manual_seed(random_seed)


# Data Loader to gather the data and then normalize them
train_loader = torch.utils.data.DataLoader(
  torchvision.datasets.MNIST('./data/', train=True, download=True,
                             transform=torchvision.transforms.Compose([
                               torchvision.transforms.ToTensor(),
                               torchvision.transforms.Normalize(
                                 (global_mean_mnist,), (global_std_mnist,))
                             ])),
  batch_size=batch_size_train, shuffle=True)

test_loader = torch.utils.data.DataLoader(
  torchvision.datasets.MNIST('./data/', train=False, download=True,
                             transform=torchvision.transforms.Compose([
                               torchvision.transforms.ToTensor(),
                               torchvision.transforms.Normalize(
                                 (global_mean_mnist,), (global_std_mnist,))
                             ])),
  batch_size=batch_size_test, shuffle=True)

# Initialize network and optimizer
network = Net()
optimizer = optim.SGD(network.parameters(), lr=learning_rate,
                      momentum=momentum)

# Test first to show that the model didn't learn a thing
test(network, test_loader)

# Train on the whole dataset multiple time and test
for epoch_id in range(1, n_epochs + 1):
  train(network, optimizer, train_loader, epoch_id)
  test(network, test_loader)

Note that it's very important to test your network on a held-out set to avoid over-fitting on the training data.

All of the above scripts can be found in the file train_mnist.py in the repository.

At this point, we can train a model and have it saved at regular intervals in a pickle format.

We can now use that saved trained mode to evaluate a digit in a .png file.

Image Recognition Code

Let's say we have as an input the following image:

a small 0 digit

or this one:

a bigger 7 digit

How can we make our network, which works on a 28 by 28 PyTorch tensor, evaluate the numbers?

It's fairly straightforward if we follow roughly the same process that the training datasets went through, which is:

• Have grayscale images (no color or alpha channels)
• Resize the images to be 28 by 28 pixels
• Normalize the images using the mean and standard deviation of the MNIST datasets.

if __name__ == "__main__":

    # Variable iniatilization
    global_mean_mnist = 0.1307
    global_std_mnist = 0.3081

    # Loading of the network with right weight
    result_path = './model.pth'
    model = Net()
    model.load_state_dict(torch.load(result_path))
    model.eval()

    # Setup the transform from image to normalized tensors
    transform = transforms.Compose([
                        transforms.Resize((28,28)),
                        transforms.ToTensor(),
                        transforms.Normalize(
                            (global_mean_mnist,), (global_std_mnist,))
                        ])

    # Parse the input from the user which should be a filename with the --image flag
    parser = OptionParser()
    parser.add_option("--image", dest = "input_image_path",
                      help = "Input Image Path")
    (options, args) = parser.parse_args()

    # Get the path to the image to decode
    input_image_path = str(options.input_image_path)

    # Open the image(s) and do the inference
    images=glob.glob(input_image_path)
    for image in images:

        # Convert the image to grayscale
        img = Image.open(image).convert('L')

        # Transform the image to a normalized tensor
        img_tensor = transform(img).unsqueeze(0)

        # Make and print the prediction
        output = model(img_tensor).data.max(1, keepdim=True)[1][0][0]
        print(f"Image is a {int(output)}")

As you can see, we use a parser to accept an image path on the command line before applying our transformations. Once they are applied we can feed that to our loaded model and collect the output prediction.

⚠️ Don't forget to include the definition of the network in the script (by importing or copy pasting), otherwise the pickled model will not be able to load properly.

We can now run our code like this:

python run_mnist.py --image NAME_OF_IMAGE.png

This will simply print the model's inference about what that particular image contains.

Now that we have the basic training and evaluation code set up, let's discuss a bit more about how to use git branching to our advantage to publish this model to the package registry.

Branching Methodology

If you are working alone on a project, it is very tempting to simply commit to master/main and be done with it. However, this way of working is very difficult to maintain and it makes incorporating proper CI/CD tools a pain.

A main / develop branch strategy as shown below is more maintainable:

Image from: https://nvie.com/posts/a-successful-git-branching-model/

By always keeping the main branch clean, we can easily flag our CI/CD pipeline to be triggered as soon as we push to the main. We will be also free to commit as much as we need in the develop branch while we improve our models.

When we are ready for a new deploy we will only need to merge with the main branch (or better yet do a merge-request / pull-request and then merge).

This merge to main should trigger Gitlab to upload the new version of our model to the package registry.

Let's take a look at the simple way to automate publishing to the package registry using the .gitlab-ci.yml file.

CI/CD Pipeline

The .gitlab-ci.yml file is a special file in your repository used by Gitlab to define what the Gitlab server should do when you push to a repository.

To learn more about how CI/CD works in Gitlab, head over to this Gitlab CI/CD crash course.

In this tutorial our .gitlab-ci.yml file looks like this:

image: pytorch/pytorch

variables:
  VERSION: "0.0.4" # To Change if needs be

stages:
  - upload

upload:
  stage: upload
  only:
    - master
  script:
    - apt-get update
    - apt-get install -y curl wget
    - 'curl --header "JOB-TOKEN: $CI_JOB_TOKEN" --upload-file ./model.pth "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/generic/example-ml-packaging-pipeline/${VERSION}/model.pth"'
    - 'curl --header "JOB-TOKEN: $CI_JOB_TOKEN" --upload-file ./run_mnist.py "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/generic/example-ml-packaging-pipeline/${VERSION}/run_mnist.py"'
    - 'curl --header "JOB-TOKEN: $CI_JOB_TOKEN" --upload-file ./requirements.txt "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/generic/example-ml-packaging-pipeline/${VERSION}/requirements.txt"'
    - 'curl --header "JOB-TOKEN: $CI_JOB_TOKEN" --upload-file ./INSTRUCTION.md "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/generic/example-ml-packaging-pipeline/${VERSION}/INSTRUCTION.md"'

The anatomy of this .yml file is very bare bones. We have only one stage in our pipeline which is the upload stage.

In the upload stage, we will run the script section only when the master branch gets updated. The script that we ran is simply using curl to transfer the data from this repository (4 files) into the package registry.

Let's take a look at the anatomy of the curl command we are using:

 - 'curl --header "JOB-TOKEN: $CI_JOB_TOKEN" --upload-file ./NAME_OF_FILE "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/generic/example-ml-packaging-pipeline/${VERSION}/NAME_OF_FILE"'

• --header is used to tell curl that you will be including an extra header to the request.
• JOB-TOKEN is our header and $CI_JOB_TOKEN is its value. It's a variable that lives within Gitlab servers when a job is created
• --upload-file is a flag to tell that we will transfer a local file to the remote URL.
• ./NAME_OF_FILE is the name of the local file we want to transfer.
• ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/generic/example-ml-packaging-pipeline/${VERSION}/NAME_OF_FILE is the location of the remote URL that we want to transfer a file.

Here $CI_API_V4_URL is the URL of the Gitlab API we are using, $CI_PROJECT_ID is defined within Gitlab CI as the id for our project, and finally VERSION is the version number we defined at the top of the .yml file.

That's it! When you update the main branch to the remote repository on Gitlab it will fire up a pipeline that will run your packaging job.

The job will then be available and you will be able to check the trace on Gitlab!

You and your teammates will be able to see the document in the package registry section and get the right versioned files in the package:

This is our v.0.0.5 of the example package!

To get a more complete idea of what is possible with the Packages API, head over to the official documentation.

Conclusion

In this tutorial you've learn how to bundle, upload, and automatize a machine learning model packaging using Gitlab CI/CD.

Congratulation! 🎉🎉🎉

There is still a lot more you can do with Gitlab CI/CD, for instance:

• Add a testing stage before the bundling in order to make sure that there is no regression in the code.
• Add a testing stage after the bundling to make sure that the performance of your model is satisfactory in terms of inference latency.
• Use a more optimized version of the model with TorchScript.
• Add automatic social notification of new release after the upload step.

To learn more about Gitlab CI/CD the official docs is a great place to start out, and the get started section is very beginner friendly.

If you want to read more of this type of content, check out my mechanical/software engineering articles. If you want to discuss any of this feel free to send me a DM on LinkedIn or Twitter :)

As part of some recent explorations into automatically re-creating my laptop with Bash scripts (post to come!), I wanted to find a way to easily clone my GitHub-hosted repositories to a new machine. After a bit of digging around, I wrote a one-liner that did just that.

Then, in the spirit of not putting all our eggs in the same basket, I wrote another one-liner to automatically create and push to GitLab-hosted backups as well. Here they are.

A Bash one-liner to clone all your GitHub repositories

Caveat: you’ll need a list of the GitHub repositories you want to clone. The good thing about that is it gives you full agency to choose just the repositories you want on your machine, instead of going in whole-hog.

You can easily clone GitHub repositories without entering your password each time by using HTTPS with your 15-minute cached credentials or, my preferred method, by connecting to GitHub with SSH. For brevity I’ll assume we’re going with the latter, and our SSH keys are set up.

Given a list of GitHub URLs in the file gh-repos.txt, like this:

git@github.com:username/first-repository.git
git@github.com:username/second-repository.git
git@github.com:username/third-repository.git

We run:

xargs -n1 git clone < gh-repos.txt

This clones all the repositories on the list into the current folder. This same one-liner works for GitLab as well, if you substitute the appropriate URLs.

What’s going on here?

There are two halves to this one-liner: the input, counterintuitively on the right side, and the part that makes stuff happen, on the left. We could make the order of these parts more intuitive (maybe?) by writing the same command like this:

<gh-repos.txt xargs -n1 git clone

To run a command for each line of our input, gh-repos.txt, we use xargs -n1. The tool xargs reads items from input and executes any commands it finds (it will echo if it doesn’t find any). By default, it assumes that items are separated by spaces; new lines also works and makes our list easier to read. The flag -n1tells xargs to use 1 argument, or in our case, one line, per command. We build our command with git clone, which xargs then executes for each line. Ta-da.

A Bash one-liner to create and push many repositories on GitLab

GitLab, unlike GitHub, lets us do this nifty thing where we don’t have to use the website to make a new repository first. We can create a new GitLab repository from our terminal. The newly created repository defaults to being set as Private, so if we want to make it Public on GitLab, we’ll have to do that manually later.

The GitLab docs tell us to push to create a new project using git push --set-upstream, but I don’t find this to be very convenient for using GitLab as a backup. As I work with my repositories in the future, I’d like to run one command that pushes to both GitHub and GitLab without additional effort on my part.

To make this Bash one-liner work, we’ll also need a list of repository URLs for GitLab (ones that don’t exist yet). We can easily do this by copying our GitHub repository list, opening it up with Vim, and doing a search-and-replace:

cp gh-repos.txt gl-repos.txt
vim gl-repos.txt
:%s/\<github\>/gitlab/g
:wq

This produces gl-repos.txt, which looks like:

git@gitlab.com:username/first-repository.git
git@gitlab.com:username/second-repository.git
git@gitlab.com:username/third-repository.git

We can create these repositories on GitLab, add the URLs as remotes, and push our code to the new repositories by running:

awk -F'\/|(\.git)' '{system("cd ~/FULL/PATH/" $2 " && git remote set-url origin --add " $0 " && git push")}' gl-repos.txt

Hang tight and I’ll explain it; for now, take note that ~/FULL/PATH/ should be the full path to the directory containing our GitHub repositories.

We do have to make note of a couple assumptions:

1. The name of the directory on your local machine that contains the repository is the same as the name of the repository in the URL (this will be the case if it was cloned with the one-liner above);
2. Each repository is currently checked out to the branch you want pushed, ie. master.

The one-liner could be expanded to handle these assumptions, but it is the humble opinion of the author that at that point, we really ought to be writing a Bash script.

What’s going on here?

Our Bash one-liner uses each line (or URL) in the gl-repos.txt file as input. With awk, it splits off the name of the directory containing the repository on our local machine, and uses these pieces of information to build our larger command. If we were to print the output of awk, we’d see:

cd ~/FULL/PATH/first-repository && git remote set-url origin --add git@gitlab.com:username/first-repository.git && git push
cd ~/FULL/PATH/second-repository && git remote set-url origin --add git@gitlab.com:username/second-repository.git && git push
cd ~/FULL/PATH/third-repository && git remote set-url origin --add git@gitlab.com:username/third-repository.git && git push

Let’s look at how we build this command.

Splitting strings with awk

The tool awk can split input based on field separators. The default separator is a whitespace character, but we can change this by passing the -F flag. Besides single characters, we can also use a regular expression field separator. Since our repository URLs have a set format, we can grab the repository names by asking for the substring between the slash character / and the end of the URL, .git.

One way to accomplish this is with our regex \/|(\.git):

• \/ is an escaped / character;
• | means “or”, telling awk to match either expression;
• (\.git) is the capture group at the end of our URL that matches “.git”, with an escaped . character. This is a bit of a cheat, as “.git” isn’t strictly splitting anything (there’s nothing on the other side) but it’s an easy way for us to take this bit off.

Once we’ve told awk where to split, we can grab the right substring with the field operator. We refer to our fields with a $ character, then by the field’s column number. In our example, we want the second field, $2. Here’s what all the substrings look like:

1: git@gitlab.com:username
2: first-repository

To use the whole string, or in our case, the whole URL, we use the field operator $0. To write the command, we just substitute the field operators for the repository name and URL. Running this with print as we’re building it can help to make sure we’ve got all the spaces right.

awk -F'\/|(\.git)' '{print "cd ~/FULL/PATH/" $2 " && git remote set-url origin --add " $0 " && git push"}' gl-repos.txt

Running the command

We build our command inside the parenthesis of system(). By using this as the output of awk, each command will run as soon as it is built and output. The system() function creates a child process that executes our command, then returns once the command is completed. In plain English, this lets us perform the Git commands on each repository, one-by-one, without breaking from our main process in which awk is doing things with our input file. Here’s our final command again, all put together.

awk -F'\/|(\.git)' '{system("cd ~/FULL/PATH/" $2 " && git remote set-url origin --add " $0 " && git push")}' gl-repos.txt

Using our backups

By adding the GitLab URLs as remotes, we’ve simplified the process of pushing to both externally hosted repositories. If we run git remote -v in one of our repository directories, we’ll see:

origin  git@github.com:username/first-repository.git (fetch)
origin  git@github.com:username/first-repository.git (push)
origin  git@gitlab.com:username/first-repository.git (push)

Now, simply running git push without arguments will push the current branch to both remote repositories.

We should also note that git pull will generally only try to pull from the remote repository you originally cloned from (the URL marked (fetch) in our example above). Pulling from multiple Git repositories at the same time is possible, but complicated, and beyond the scope of this post. Here’s an explanation of pushing and pulling to multiple remotes to help get you started, if you’re curious. The Git documentation on remotes may also be helpful.

To elaborate on the succinctness of Bash one-liners

Bash one-liners, when understood, can be fun and handy shortcuts. At the very least, being aware of tools like xargs and awk can help to automate and alleviate a lot of tediousness in our work. However, there are some downsides.

In terms of an easy-to-understand, maintainable, and approachable tool, Bash one-liners suck. They’re usually more complicated to write than a Bash script using if or while loops, and certainly more complicated to read. It’s likely that when we write them, we’ll miss a single quote or closing parenthesis somewhere; and as I hope this post demonstrates, they can take quite a bit of explaining, too. So why use them?

Imagine reading a recipe for baking a cake, step by step. You understand the methods and ingredients, and gather your supplies. Then, as you think about it, you begin to realize that if you just throw all the ingredients at the oven in precisely the right order, a cake will instantly materialize. You try it, and it works!

That would be pretty satisfying, wouldn’t it?

If you’re reading this article, you already know that the benefit of open source contribution abounds. You can skip the article and navigate to the end if you’re here for the cheat sheet.

The common problem faced by aspiring open source contributors is how to take the first step from fork to pull request. After reading this article, you should be well equipped with all you need to make your first open source pull request.

Apart from making the process easier for you, the git workflow defined in this piece also makes your contributions look professional. This is especially useful in case you want to add your open source contributions to your portfolio.

Prerequisites

_Photo by [Unsplash](https://unsplash.com/@randyfath?utm_source=ghost&utm_medium=referral&utm_campaign=api-credit">Randy Fath / <a href="https://unsplash.com/?utm_source=ghost&utm_medium=referral&utmcampaign=api-credit)

This article assumes you already know the steps to take to contribute to open source. If you don’t know, you may want to read this article written by Maryna. This piece also assumes that you’ve already setup Git on your PC. If you haven’t, you may want to check the setting up Git section of this article and do that first.

Step 1: Fork the project

This is as simple as clicking a button on GitHub. Navigate to the repository of the project you want to contribute to, then click the fork button at the top right corner as illustrated in the picture below.

After using the fork button, you’d now have the repository on your GitHub account.

Step 2: Clone the project to your local machine

This is the simplest part of Git. Navigate to your forked repository (the repository is now one of your GitHub repositories). Follow steps 1 and 2 as shown in the image below to copy the clone address. This address should look like this: https:[github.com/suretrust.com/freeCodeCamp.git](http://github.com/suretrust.com/freeCodeCamp.git)

Then, clone the project by typing git clone <the copied address> into your command terminal as shown below:

git clone [https://github.com/suretrust/freeCodeCamp.git](https://github.com/suretrust/freeCodeCamp.git)

Step 3: Create upstream

The upstream is necessary to keep track of the difference between the forked repository that is on your Git account and the original repository. This is most useful if you want to contribute to a popular repository.

Some repositories merge pull requests hourly or less, so be safe and assume that the forked repository you have will be behind the original repository.

Note that the upstream is in the freeCodeCamp repository and not your forked repository. Follow steps 1 and 2 as shown below to copy the upstream address:

To create a link to the original repository, copy and paste the following command into your terminal:

git remote add upstream <upstream address>

You can use git pull upstream master to confirm if there has been any change at the moment (from when you forked the repository to now).

Step 4: Create the branch you want to work on

_Photo by [Unsplash](https://unsplash.com/@zachreiner?utm_source=ghost&utm_medium=referral&utm_campaign=api-credit">Zach Reiner / <a href="https://unsplash.com/?utm_source=ghost&utm_medium=referral&utmcampaign=api-credit)

It is nice to create a new branch whenever you want to contribute. This illustrates that the branch is only for that contribution you are about to make. It could be as small as fixing a typo or as large as implementing a new feature. Either way, it’s good practice to create a branch.

Another important part of the branch creation is naming. It is pleasing to use a name that a stranger who knows nothing about the repository can easily understand. If you want to add a login feature, for example, you could create a branch called add-login-feature or login-feature.

To create a branch type the following command into your terminal:

git checkout -b <your branch name>

This command will create the branch and navigate into it. If your branch name is login-feature, then you can use the following command:

git checkout -b login-feature

Then add your contributions. After adding your contribution, move on to Step 5.

Step 5: Git add and commit your contributions

This is quite simple as well. Stage and commit your changes by typing the following into your terminal.

git add .

git commit -m 'Commit message'

Now, you have the changes staged and committed. What next?

Step 6: Pull from upstream to the branch

As I explained in step 4, this step is to merge any difference in the upstream into the branch so as to prevent conflicts.

git pull upstream <branch name>

This merges the upstream changes into your current branch.

Step 7: Push to the branch you’re working on

Now, you are almost there. Push your changes to the branch you are working on as shown below:

git push origin <branch-name>

Step 8: Open a pull request

This is the final step for any open source contribution, you are simply saying ‘I have made some changes, would you mind adding it to the project?’.

You open a pull request and if the repository owner or members like what they see, they’ll merge it. Otherwise, they could make changes then merge or request for changes.

To open a pull request, navigate to the forked repository as shown below. You’ll see your last push branch ‘login-feature’, then click on ‘compare and pull request’.

Explain clearly, the changes you made, then open a pull request as shown below:

And that’s it. :) You can now go ahead and contribute like a PRO!

Git cheat sheet for open source contributors

Peace out and happy contributing!

Adding an SSL certificate and custom Namecheap domain to a GitLab Pages site can be a bit more challenging than it seems.

Crucial pieces of the setup information live in sometimes dense documentation across different sites. It can be hard to tell if you’ve set things up correctly given that you have to wait hours to confirm your changes have propagated.

Even when you know something is wrong, you can’t always tell what. This makes debugging the problem frustrating and challenging to fix.

This guide aims to make the process a bit more straightforward and less frustrating. It assumes that you’ve:

• Already set up your project on GitLab Pages and are able to access it by entering <your-username>.gitlab.io/<your-project-name> in your browser
• Have purchased a custom domain name along with an SSL certificate through Namecheap

Step 1: Activate the SSL certificate

In Namecheap, go to the ‘Product List’ > ‘SSL Certificates’ page. You should see a list of SSL certificates that you have purchased, but have not yet activated. Click ‘Activate’ on the SSL certificate that you wish to activate for your site.

Step 2: Generate the SSL certificate request

You should have been brought to a page that looks like the following:

In order to generate a CSR, you’ll need to run the following command in your terminal: openssl req -new -newkey rsa:2048 -nodes -keyout <your-domain-name>.key -out <your-domain-name>.csr.

A private key will be generated as a result of that command. DO NOT LOSE THIS KEY. You will need it later on when you go to install your certificate on GitLab. Should you lose it, you will have to submit another CSR request.

You can read the nitty-gritty details here about generating a CSR if you wish, but the TL;DR is:

• It’s strongly encouraged that you fill out all the required fields. Your CSR could be rejected during activation of you do not. If you are filling this CSR out for a personal or hobby site, you can enter NA for the ‘Organization’ and ‘Organization Unit’ fields.
• If the certificate is being issued for a specific subdomain, you need to specify the subdomain in the ‘Common Name’ field. Example: subdomain.ssl-certificate-host.com
• If the certificate is meant to be a wildcard certificate, the domain should start with an asterisk. Example: *.ssl-certificate-host.com

For the purposes of this guide, the assumption will be made that you are getting the certificate for something like <example-domain&gt;.com .

Once you’ve run the command, you should have a .csr and .key file in your working directory. Open the .csr file, and copy the contents in it. It should have the header ----- BEGIN CERTIFICATE REQUEST -----.

Paste the contents of the file into the Enter CSR field. The page will automatically fill out the domain field on the form based on the information in the CSR.

Once you click ‘Next’, you should see the following page:

Check that the information is correct, and then click ‘Next’ again to go to the ‘Confirm you own the domain’ step.

Step 3: Confirm you own the domain

There are a few different options that are available to you in order to do this:

• Email
• HTTP-based
• DNS-based

I personally have had issues validating through email, so for the purposes of this guide, select ‘DNS-based’. This requires you to set up a CNAME value in your domain’s DNS settings, which we will cover later on in this guide.

For now, click ‘Next’ after selecting ‘DNS-based’, but if you change your mind about this form of validation later on, it’s possible to change it.

Step 4: Specify who will receive the SSL file

Confirm that the email in the field is correct. This is the email that will receive the certificate once it’s been activated.

Step 5: Review and Submit

Confirm the information shown is correct, and then click ‘Submit’.

Step 6: Set up the CNAME record for validating ownership of the domain

Once you submit the form, you will be redirected to a page showing the SSL certificate details with a helpful notification window that looks like the following:

Click on the link for the DNS-based DCV method. You’ll be brought to a page that shows information that you entered earlier, such as:

• The domain name
• The type of web server that will have the certificate installed (should be Apache, Nginx, cPanel, or other)
• DCV Methods In Use

Access the dropdown options for the ‘Edit Methods’ button to the right of ‘DCV Methods in Use’ in order to access and click the ‘Get Record’ option.

A popover will appear showing the CNAME record you need to set up in order to confirm ownership of the domain. Copy these values to an empty text file as you’ll need to go to the ‘Advanced DNS’ page for your domain. This is accessible through ‘Dashboard’ or ‘Domain List’ > ‘Manage’ (besides your domain in the list) > ‘Advanced DNS’.

Under the ‘Host Records’ section:

• Click ‘Add New Record’
• Select ‘CNAME Record’.
• Paste the values that you copied earlier from the ‘Get Record’ popover into the corresponding fields.

Before you save those values though, there’s a bit of a ‘gotcha’.

As Namecheap points out in their documentation, they “add the domain name automatically to the values submitted during record creation”. This means that the domain name that appears in the ‘host’ value is a duplicated value. Remove <your-custom-domain&gt;.com at the end of the ‘host’ value and you’ll be good to go.

After you save that record, it’ll take a bit of time before the certificate is issued. Once you receive the certificate in your email, proceed to step 8. If you haven’t already though, let’s set up the additional records needed in order to send people to <your-username>.gitlab.io/<your-project> when they enter <your-custom-domain>.com.

Step 7: Set up your host records in Namecheap

As outlined in GitLab’s docs, you’ll also need to prove on GitLab’s end of things that you own the custom domain that you want to serve your GitLab Pages site on.

As mentioned earlier, this guide assumes that you are just looking to use example.com (or www.example.com), so you’ll want to add the following host records:

• Type A Record, Host @, Value 35.185.44.232 (this is the current GitLab Pages IP at the time of writing)
• Type CNAME Record, Host www , Value example.com (this ensures that people who enter the 'www’ subdomain (i.e: www.example.com) still reach your site)
• Note: You won’t be able to enter this one until you’ve added the domain through the ‘New Pages Domain’ flow outlined in Step 8. Type TXT Record, Host @ , Value gitlab-pages-verification-code=11112222aaaabbbb

Step 8: Install the certificate in GitLab

Head on over to the ‘Pages’ page of your GitLab project that you’re trying to set up (under ‘Settings’ > ‘Pages’ in the sidebar).

To add your custom domain that GitLab serves your Pages site on, click on the ‘New Domain’ button on the top right. You should see something like the following:

Enter your custom domain (example.com) in the domain field, and then the next part is where it gets interesting.

If you try just to enter your certificate (example_com.crt) and your private key (generated when you initially sent the certificate request) in the fields, you’ll likely get a ‘Certificate is missing intermediates’ error.

This is because GitLab is using something like NGINX to receive requests on it’s Pages IP before routing the request to the correct site. Namecheap, in their documentation, calls out that “it is required to combine your certificate with CA certificates in a single file”.

What this means for you is that you need to combine the text found in your example_com.crt and example_com.ca-bundle files in the ‘certificate field’. In the end you should have something like:

Add the private key to the last field, and you’re done. It will take time for the changes to propagate. If you check back in a few hours, you should see an indication beside your address in the URL bar showing that your connection to your site is now secure.

Resources/References

• https://about.gitlab.com/features/pages/
• https://docs.gitlab.com/ee/user/project/pages/getting_started_part_three.html#dns-txt-record
• https://www.namecheap.com/support/knowledgebase/article.aspx/9474/69/how-do-i-create-a-pem-file-from-the-certificates-i-received-from-you
• https://www.namecheap.com/support/knowledgebase/article.aspx/9637/68/how-can-i-complete-the-domain-control-validation-dcv-for-my-SSL-certificate#dns
• https://stackoverflow.com/a/49124195/2719852

What would technology be without a computer doing periodic work?

Whether it’s your phone constantly checking your inbox for you, or getting timely alerts for weather or flight delays.

What about a bitcoin vs Canadian dollar price service, in just 56 seconds? No IFTTT, no Zapier, but no programming languages either — and no frameworks, no server or docker configuration, no Raspberry Pi, no AWS and no tests!

To make the example as universal as possible, we will only use 2 command lines:

• one to GET the bitcoin price from an API
• and another to POST it to another service.

Of course you can make this more useful by posting the price to Twitter, Twilio, Telegram, Slack and so on. But here we will simply post it to putsreq.com so we can inspect the POST request.

Then we will use GitLab-CI to schedule it to run everyday.

Level : All levels

Requirements : Any web browser

Let’s get you started :

1. Create a free account at gitlab.com (20 seconds)

2. Create a new Project : Click on the New Project button to create a new repo, and in the name field type periodic-job or any other name. (9 seconds)

Then save it by clicking on Create Project (1 second).

3. Create a .gitlab-ci.yml file in this new project: Click on New File, copy and paste the following snippet into the .gitlab-ci.yml file, then click save (5 seconds)

test:

 script:

 - btc=$(curl https://min-api.cryptocompare.com/data/price?fsym=BTC\&tsyms=CAD)

- curl -i -X POST https://putsreq.com/wkDdMQWhaOyalisaIe49 — data ‘price=CA$ ‘“${btc//[0-9\.]/}”

These are basically two simple commands. Here we can go further and add

if [ $btc -ge 15000 -a $btc -lt 7000 ]; then

conditions, or even run a full bash script file, but let’s keep it simple.

Click on the Commit changes button, and this will trigger it to build and run.

4. Schedule it to run everyday: click on the CI/CD icon to expand the menu, and select Schedules to set up a name and a timer for your periodic job to trigger. (11 seconds)

click on New schedule button

Type in a name for the new schedule daily-bitcoin-price-job, select to run it daily then click Save

Your scheduled job has been saved

5. Congrats! You’re done. Go to to this link in putsreq.com to see it in action. (10 seconds)

This job will run everyday as long as your free 2000/month build minutes do not run out.

We haven’t even scratched the surface of what we can do with GitLab-CI — just think of all the possibilities of using it to create webhooks or connecting it to IFTTT and Zapier, which in turn would connect it to hundreds of services.

In the next article we will go through what we have just done, and how we can take it up a notch and create a webhook and use it to post to social media.

You can find the sample code here, and you can read the build logs here.

An example using Docker to test and build your pipeline

In the past you may have tried different tools to manage the deployment of your applications effectively. In this tutorial, I’ll be showing you a quick and easy way to set up continuous integration for your environment using GitLab and Docker. So lets get started.

Here is what we’ll be setting up:

• Version control
• Issue Tracker
• Documentation
• Continuous Integration
• Continuous Delivery
• Repository (artifacts/docker images)

Tools like Jenkins are good for continuous integration and delivery. Mantis helps with issue tracking. But to improve the efficiency and quality of our project, we need to bring all these tools together. For example, we may want a git commit hook up with existing issues, or trigger an automated test after pushing commits to the master branch.

Usually, most tools already provide out of the box integration with other common services but it is still difficult to configure them at times. Moreover, the workflow would be broken if any one of the services in the chain go down. So it would be great if there were a single platform which could fulfil all these requires and that is why we’ve chosen GitLab.

GitLab CI

GitLab.com is a SAAS based service where you can host your Git repository, track issues and write the wiki in markdown. GitLab CI also allows you to setup continuous integration utilizing any Docker image available on Docker Hub. Let’s take a look at the following example.

The GitLab CI YML

GitLab CI uses a YAML file .gitlab-ci.yml to define the project configuration, which includes a definition of all the stages that need to be run after a CI/CD pipeline is triggered in response to a git push/merge. In this example, we have a simple Node.js project and we would like to make sure the code is good by linting and running a unit test. To play along, fork this repository and check it out.

In the above YAML configuration file we defined 3 stages. Each stage is just a gulp task defined in gulpfile.js. Anyone could run the task locally as long as the have Node.js installed. But in GitLab CI, we only need to mention which Docker image is needed. In our case, that is node:6.11.2. Furthermore, this image attribute could be defined within the stage definition so that you could use different tool for each stage.

The stage definition

Let’s take a deeper look in the stage definition.

The attributes of before_script and script can have multiple values (array in .yml). And if the script execution fails, the stage will be classified as failed.

Trigger the pipeline

Just make some changes on the master branch and you can find the pipeline running on the CI / CD -> Pipeline page.

The pipeline history

View the stage in detail

Click on a specific pipeline and you can read the console output of each stage. This is useful when the stage/job fails.

The stage output

The benefits of using GitLab CI with Docker

Different projects may require different dependencies such as Node.js, Ant, Maven. In the past when using a tool such as Jenkins, I’d have to make sure that all of these are installed on the server. By using Docker, the developer can reference dependencies available on Docker Hub without asking the server admin to setup such dependencies on the server each time. Actually Jenkins also has a pipeline plugin and it could work with Docker to serve exactly the same purpose. But extra effort on integrating Jenkins with version control is needed and as I mentioned before.

Although i prefer using GitLab CI, it doesn’t mean it could completely replace Jenkins. Jenkins offers a configurable user interface which is convenient for non-developers such as QA to execute certain tasks like deployments and integration tests.

Pick a suitable tool - It doesn’t have to be perfect

The key is not about choosing the perfect tool. Instead it is more about the people who use it. So before searching for a new tool, try to identify the problem which you would like to solve first.

— Originally posted on Boatswain Blog.

Your first app is a hit the day it’s launched. But one week later, you realize that it has no retention. You discover that this is because whenever a user clicks the “send” button, their comments get posted twice.

The bug was so minor, but it killed your momentum. But that’s okay. For your second app, you and your partner check more carefully. You both click, click, click your app all day and night to prevent minor bugs like that from happening again.

For one app, that’s okay. But after a year, you have a company building 7 apps on different platforms, including web, iOS, and Android. Your team now does code review before any app launch. You test through the apps and do your clicking before they’re shipped. But your nightmare from app #1 returns: users drop the app and this time it’s because their posts showing strange characters when they type an emoji. You end up with 1-star ratings after launch.

There are 3 types of product-making companies: those who do not test, those who test, and those who test fast, accurately and frequently.

Is an automated testing system with continuous integration (CI) just a dream? CI seems like a “nice-to-have”, especially since services that run tests and generate reports like Sauce Labs, BrowserStack, Test Complete are expensive. The good news is, there are many free and popular tools out there that you can mix and match to set up a free automated testing system. As a QA tester, I figured out a free testing pipeline setup, so I’m sharing it to save you time and money.

Why did my company want to set up an automated system? Here are some reasons below:

• We hate manual repetitive jobs that are prone to human error.
• We want a smoother process (test it when there is a code update), and reduce the waiting time.
• We want to schedule the tests and make them under control.

Setting up your UI tests

This post introduces a practical pipeline setup that can run web-based UI (User Interface) tests automatically and continuously. The later part maybe a bit technical but it’s quite fun to build!

I have set up the whole system with these free and open-source tools:

• Selenium — to script and automate browsers to perform tests
• Docker — to build an image for test environment and ship it fast
• Gitlab CI — to trigger, build and run the test upon code updates
• Skygear — to save test result for report on demand

Setting up takes 4 steps. Here we go!

Step #1: Write the script and run it locally

First of all, we write the test script, to let our original manual test run automatically. Selenium is a well-known tool for web automation. It supports different client languages including Java, Ruby, Python, etc.

Here’s an example on perform a button click on a website in Python.

Update: Added usage of Chrome official headless mode (details here).

With the idea of a basic unit test model, we could easily identify these three major components in the script:

• Set up
• Run test case
• Tear down

In this script, it will run test_case_1 and test_case_2 respectively, both with setUp before the test and tearDown after the test. We use unittest as our testing framework in this example. Feel free to use what you like, such as pytest or nose in Python.

You can add more test cases, such as filling in forms and clicking on elements, depending on your website's interface.

Step #2: Build an image with your testing environment

Test running requires a clean environment. To create a clean environment, we definitely do not want to set up a real machine every time and wait for hours to install all the software needed. The container concept helps.

Docker helps you build your testing environment into an image. The image includes all the software that needs to be pre-installed and run on that container like a virtual machine. With Docker, you can just create a new container and pull the same image every time you want to start over from your default environment.

To perform our test with the Selenium Python client, we want our image to pre-install the following:

• Python
• Google Chrome
• Chrome driver
• Xvfb

Xvfb is a virtual display server that helps you to launch a browser in a headless mode (without display). It is necessary to run the UI tests in a container. It cannot connect to a display output to show the browser visually.

Then, we will also install the Selenium package inside the container. Not all projects need the same list of packages.

We create a Dockerfile, build the image and upload to our Docker Cloud.

You could find this image through this link, or directly pull this image with this command:

docker pull joyzoursky/python-chromedriver:3.6-xvfb

Then you will have an environment ready for performing the UI tests.

Update: Added new Docker image built with Xvfb deprecated (details here).

Step #3: Set up GitLab CI

GitLab provides a CI/CD Pipelines feature, to continuously build and run your projects. The setup is like other CI tools such as Travis CI or Jenkins. This requires a .gitlab-ci.ymlfile to configure your build process.

Take a look at this example:

Update: Added example with Xvfb deprecated (details here).

When new codes are pushed to the repository, GitLab will look for .gitlab-ci.yml from the root directory, and trigger a build according to your settings.

In this script, it pulls the environment image from joyzoursky/python-chromedriver:3.6-xvfb in the first line. Then it installs the required packages like Selenium, sets variables needed, and then it starts the process.

Note that there are 2 stages of the build process in this example: test and report. In each stage, the jobs in that stage will be run concurrently. You can define tests in the same stage if they could run in sync.

Go to the Pipelines page to see the flow and completion here:

So where do we run our tests actually?

GitLab hosts some shared runners which are free. By looking into the build log, we can find the container information in the first few lines:

Running with gitlab-ci-multi-runner 1.10.4 (b32125f)Using Docker executor with image joyzoursky/python-chromedriver:3.5 ...Pulling docker image joyzoursky/python-chromedriver:3.5 ...Running on runner-4e4528ca-project-2749300-concurrent-0 via runner-4e4528ca-machine-1489737516-5e0de836-digital-ocean-4gb...

It shows the container name running on Digital Ocean.

Of course, you can also create your specific runners to run the test on your self-hosted machines. GitLab supports runners on different platforms including Docker and Kubernetes. But, as GitLab is a new platform, it goes through many updates. So the specific runners may sometimes break when they are out-of-date. You should always refer to the official repository when configuring the setup.

Step #4: Run and report periodically

You may want to have your tests run periodically. You can achieve this by setting up cron jobs, but you may not want to set up a server just to run a one-line cron job. My company’s open source serverless back-end is Skygear. We can use it to write a simple cloud code function with the @every decorator and trigger the test pipeline on an interval of time.

_[Skygear](https://skygear.io/" rel="noopener" target="blank" title=")’s developer portal. Look for your Cloud Code Git URL.

• Login to your Skygear portal
• Find your Cloud Code Git URL
• Clone the quick start codes
• Edit to add the little piece of code below
• Push the codes and the cron job will trigger the test every hour

Update: Used GitLab 10.0 pipeline scheduler instead of cron job (details here).

Assume that you have already written some code to generate test reports. Would you like to receive and read the test reports every hour? Of course not. So, we also link Skygear’s free Cloud DB service to store the test result. The system only sends alerts when a test case changes from PASS to FAIL, or FAIL to PASS. This notification approach may vary according to the project need.

To save and retrieve data from the Skygear database, we could use the existing SDK. Or if you are a Python user, you may use this little Python DB Client to help write your data handler . We use it to save test results after each test case, and retrieve the reports after running all test suites.

Finally, we can have the test result alerts sent on demand.

P.S. We use Slack real time messaging API to do the reporting, so we can receive notifications in the corresponding project channels.

Conclusion

Now, whenever there is a code update on the production branch, this automated UI test is triggered and tests are done automatically. Failure results will be pushed back to our Slack channel to notify our developers.

The biggest barrier to setting up a free automated UI test is probably researching the tools if you are not already a professional QA tester. Since QA is my full-time job, I hope that sharing our upgraded UI Test stack will help free up your time as well!

If you found this useful, please click the ? below so other people can see it too. Thank you!

Recently I came across a challenge to deploy a Python Flask web application to Heroku. The code of the App was hosted in GitLab.

Heroku supports deploying an App from GitHub, Dropbox along with the usual Heroku git. It had been quite a while since I used Heroku. I was wondering if I can deploy code directly from my GitLab repository instead of using any of the sources mentioned above.

I couldn’t find any information or documentation around deploying applications hosted in a GitLab repository to Heroku. I browsed a bit on GitLab and it turned out that apart from helping test and build your project, GitLab CI can also help deploy you App to your hosting infrastructure. I was now intrigued.

Before I delve into how I deployed the App, I’d like to explain the benefits of using GitLab or GitHub when you can easily get things done with Heroku Git.

1. Easier code maintenance - With code repository hosting services like GitHub and GitLab, maintenance of code is easy.
2. Customising pipelines - With GitLab, we can write our own yaml file and include the libraries required for running our application.
3. For better understanding of Continuous Integration and Continuous Development (CI/CD) - For beginners, this setup helps you understand the coding workflow of testing -> version control -> code maintenance -> application deployment.

Here are the steps required to deploy your App hosted in GitLab to Heroku. The steps here assume you already have a good understanding of Python, Flask, version control, GitLab and Heroku. This write up is also helpful for someone who is just starting out. I have kept it as simple as possible to get things up and running.

Uploading the project to GitLab

1. Create a Python virtual environment for us to use. Get into the virtual environment.
2. Create a sample Python Flask application on your machine.
3. Verify that everything is running fine.
4. Run the command pip freeze > requirements.txt from within the main application folder to catch all the requirements for running your application.
5. Create a Procfile which is used by Heroku to declare what commands are run by your application on the Heroku platform. Procfile usually consists of the web server used to run the application. In our case let us use Gunicorn, the default Python WSGI HTTP server. The content of your Procfile will be web: gunicorn <name of the app.py file>:<app-name> where app-name is usually “app”. Place this file within the main application folder.
6. Now login (or signup) to GitLab and create a project. The moment you do this, you will get a standard set of instructions on how to “link” your code on your development machine to GitLab project. Just follow the commands, and after that you can do a git push or git pull to/from this project. This is a bit of an elaborate step and your last statement should look something like git push -u origin master. Once done, on refreshing the project page on GitLab, you should see all your code appear in GitLab.

Project repository in GitLab

Linking GitLab and Heroku

1. Login to the Heroku web portal and create an application. Give it a nice name and runtime selection.
2. Now within the my_app folder on your development machine, create a file called “.gitlab-ci.yaml” (note the ‘.’ in the beginning).
3. This yaml file will have the following structure.

my_app_file_name:
 script:
 — apt-get update -qy
 — apt-get install -y python-dev python-pip
 — pip install -r requirements.txt
 — export MONGOHQ_URL=$MONGO_URL

production:
 type: deploy
 script:
 — apt-get update -qy
 — apt-get install -y ruby-dev
 — gem install dpl
 — dpl — provider=heroku — app=task-mgmt-app — api-key=$HEROKU_SECRET_KEY
 only:
 — master

1. Change my_app_file_name to the file name of your flask application. You need to set the HEROKU_SECRET_KEY variable in the project variables. You will get this key in the Heroku dashboard. To set it in your GitLab project, go to Settings > CI/CD Pipelines and search for Secret Variables. While using these variables in the yaml, we need to prepend the variable with the ‘$’ sign. It is a good practice not to share secret keys with anyone and also restrict access to them in the project.
2. You are almost there. Run the command git add .gitlab-ci.yml and git commit -m <msg> and git push -u origin master. You will see the file in the GitLab repository now.
3. On the GitLab “My Dashboard” page, click on Pipelines > Jobs to see the job which has started running.
4. In case you are using a database in your App, you might want to link it to the App by placing the details in the .gitlab-ci.yaml file. Please check here for an example. I have used MongDB in my application. Heroku provides adding a set of free libraries/apps to your application. There is an mLab link for adding MongoDB.

Jobs list in the Pipeline

_A running job. Screenshot related to my_app_filename in yaml.

Successful App deploy on Heroku

Yay! You have now successfully integrated your GitLab with Heroku with CI/CD configuration. Make all code changes you want in your repository, push it to the GitLab project, and see a job start every time there is a code push. For the current setup, I used GitLab public runners available here. You can set up a custom GitLab runner and set appropriate configuration.

Useful links :

1. Creating heroku remote
2. Setting up CI/CD from GitLab on Heroku
3. A task management application - repository : GitLab, deployed on Heroku