As you continue to develop your software, you must also continue to integrate it with previous code and deploy it to servers.

Manually doing this is a time-consuming process that can occasionally result in errors. So we need to do this in a continuous and automated manner – which is what you will learn in this article.

We'll go over how you can improve your MERN (MongoDB, Express, React, and NodeJs) app development process by setting up a CI/CD pipeline with Jenkins. You'll see how to automate deployment for faster, more efficient releases.

Let's Get Started

Prerequisites

• Basic understanding of MERN stack technologies.
• Basic understanding of Docker.
• Get source code from GitHub

The Problem

Consider this productivity app – it's a MERN project that we are going to use in this article. There are numerous steps we must complete, from building the application to pushing it to the Docker hub.

First, we must run tests with a command to determine whether all tests pass or not. If all tests pass, we build the Docker images and then push those images to Docker Hub. If your application is extremely complex, you may need to take additional steps.

Now, imagine that we're doing everything manually, which takes time and can lead to mistakes.

Waiting for deployment without devops meme

The Solution

To address this problem, we can create a CI/CD Pipeline. So, whenever you add a feature or fix a bug, this pipeline gets triggered. This automatically performs all of the steps from testing to deploying.

What is CI/CD and Why is it Important?

Continuous Integration and Continuous Deployment is a series of steps performed to automate software integration and deployment. CI/CD is the heart of DevOps.

CI/CD steps

From development to deployment, our MERN app goes through four major stages: testing, building Docker images, pushing to a registry, and deploying to a cloud provider. All of this is done manually by running various commands. And we need to do this every time a new feature is added or a bug is fixed.

But this will significantly reduce developer productivity, which is why CI/CD can be so helpful in automating this process. In this article, we will cover the steps up until pushing to the registry.

CI/CD meme

The Project

The project we are going to use in this tutorial is a very simple full-stack MERN application.

Project demo

It contains two microservices.

1. Frontend
2. Backend

You can learn more about the project here.

Both of these applications contains a Dockerfile. You can learn how to dockerize a MERN application here.

What is Jenkins?

To run a CI/CD pipeline, we need a CI/CD server. This is where all of the steps written in a pipeline run.

There are numerous services available on the market, including GitHub Actions, Travis CI, Circle CI, GitLab CI/CD, AWS CodePipeline, Azure DevOps, and Google Cloud Build. Jenkins is one of the popular CI/CD tools, and it's what we'll use here.

How to Set Up Jenkins Server on Azure

Because Jenkins is open source and it doesn't provide a cloud solution, we must either run it locally or self-host on a cloud provider. Now, running locally can be difficult, particularly for Windows users. As a result, I've chosen to self-host it on Azure for this demo.

If you want to run locally or self-host somewhere other than Azure (follow these guides by Jenkins), skip this section and proceed to the How to Configure Jenkins section.

First, you'll need to sign in to your Azure account (Create one if you don't have one already). Open Azure Cloud Shell.

Opening Azure Cloud Shell

Then create a directory called jenkins to store all the Jenkins config, and switch to that directory:

mkdir jenkins
cd jenkins

Create a file called cloud-init-jenkins.txt. Open with nano or vim,

touch cloud-init-jenkins.txt
nano cloud-init-jenkins.txt

and paste this code into it:

#cloud-config
package_upgrade: true
runcmd:
  - sudo apt install openjdk-11-jre -y
  - wget -qO - https://pkg.jenkins.io/debian-stable/jenkins.io.key | sudo apt-key add -
  - sh -c 'echo deb https://pkg.jenkins.io/debian-stable binary/ > /etc/apt/sources.list.d/jenkins.list'
  - sudo apt-get update && sudo apt-get install jenkins -y
  - sudo service jenkins restart

Here, we'll use this file to install Jenkins after creating a virtual machine. First, we install openjdk, which is required for Jenkins to function. The Jenkins service is then restarted after we install it.

Next, create a resource group. (A resource group in Azure is like a container that holds all the related resources of a project in one group. Learn more about resource groups here.)

az group create --name jenkins-rg --location centralindia

Note: make sure to change the location to the one closest to you.

Now, create a virtual machine.

az vm create \
--resource-group jenkins-rg \
--name jenkins-vm \
--image UbuntuLTS \
--admin-username "azureuser" \
--generate-ssh-keys \
--public-ip-sku Standard \
--custom-data cloud-init-jenkins.txt

You can verify the VM installation with this command:

az vm list -d -o table --query "[?name=='jenkins-vm']"

Don't be confused. This command simply displays JSON data in a tabular format for easy verification.

Jenkins server runs on port 8080, so we need to expose this port on our VM. You can do that like this:

az vm open-port \
--resource-group jenkins-rg \
--name jenkins-vm  \
--port 8080 --priority 1010

Now we can access the Jenkins dashboard in the browser with the URL http://<your-vm-ip>:8080. Use this command to get the VM IP address:

az vm show \
--resource-group jenkins-rg \
--name jenkins-vm -d \
--query [publicIps] \
--output tsv

You can now see the Jenkins application in your browser.

Jenkins dashboard

As you'll notice, Jenkins is asking us to provide an admin password which is automatically generated during its installation.

But first let's SSH into our virtual machine where Jenkins is installed.

ssh azureuser@<ip_address>

Now, type in the below command to get the password:

sudo cat /var/lib/jenkins/secrets/initialAdminPassword

Copy and paste it. Then click Continue.

How to Configure Jenkins

First, you'll need to click Install suggested plugins. It will take some time to install all the plugins.

Installing suggested plugins

An admin user is needed to restrict access to Jenkins. So go ahead and create one. After finishing, click Save and continue.

Create an admin user

Now you will be presented with the Jenkins dashboard.

The first step is to install the "Blue Ocean" plugin. Jenkins has a very old interface, which may make it difficult for some people to use. This blue ocean plugin provides a modern interface for some Jenkins components (like creating a pipeline).

To install plugins, go to Manage Jenkins -> click Manage Plugins under "System Configuration" -> Available plugins. Search for "Blue Ocean" -> check the box and click Download now and install after restart.

Blue ocean

Great, we're all set. Now let's create a pipeline.

How to Write a Jenkinsfile

To create a pipeline, we need a Jenkinsfile. This file contains all the pipeline configurations – stages, steps, and so on. Jenkinsfile is to Jenkins as a Dockerfile is to Docker.

Jenkinsfile uses the Groovy syntax. The syntax is very simple. You can understand everything by just looking at it.

Let's start by writing:

pipeline {

}

The word 'agent' should be the first thing you mention in the pipeline. An agent is similar to a container or environment in which jobs run. You can use multiple agents to run jobs in parallel. You can find more information about Jenkins agents can here.

pipeline {
    agent any
}

Here we are telling Jenkins to use any available agent.

We have a total of 5 stages in our pipeline:

CI/CD pipeline stages

Stage 1: Checkout code

Different CI/CD tools use different naming conventions. In Jenkins, these are referred to as stages. In each stage we write various steps.

Our first stage is checking out code from a source code management system (in our case, GitHub).

pipeline {
    agent any

    stages {
        stage('Checkout') {
            steps {
                checkout scm
            }
        }
    }
}

Commit the changes and push to your GitHub repo.

Since we haven't created any pipelines yet, let's do that now.

Before we begin, we must ensure that Git is installed on our system. If you followed my previous steps to install Jenkins on an Azure VM, Git is already installed.

You can test it by running the following command (make you are still SSHed into the VM):

git --version

If it isn't already installed, you can do so with:

sudo apt install git

Open blue ocean. Click Create new pipeline.

Creating new pipeline

Then select your source code management system. If you chose GitHub, you must provide an access token for Jenkins to access your repository. I recommend clicking on Create an access token here because it is a template with all of the necessary permissions. Then click Connect.

Selecting scm

After that, a pipeline will be created. Since our repository already contains a Jenkinsfile, Jenkins automatically detects it and runs the stages and steps we mentioned in the pipeline.

If everything went well, the entire page will turn green. (Other colors: blue indicates that the pipeline is running, red indicates that something went wrong in the pipeline, and gray indicates that we stopped the pipeline.)

Stage one successful

Stage 2: Run frontend tests

In general, all the CI/CD pipelines contains some tests that needs to be run before deploying. So I added simple tests to both the frontend and backend. Let's start with the frontend tests.

stage('Client Tests') {
    steps {
        dir('client') {
            sh 'npm install'
            sh 'npm test'
        }
    }
}

We're changing the directory to client/ because that's where the frontend code is. And then install the dependencies with npm install and run the tests with npm test in a shell.

Again, before we restart the pipeline, we have to make sure node and npm are installed or not. Install node and npm with these commands in the virtual machine:

curl -sL https://deb.nodesource.com/setup_16.x | sudo -E bash -

After that, run the following:

sudo apt-get install -y nodejs

Now, commit the code and restart the pipeline.

Run client tests

Stage 3: Run backend tests

Now do the same thing for the backend tests.

But there is one thing we need to do before we proceed. If you take a look at the codebase and activity.test.js, we are using a few environment variables. So let's add these environment varibales in Jenkins.

How to add environment variables in Jenkins

To add environment variables, go to Manage Jenkins -> click Manage Credentials under "Security" -> System -> Global credentials (unrestricted) -> click + Add Credentials.

For Kind select "Secret text", leave Scope default, and for Secret write the secret value and ID. This is what we use when using these environment variables in the Jenkinsfile.

Add the following env variables:

Environment variables

Then in the Jenkinsfile, use these env variables:

environment {
    MONGODB_URI = credentials('mongodb-uri')
    TOKEN_KEY = credentials('token-key')
    EMAIL = credentials('email')
    PASSWORD = credentials('password')
}

Add a stage to install dependencies, set these variables in the Jenkins environment, and run the tests:

stage('Server Tests') {
    steps {
        dir('server') {
            sh 'npm install'
            sh 'export MONGODB_URI=$MONGODB_URI'
            sh 'export TOKEN_KEY=$TOKEN_KEY'
            sh 'export EMAIL=$EMAIL'
            sh 'export PASSWORD=$PASSWORD'
            sh 'npm test'
        }
    }
}

Again, commit the code and restart the pipeline.

Run server tests

Stage 4: Build Docker images

Now, we have to specify a step to build the Docker images from the Dockerfiles.

Before we proceed, install Docker in the VM (if you don't already have it installed).

To install Docker:

sudo apt install docker.io

Add the user jenkins to the docker group so that Jenkins can access the Docker daemon – otherwise you'll get a permission denied error.

sudo usermod -a -G docker jenkins

Then restart the jenkins service.

sudo systemctl restart jenkins

Add a stage in the Jenkinsfile.

stage('Build Images') {
    steps {
        sh 'docker build -t rakeshpotnuru/productivity-app:client-latest client'
        sh 'docker build -t rakeshpotnuru/productivity-app:server-latest server'
    }
}

Commit the code and restart the pipeline.

Build docker images

Stage 5: Push images to the registry

As a final stage, we will push the images to Docker hub.

Before that, add your docker hub username and password to the Jenkins credentials manager, but for Kind choose "Username with password".

Username with password type credential

Add the final stage where we login and push images to Docker hub.

stage('Push Images to DockerHub') {
    steps {
        withCredentials([usernamePassword(credentialsId: 'dockerhub', passwordVariable: 'DOCKER_PASSWORD', usernameVariable: 'DOCKER_USERNAME')]) {
            sh 'docker login -u $DOCKER_USERNAME -p $DOCKER_PASSWORD'
            sh 'docker push rakeshpotnuru/productivity-app:client-latest'
            sh 'docker push rakeshpotnuru/productivity-app:server-latest'
        }
    }
}

Push images to dockerhub

Here is the complete Jenkinsfile:

// This is a Jenkinsfile. It is a script that Jenkins will run when a build is triggered.
pipeline {
    // Telling Jenkins to run the pipeline on any available agent.
    agent any

    // Setting environment variables for the build.
    environment {
        MONGODB_URI = credentials('mongodb-uri')
        TOKEN_KEY = credentials('token-key')
        EMAIL = credentials('email')
        PASSWORD = credentials('password')
    }

    // This is the pipeline. It is a series of stages that Jenkins will run.
    stages {
        // This state is telling Jenkins to checkout the source code from the source control management system.
        stage('Checkout') {
            steps {
                checkout scm
            }
        }

        // This stage is telling Jenkins to run the tests in the client directory.
        stage('Client Tests') {
            steps {
                dir('client') {
                    sh 'npm install'
                    sh 'npm test'
                }
            }
        }

        // This stage is telling Jenkins to run the tests in the server directory.
        stage('Server Tests') {
            steps {
                dir('server') {
                    sh 'npm install'
                    sh 'export MONGODB_URI=$MONGODB_URI'
                    sh 'export TOKEN_KEY=$TOKEN_KEY'
                    sh 'export EMAIL=$EMAIL'
                    sh 'export PASSWORD=$PASSWORD'
                    sh 'npm test'
                }
            }
        }

        // This stage is telling Jenkins to build the images for the client and server.
        stage('Build Images') {
            steps {
                sh 'docker build -t rakeshpotnuru/productivity-app:client-latest client'
                sh 'docker build -t rakeshpotnuru/productivity-app:server-latest server'
            }
        }

        // This stage is telling Jenkins to push the images to DockerHub.
        stage('Push Images to DockerHub') {
            steps {
                withCredentials([usernamePassword(credentialsId: 'dockerhub', passwordVariable: 'DOCKER_PASSWORD', usernameVariable: 'DOCKER_USERNAME')]) {
                    sh 'docker login -u $DOCKER_USERNAME -p $DOCKER_PASSWORD'
                    sh 'docker push rakeshpotnuru/productivity-app:client-latest'
                    sh 'docker push rakeshpotnuru/productivity-app:server-latest'
                }
            }
        }
    }
}

Pipeline ran successfully

Conclusion

In summary, let's review what we've covered:

• We explored the significance of implementing Continuous Integration and Continuous Deployment (CI/CD) in software development.
• We delved into the fundamentals of Jenkins and acquired knowledge on how to deploy a Jenkins server on the Azure cloud platform.
• We customized Jenkins to meet our specific requirements.
• Lastly, we wrote a Jenkinsfile and built a pipeline utilizing the user-friendly interface of Jenkins Blue Ocean.

That's all for now! Thanks for reading 🙂.

Connect with me on twitter.

Create a MongoDB Atlas Account

MongoDB Atlas is a MongoDB Database-as-a-Service platform, which means that they configure and host the database for you. Then, your only responsibility will be to populate your database with what matters: data.

• Go here to sign up for a new MongoDB Atlas account.
• Fill in the registration form with your information and click Sign up.

Create a New Cluster

• On the next page, fill in your organization's name, project's name, select JavaScript as your preferred programming language, and click the green Continue button.
• Once you create and verify your account, answer the onboarding questions (your goal, the type of application you're building, your preferred programming language, etc.) and click the green Finish button.
• On the "Deploy a cloud database" page, click the Create button under the Shared cluster type. This should be the only free option:

• In the Cloud Provider & Region dropdown, leave everything as default. Your value there will likely depend on the region you're in.
• In the Cluster Tier dropdown, leave this as the default, M0 Sandbox (Shared RAM, 512 MB Storage).
• In the Cluster Name dropdown, you can give your cluster a name, or leave it as the default, Cluster0.
• Click the green Create Cluster button at the bottom of the screen.
• You should now see the message "M0 Cluster Provisioning... This process will take 3-5 minutes." Wait until the cluster is created before going to the next step.

Create a New User for the Database

• On the left side of screen under SECURITY, click on Database Access.
• Click the green Add New Database User button.
• Under Authentication Method, make sure Password is selected, then enter in a username and password for your user.
• Under Database User Privileges, leave this as the default option if there is one – it should be Read and write to any database. You may have to select this manually if the default is blank. You can select "Read and write to any database" from the "Add Built In Role" button as shown here:

Click "Add Built In Role" to select "Read and Write to Any Database".

• Click the Add User button to create your new user.

Allow Access From All IP Addresses

• On the left side of the screen under SECURITY, click on Network Access.
• Click the green Add IP Address button.
• In the modal, click the ALLOW ACCESS FROM ANYWHERE button. You should see 0.0.0.0/0 in the Access List Entry field.
• Click the green Confirm button.

Connect to Your Cluster

• On the left side of the screen under DEPLOYMENT, click on Database.
• Click the Connect button for your cluster:

• In the popup modal, click on Connect your application.
• You should see the URI string you'll use to connect to your database similar to this: mongodb+srv://<username>:<password>@<cluster-name>.prx1c.mongodb.net/<db-name>?retryWrites=true&w=majority.
• Click the Copy button to copy your URI to your clipboard.

Notice that the <username> and <cluster-name> fields of the URI you copied are already filled out for you. All you need to do is replace the <password> field with the one you created in the previous step, and be sure to add the name of your database before the query string (?retryWrites=true&w=majority).

You can call your database anything, but it's good to give it a memorable name for your project. For example, if you're working on the "MongoDB and Mongoose" challenges, you could replace <db-name> with fcc-mongodb-and-mongoose or something similar.

Connect to An Existing Database

If you've already created a cluster and a database and would like to connect it to a new application, follow these steps:

• On the left side of the screen under DEPLOYMENT, click on Database.
• Find your cluster and click the Browse Collections button to see a list of existing databases and collections.
• Copy the database name you want to connect to and replace <db-name> with it in the URI string above.

And that's it — you now have the URI to add to your application and connect to your database. Keep this URI safe somewhere so you can use it later.

Why you need this?

Because JS + Python + Mongo = full data development

I am a Meteor developer. More precisely I use Vulcan.js, but that’s a whole other story. Meteor is a full-fledged Node.js framework, pretty nice for SaaS, real-time apps, prototyping, management softwares and a lot of other use cases.

As a default, it works with MongoDB. MongoDB is a NoSQL database management system (DBMS). It stores documents as JSON and its shell uses JavaScript, which are some reasons why it is so appreciated by Node.js developers.

There is a neat tool called Meteor Up to automatically deploy Meteor apps and the associated Mongo databases on production servers. Neat is actually an understatement, it’s freakin’ awesome.

And I am also a Python developer. Python is excellent for web scraping and data science. It’s both easy to use and suited for high performance.

Sometimes, I like to use both Python to extract and process data and Meteor to create the user interface. I call this “full data development”, as it goes from the raw data source to the end user interface.

The sacred three of the full data developer: Meteor, Mongo, Python

But there is a problem: Meteor Up currently does not expose the Mongo database, only the local Meteor app can connect to it. So I cannot plug my Python servers to my Meteor managed databases out-of-the-box :(

A paid service would do the job by providing an URL for the hosted database, such as this one:

mongo://username:password@somedomain.com:27017

But why spend money when you can struggle for hours to set up your own Mongo database on AWS using Docker, and learn tons of useful tricks in the process? It’s even better if someone, say, me, does it first and writes a lengthy tutorial to make things easier for you!

Important note: it’s perfectly fine if it takes you multiple sessions to finish this looong tutorial. Don’t give up! The result is worth the hassle, as mastering Docker and AWS are two skills very appreciated by employers, and very useful in real life. Post your questions as comments if you have any, I’ll do my best to answer all of them.

1 — Let’s discover Docker

Our first container

I invite you to read Docker’s official installation doc here and install it, which takes only a few minutes. Then, let’s play a little. Run the following commands in your terminal and observe the results:

docker run --name my-lame-db -d mongo
docker ps

As simple as that, we just created an isolated container that runs Mongo!

You can access the official Mongo image Docker files somewhere in this folder hosted on GitHub to understand better what happens here. According to the last lines of the Dockerfile our db is available on port “27017” as a default:

EXPOSE 27017 CMD ["mongod"]

But Mongo exposes itself on this port INSIDE the container. But the container is isolated, so only programs INSIDE the container can talk to Mongo. Our database is up and running but trapped alone in its container :/

That’s lame! Let’s free it!

Open the container, map the ports

If you want to access Mongo from OUTSIDE the container, you’ll have to map the exposed port and a port of the machine. The -p option is specifically meant for this:

docker run -p 27017:27017 --name my-local-db -d mongo

If you had a Node server, you’d write docker run -p 80:3000 my-node-app for example. Your server running on port 3000 would thus be available through HTTP (port 80). First the container port, then the image port.

Let’s try to access our db in the browser, just for fun:

Nice! Our DB is running! You can try docker stop my-awesome-db and docker start my-awesome-db commands. You’ll see this address becoming not available and then available again.

As expected, you can’t connect to your db through the browser. But this rejection message comes from Mongo, which is a good sign. Let’s try again using the official CLI tool:

mongo localhost:27017 #or just "mongo", as this is the default uri

You can access your database shell, we are making progress!

But… I want my database to be on a distant server, not to run locally on my isolated machine.

2 — Access AWS from your terminal

We picked AWS as our cloud provider because it is widely spread, but keep in mind this is just an example. AWS offers to new users free hosting for 12 months for one server instance, so you don’t need to pay to follow this tutorial. The steps are mostly similar if you pick another hosting service.

First step is to create a programmatic access to Amazon Web Services using the IAM service (Identity and Access Management). This key will be used by Docker Machine (see below) to do some operations, such as creating an AWS EC2 instance for you.

Setting this up is outside the scope of our tutorial. Therefore, I invite you to read the first part of this article from Vishal Kumar. Follow the first 8 steps until you get AWS credentials. The rest of the article is interesting too but a bit advanced at this point and targeted to the MEAN stack. Let’s focus on Mongo only for the moment.

At this point you should have your credentials.

[default] 
aws_access_key_id = [access key from the downloaded credential file] 
aws_secret_access_key = [secret access key from the downloaded credential file]

Keep them safe! You will have to recreate a key pair if you lose them… and you certainly DON’T want anybody to discover them either!

3 —Put Docker in the cloud with Docker Machine

Never need the AWS console again

Docker Machine is an utility to manage the different machines hosting your containers (local machines, cloud servers). Hence the name, Docker… Machine. You can set it up for multiple cloud providers and it works very well with AWS.

The official documentation gives all the information you need for AWS. I just jumped the VPC part, it’s a bit too advanced for the moment, but the rest is very useful.

Create the EC2 instance

You will end up writing a command similar to the one below.

docker-machine create \
--driver amazonec2 \
--amazonec2-access-key ***** --amazonec2-secret-key **** \
--amazonec2-region ***** \
--amazonec2-open-port 27017 my-awesome-server

Let’s break it down.

• I use the EC2 driver, since I am an AWS user.
• I pass the credentials inline as I have multiple accounts to manage
• I pass the region where I usually host my apps (“eu-west-3” for me),
• Don’t forget to open port 27017 (--amazonec2-open-port 27017), otherwise AWS will block connections even if your container is correctly set.

If you did not specify the region correctly, you may have trouble finding your instance on the AWS console. I still don’t get why you can’t have all zones displayed easily in this interface, but that’s how it works.

You should also be able to get rid of the access key and secret key somehow by setting your local machine AWS credentials, or store them as environment variables. I personally prefer to have them in the command line as I may replace them with environment variables.

At this point, you can check the AWS console and see your instance set as expected!

4 — Run a Mongo container on your instance

Activate the correct machine

Now the tricky part, it took me a while to get it right. Docker Machine’s role is to manage your distant machines, launch instances and install Docker on it. That’s all. It’s NOT Docker.

So you still have to use Docker. But how does Docker know which machine to connect too?

Try this and look at the result:

docker-machine env my-awesome-server

It will show a small shell script to setup environment variables.

It also tells you to run this command:

eval $(docker-machine env my-awesome-server)

This will simply run the displayed script in your shell. When those env variables are set, your machine becomes “active”.

Type docker-machine active to check that the correct machine is listed.

\o/

Now type docker info , you should see that the Name matches your app. Magic! Docker is “connected” to the active machine, configured by Docker Machine.

We can now run the same command as earlier:

docker run -p 27017:27017 --name my-awesome-db -d mongo .

It will create the database not on our local machine, but on the distant server. As simple as that!

Run this command to get your machine IP:

docker-machine ip my-awesome-server

And open the address http://<ip-given-by-docker-machine>:27017 : you should get a beautiful error message, telling you that you are trying to access MongoDB over HTTPS: it works!!!

Run mongo <ip-given-by-docker-machine>:27017 … and… you are in ! Congratulations, you just setup your Docker container on a production server.

Ok now, we definitely DON’T want the whole world to access our db, so next step is setting up authentication.

5 — Setup authentication — 2 containers solution

Sorry but you can already delete the container you’ve just created (exercise: I let you figure out the container stopping and removing commands). It was just an example. Sadly, you won’t be able to setup auth with it.

Stay focused because the process is a bit trickier than you would expect:

• We will create a first container, let’s name it “Container #1”, WITHOUT auth.
• We will setup this container so data is saved on the server disk (that’s an important step), create and admin, and delete the container.
• We will create a second container, “Container #2” WITH auth. The admin user credentials will still be valid, because they are saved on the disk.

There might be simpler solutions I am not yet aware of, for example by providing the admin user credentials during the container creation, so feel free to comment if you are a Docker/Mongo superhero! And a one container solution is also provided in the annexes at the end of the article.

Share data between containers

At this point you should think “well, containers are isolated, so how can Container #1 and Container #2 share the same admin user”? And you are right. The solution lies in storing files on the server.

Remember when we mapped ports, so that the container’s open port maps Mongo’s port? We will apply the same logic to the file system: we can map folders on the container and folders on the server.

Let’s add an option to our command, and let’s call it the Final command for Container #1:

docker run \
-d \
-p 27017:27017 \
--name my-awesome-db \
-v ~/dataMongo:/data/db mongo \
mongod

Smart! Now when our Mongo containers will read/write their data in data/db they will also be available to the host server, in the ~/dataMongo folder (host folder first, then container folder).

Create the admin user

It’s all in the title: you will now connect to your newly created Mongo instance and create the admin user.

docker-machine ip my-awesome-server
mongo <resulting-ip>

Now you should be connected to your database shell. You simply need to create a super admin user:

db.createUser(
  {
    user: "admin",
    pwd: "yourpassword",
    roles: [ { role: "userAdminAnyDatabase", db: "admin" } ],
    passwordDigestor : "server"
  }
)

You can look at the official doc, this article and this stack overflow thread for more information.

Also, see this thread about the difference between the root role, which is an actual super admin, and userAdminAnyDatabase role, which is a kind of “user creator”.

Delete the container and activate auth

We can remove Container #1, it’s role was only to let us access a Mongo shell on our server, but it lacked authentication.

docker stop my-awesome-db
docker rm my-awesome-db

We can then create our final container, with authentication activated. Since it will be connected to the server file system too, the data created using Container #1 are still available, including our admin user.

We simply add the --auth option to the initial command, it tells Mongo… well, to enable authentication. You guessed right.

Final command for Container #2:

docker run \
-d \
-p 27017:27017 \
--name my-awesome-db \
-v ~/dataMongo:/data/db mongo \
--auth
mongod

Now connect again to your Mongo instance and run:

db.createCollection('IAMAHACKER')

You’ll get a nice error message as expected!

And now try again while being authenticated:

mongo $(docker-machine ip my-awesome-server):27017 -u admin -p yourpassword

If it works, you are done! Congratulations ;)

Now what?

Your Mongo database is up and running in the cloud, safely isolated in its Docker container, and secured with username/password authentication.

The next steps will be plugging your applications to this database. Remember you can access your server IP using the command docker-machine ip my-awesome-server. You will certainly have to create additional users to administrate your databases.

Also, you probably won’t want anybody to connect to your database shell, even with authentication set. AWS will let you whitelist a few IPs, corresponding to your applications and your own computer, so that only trusted sources can connect to your database.

Below this article, you’ll find a glossary, the final script, and a one-container solution to setup authentication, and a few additional tips.

I hope you found this tutorial helpful! If it’s the case, don’t forget to clap ;) And please check out the annexes below…

Annex 1 — Final script

# Launch an EC2 instance
docker-machine create \
--driver amazonec2 \
--amazonec2-access-key ***** --amazonec2-secret-key **** \
--amazonec2-region ***** \
--amazonec2-open-port 27017 my-awesome-server
# Activate the instance
eval $(docker-machine env my-awesome-server)
# Create container 1
docker run \
-d \
-p 27017:27017 \
--name my-awesome-db \
-v ~/dataMongo:/data/db mongo \
mongod
# Connect to your DB
mongo $(docker-machine ip my-awesome-server):27017
# IN THE MONGO SHELL create a super admin
db.createUser(
  {
    user: "admin",
    pwd: "yourpassword",
    roles: [ { role: "userAdminAnyDatabase", db: "admin" } ],
    passwordDigestor : "server"
  }
)
# QUIT THE MONGO SHELL
# Delete the container and recreate one with auth
docker stop my-awesome-db
docker rm my-awesome-db
# Run the final container
docker run \
-d \
-p 27017:27017 \
--name my-awesome-db \
-v ~/dataMongo:/data/db mongo \
--auth \
mongod
# IF CA FAILS WITH PERMISSION DENIED ERROR
# see https://github.com/kubernetes/minikube/issues/3083
# sudo vim /var/lib/snapd/apparmor/profiles/snap.docker.docker
# Add following line inside file (e.g alongside with other "owner" lines): 
# owner @{HOME}/.docker/machine/machines/** r,
# sudo apparmor_parser -r /var/lib/snapd/apparmor/profiles/snap.docker.docker

Annex 2 — Neat tricks I wish I did not have to learn by myself

• If you are in devops, you will eventually get a muscle memory of Docker commands. Otherwise, if you setup projects for production only once a year, WRITE DOWN EVERYTHING. For example, you could write a Medium article.
  Seriously I’ve lost a few hours of work because I did not write down my progression. The next week I had forgotten everything. Docker is not trivial.
• I ran into a permission issue with cert authentication when using Docker Machine, the solution is in the comments of this thread: https://github.com/kubernetes/minikube/issues/3083
• docker ps -a will print all containers, including inactive ones, while docker ps only prints active containers.
• An image is NOT a Dockerfile. A Dockerfile is a configuration file. Think of the blueprint of a boat. An image is a built Docker file. Think of a boat THAT IS NOT YET in the sea. A container is an image that you are running. Think of a boat ROAMING THE SEAS.

Differences between a Dockerfile, Docker Image and Docker Container
_Quick Jump: It All Starts With a Dockerfile | If You Build It, They Will Run (Usually) In casual conversation you may…_nickjanetakis.com

• docker pull mongo will for example pull a Mongo Docker image, already built. But how to display the Dockerfile associated? You can’t. If you want to check it, you can look for the relevant GitHub repository (if it exists). That’s the exact same relation between an installed NPM package and its code. npm install does install a build (an “image”) but does not necessarily download the code (a “Dockerfile” here), which usually lives on GitHub but can also be private.

How can I view the Dockerfile in an image?
_Hi, Looking at some images in the repository (this one, for example: https://hub.docker.com/r/filippobosi/mfi/) I do…_forums.docker.com

• On EC2, your instance IP can change over restarts! You’ll have to regenerate certs to keep connecting to the instance using docker-machine regenerate-certs my-awesome-server

Annex 3— Setup authentication — one container solution

There is also a “one container” solution. It’s less instructive, but faster and requires only one container.

Note : if you already setup auth using the 2 containers approach, you are already done. If you still want to try the 1 container approach, you’ll need to create a new instance on EC2 or remove the ~/dataMongo folder of the server.

You still need to share your folders with the host, otherwise your data are tied to the container, which should not happen! Containers should be easily deletable without any data loss, so documents have to be stored elsewhere.

The idea is to connect to your server and access Mongo from there, instead of accessing Mongo from your local machine. It’s a very subtle difference but that’s what saves us one step.

If you connect to Mongo while being connected to your server, you will have more permission and will be able to setup an admin user even if auth is already set, as Mongo considers you as a “local” user. To rephrase it, since you were able to connect to the machine Mongo is running on, your AWS instance, Mongo already considers you as a “safe” user, as a hacker should not be there in the first place.

Run only the Final command for Container #2

docker run \
-d \
-p 27017:27017 \
--name my-awesome-db \
-v ~/dataMongo:/data/db mongo \
--auth
mongod

You can always ssh your server using docker-machine ssh:

docker-machine ssh my-awesome-server

Running mongo will open the Mongo shell. From there, you can add an admin user EVEN is auth is set, as we have done previously.

Annex 4— Glossary

A bit of devops people mumbling.

AWS: Amazon Web Services, a famous collection of cloud services with cheap offers.

AWS EC2: service for hosting server instances. That’s where you’d host your API or your whole website. Note that there might be services more suited for DB hosting but since EC2 is a must-know, it is my choice as a beginner.

Docker: it’s a… well, a program? In a nutshell that’s a bunch of things that helps you to run programs in containers, isolated from the other programs running on the server, without costing as many resources as a virtual machine. Containers are also helpful to manage services (APIs, databases etc.) on a daily basis: kill/restart it, create new instance in one line…

Container: that will be my simplified definition, think of a virtual machine without an OS. It provides isolation while not consuming too many computing resources.

Dockerfile: that’s a config file for docker, which defines all the programs/files/commands/options you need to run your app.

Docker Machine: a CLI to push your Docker containers where you want, in the cloud or on your own servers.

I am the co-founder of the French company Lebrun Burel Knowledge Engineering (LBKE) — https://www.lbke.fr

Always happy to talk about code, machine learning, innovation and entrepreneurship!