There are many public and private registries available to developers such as Docker Hub, Amazon ECR, and Google Cloud Artifact Registry. But sometimes, instead of relying on an external vendor, you might want to host your images yourself. This gives you more control over how the registry is configured and where the container images are hosted.

This article is a hands-on tutorial that’ll teach you how to self-host a Container Registry.

Table of Contents

• What is a Container Image?

• What is a Container Registry?

• Why you might want to self-host a Container Registry

• How to self-host a Container Registry

• Step 1: Install Docker and Docker Compose on the server

• Step 2: Configure and run the registry container

• Step 3: Run NGINX for handling TLS

• Ready to go!

• Other options

• Conclusion

You will get the most out of this article if you’re already familiar with the tools like Docker and NGINX, and have a general understanding of what a container is.

What is a Container Image?

Before we talk about container registries, let's first understand what a container image is. In a nutshell, a container image is a package that includes all of the files, libraries, and configurations to run a container. They are composed of layers where each layer represents a set of file system changes that add, remove, or modify files.

The most common way to create a container image is to use a Dockerfile.

# build an image
docker build -t pliutau/hello-world:v0 .

# check the images locally
docker images
# REPOSITORY    TAG       IMAGE ID       CREATED          SIZE
# hello-world   latest    9facd12bbcdd   22 seconds ago   11MB

This creates a container image that is stored on your local machine. But what if you want to share this image with others or use it on a different machine? This is where container registries come in.

What is a Container Registry?

A container registry is a storage catalog where you can push and pull container images from. The images are grouped into repositories, which are collections of related images with the same name. For example, on Docker Hub registry, nginx is the name of the repository that contains different versions of the NGINX images.

Some registries are public, meaning that the images hosted on them are accessible to anyone on the Internet. Public registries such as Docker Hub are a good option to host open-source projects.

On the other hand, private registries provide a way to incorporate security and privacy into enterprise container image storage, either hosted in cloud or on-premises. These private registries often come with advanced security features and technical support.

There is a growing list of private registries available such as Amazon ECR, GCP Artifact Registry, GitHub Container Registry, and Docker Hub also offers a private repository feature.

As a developer, you interact with a container registry when using the docker push and docker pull commands.

docker push docker.io/pliutau/hello-world:v0

# In case of Docker Hub we could also skip the registry part
docker push pliutau/hello-world:v0

Let's look at the anatomy of a container image URL:

docker pull docker.io/pliutau/hello-world:v0@sha256:dc11b2...
                |            |            |          |
                ↓            ↓            ↓          ↓
             registry    repository      tag       digest

Why You Might Want to Self-host a Container Registry

Sometimes, instead of relying on a provider like AWS or GCP, you might want to host your images yourself. This keeps your infrastructure internal and makes you less reliant on external vendors. In some heavily regulated industries, this is even a requirement.

A self-hosted registry runs on your own servers, giving you more control over how the registry is configured and where the container images are hosted. At the same time it comes with a cost of maintaining and securing the registry.

How to Self-host a Container Registry

There are several open-source container registry solutions available. The most popular one is officially supported by Docker, called registry, with its implementation for storing and distributing of container images and artifacts. This means that you can run your own registry inside a container.

Here are the main steps to run a registry on a server:

• Install Docker and Docker Compose on the server.

• Configure and run the registry container.

• Run NGINX for handling TLS and forwarding requests to the registry container.

• Setup SSL certificates and configure a domain.

Step 1: Install Docker and Docker Compose on the server

You can use any server that supports Docker. For example, you can use a DigitalOcean Droplet with Ubuntu. For this demo I used Google Cloud Compute to create a VM with Ubuntu.

neofetch

# OS: Ubuntu 20.04.6 LTS x86_64
# CPU: Intel Xeon (2) @ 2.200GHz
# Memory: 3908MiB

Once we're inside our VM, we should install Docker and Docker Compose. Docker Compose is optional, but it makes it easier to manage multi-container applications.

# install docker engine and docker-compose
sudo snap install docker

# verify the installation
docker --version
docker-compose --version

Step 2: Configure and run the registry container

Next we need to configure our registry container. The following compose.yaml file will create a registry container with a volume for storing the images and a volume for storing the password file.

services:
  registry:
    image: registry:latest
    environment:
      REGISTRY_AUTH: htpasswd
      REGISTRY_AUTH_HTPASSWD_REALM: Registry Realm
      REGISTRY_AUTH_HTPASSWD_PATH: /auth/registry.password
      REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY: /data
    volumes:
      # Mount the password file
      - ./registry/registry.password:/auth/registry.password
      # Mount the data directory
      - ./registry/data:/data
    ports:
      - 5000

The password file defined in REGISTRY_AUTH_HTPASSWD_PATH is used to authenticate users when they push or pull images from the registry. We should create a password file using the htpasswd command. We should also create a folder for storing the images.

mkdir -p ./registry/data

# install htpasswd
sudo apt install apache2-utils

# create a password file. username: busy, password: bee
htpasswd -Bbn busy bee > ./registry/registry.password

Now we can start the registry container. If you see this message, than everything is working as it should:

docker-compose up

# successfull run should output something like this:
# registry | level=info msg="listening on [::]:5000"

Step 3: Run NGINX for handling TLS

As mentioned earlier, we can use NGINX to handle TLS and forward requests to the registry container.

The Docker Registry requires a valid trusted SSL certificate to work. You can use something like Let's Encrypt or obtain it manually. Make sure you have a domain name pointing to your server (registry.pliutau.com in my case). For this demo I already obtained the certificates using certbot and put it in the ./nginx/certs directory.

Since we're running our Docker Registry in a container, we can run NGINX in a container as well by adding the following service to the compose.yaml file:

services:
  registry:
    # ...
  nginx:
    image: nginx:latest
    depends_on:
      - registry
    volumes:
      # mount the nginx configuration
      - ./nginx/nginx.conf:/etc/nginx/nginx.conf
      # mount the certificates obtained from Let's Encrypt
      - ./nginx/certs:/etc/nginx/certs
    ports:
      - "443:443"

Our nginx.conf file could look like this:

worker_processes auto;

events {
    worker_connections 1024;
}

http {
    upstream registry {
        server registry:5000;
    }

    server {
        server_name registry.pliutau.com;
        listen 443 ssl;

        ssl_certificate /etc/nginx/certs/fullchain.pem;
        ssl_certificate_key /etc/nginx/certs/privkey.pem;

        location / {
            # important setting for large images
            client_max_body_size                1000m;

            proxy_pass                          http://registry;
            proxy_set_header  Host              $http_host;
            proxy_set_header  X-Real-IP         $remote_addr;
            proxy_set_header  X-Forwarded-For   $proxy_add_x_forwarded_for;
            proxy_set_header  X-Forwarded-Proto $scheme;
            proxy_read_timeout                  900;
        }
    }
}

Ready to go!

After these steps we can run our registry and Nginx containers.

docker-compose up

Now, on the client side, you can push and pull the images from your registry. But first we need to login to the registry.

docker login registry.pliutau.com

# Username: busy
# Password: bee
# Login Succeeded

Time to build and push our image to our self-hosted registry:

docker build -t registry.pliutau.com/pliutau/hello-world:v0 .

docker push registry.pliutau.com/pliutau/hello-world:v0
# v0: digest: sha256:a56ea4... size: 738

On your server you can check the uploaded images in the data folder:

ls -la ./registry/data/docker/registry/v2/repositories/

Other options

Following the example above, you can also run the registry on Kubernetes. Or you could use a managed registry service like Harbor, which is an open-source registry that provides advanced security features and is compatible with Docker and Kubernetes.

Also, if you want to have a UI for your self-hosted registry, you could use a project like joxit/docker-registry-ui and run it in a separate container.

Conclusion

Self-hosted Container Registries allow you to have complete control over your registry and the way it's deployed. At the same time it comes with a cost of maintaining and securing the registry.

Whatever your reasons for running a self-hosted registry, you now know how it's done. From here you can compare the different options and choose the one that best fits your needs.

You can find the full source code for this demo on GitHub. Also, you can watch it as a video on our YouTube channel.

What if it's not just one server but many? The amount of time you will need to fix all of them will grow exponentially – and because is a manual process it will be more error-prone.

And then the nightmare scenario: You don't have an up-to-date backup, or you have incomplete backups. Or the worst – there are no backups at all. This last case is more common than you think, especially in home labs where you are tinkering and playing around with stuff by yourself.

In this tutorial, I'll show you how you can do a full infrastructure provisioning of a pair of web servers on a Cloud provider, with SSL certificates and monitoring metrics with Prometheus.

What You Need for This Setup

The first thing you need is a cloud provider. Oracle Cloud offers a Free Tier version of their cloud services, which allows you to setup virtual machines for free. This is great for a home lab with lots of rich features that you can use to try new tools and techniques.

You'll also need an automation tool. I used Ansible because its doesn't have many requirements (you only need an SSH daemon and public key authentication to get things going). I also like it because it works equally well regardless of the cloud environment you are trying to provision.

In this tutorial we will use the Open Source version of this tool, as it is more than sufficient for our purposes.

What's included in the Ansible playbook

An Ansible playbook is nothing more than a set of instructions you define to execute tasks that will change the status of a host. These actions are carried out on an inventory of hosts you define.

Here, you are going to learn about the following:

• How to clean inventory sources by using the proper layout in your playbooks.

• How to provision two NGINX instances, with the request of their proper free SSL certificates using Certbot.

• How to set up the local Linux firewalls and add a Prometheus node_exporter agent and one scraper to collect that data.

• Concepts like variables, roles (with task inclusion), and conditional execution.

• Important techniques like task tagging, debug messages, and static validation with ansible-lint.

All the code can be found in this GitHub repository.

What You Should Know Before Trying This

Because we will cover several tasks here, you will probably need to be familiar with several things (I'll provide links as we go along):

• This is not an introductory course on Ansible but more of a "how all things fit together" with a more detailed, but not too complex, playbook.

• An OCI Cloud Free Tier account

• Privileged account, most likely SUDO

• Basic knowledge of TCP/IP and firewalls with firewalld

• How to use RPM and how to package applications (we will not do that here, but it helps to understand when an RPM is better than a complex task in Ansible)

What is not included here

OCI Cloud has a complete REST API to manage a lot of aspects of their cloud environment. Their setup page (specifically the SDK) is also very detailed.

You'll Probably Do Things Differently in Production.

Installing the OCI-Metrics-datasource instead of Prometheus agents on a virtual machine

You can go to this page to install it on your Grafana instance (Bare metal or Cloud). Also you need to setup your credentials and permissions as explained here.

This is probably the most efficient way to monitor your resources as you do not need to run agents on your virtual machines. But I will install instead a Prometheus node_exporter agent and scraper that will be visible from a Grafana Cloud instance.

An exposed Prometheus on the Internet endpoint is not a good idea

It is very clear, I'm exposing my Prometheus scraper to the Internet so Grafana cloud can reach it. On an Intranet with a private cloud and your local Grafana, this is not an issue – but here, a Prometheus agent pushing data to Grafana would be a better option.

Still, Grafana provides a list of public IP addresses that you can use to setup your allow list.

So the following will work:

Oracle Cloud Ingress Rules

But it is not the best. Instead, you want to restrict the specific IP addresses that can pull data from your exposed services. The prometheus exporter can be completely hidden from Grafana on port 9100. Instead we only need to expose the Prometheus scraper that listens on port 9000.

For this home lab, it is not a big deal having such services fully exposed. But if you have a server with sensitive data, you must restrict who can reach the service!

An alternative to the Prometheus endpoint is to push the data to Grafana by using a Grafana agent but I will not cover that option here.

Playbook Analysis

Ansible lets you have a single file with the playbook instructions, but eventually you will find that such a structure is difficult to maintain.

For my playbook I decided to keep the suggested structure:

tree -A 
.
├── inventory
│   └── cloud.yaml
├── oracle.yaml
├── roles
│   └── oracle
│       ├── files
│       │   ├── logrotate_prometheus-node-exporter
│       │   ├── prometheus-node-exporter
│       │   └── requirements_certboot.txt
│       ├── handlers
│       │   └── main.yaml
│       ├── meta
│       ├── tasks
│       │   ├── controller.yaml
│       │   ├── main.yaml
│       │   ├── metrics.yaml
│       │   └── nginx.yaml
│       ├── templates
│       │   ├── prometheus-node-exporter.service
│       │   ├── prometheus.service
│       │   └── prometheus.yaml
│       └── vars
│           └── main.yaml
└── site.yaml

Below is a brief description of how the content is organized:

1. You can have more than one site. You control that inside the [site.yaml](file:///home/josevnz/OracleCloudHomeLab/site.yaml) file.

2. The host list is inside the inventory directory. You can have more than one inventory file or scripts to generate the hostlist, or a combination of both.

3. The roles/oracle group the tasks. We only have one role called 'oracle' because that's the cloud provider I'm focusing on here.

4. Our playbook uses metadata in the form of variables, with each one defined on the 'vars' directory. That way we can customize the behaviour of the playbook in multiple places:

---
# Common variables for my Oracle Cloud environments
controller_host: XXXX.com
ssl_maintainer_email: YYYYYY@ZZZZ.com
architecture: arm64
prometheus_version: 2.38.0
prometheus_port: 9090
prometheus_node_exporter_nodes: "['X-server1:{{ node_exporter_port }}', 'Y-server2:{{ node_exporter_port }}' ]"
node_exporter_version: 1.4.0
node_exporter_port: 9100
internal_network: QQ.0.0.0/24

The roles/oracle files directory contains files that can be copied as is to the remote directory. The templates' directory is similar, but the files in there can be customized for each host by using the Jinja templating language.

# A template for the prometheus scraper configuration file
---
global:
    scrape_interval: 30s
    evaluation_interval: 30s
    scrape_timeout: 10s
    external_labels:
        monitor: 'oracle-cloud-metrics'

scrape_configs:
  - job_name: 'node-exporter'
    static_configs:
      - targets: {{ prometheus_node_exporter_nodes }}
    tls_config:
      insecure_skip_verify: true

The 'tasks' directory is where we store our tasks, that is the actions that will modify the server state. Note that Ansible will not execute tasks if it's not necessary. The idea is that you can re-run a playbook as many times as needed and the final state will be the same.

# Fragment of the nginx tasks file. See how we notify a handler to restart nginx after the SSL certificate is renewed.
---
- name: Copy requirements file
  ansible.builtin.copy:
    src: requirements_certboot.txt
    dest: /opt/requirements_certboot.txt
  tags: certbot_requirements

- name: Setup Certbot
  pip:
    requirements: /opt/requirements_certboot.txt
    virtualenv: /opt/certbot/
    virtualenv_site_packages: true
    virtualenv_command: /usr/bin/python3 -m venv
  tags: certbot_env

- name: Get SSL certificate
  command:
    argv:
      - /opt/certbot/bin/certbot
      - --nginx
      - --agree-tos
      - -m {{ ssl_maintainer_email }}
      - -d {{ inventory_hostname }}
      - --non-interactive
  notify:
    - Restart Nginx
  tags: certbot_install

There is one special directory called 'handlers'. There we define actions that must happen if a task changes the state of our host.

We now have a picture of how all the pieces work together, so let's talk about some specific details.

Firewall provisioning

With Ansible, you can replace a sequence of commands like this:

sudo firewall-cmd --permanent --zone=public --add-service=http
sudo firewall-cmd --permanent --zone=public --add-service=https
sudo firewall-cmd --reload

With a firewalld module:

---
- name: Enable HTTP at the Linux firewall
  firewalld:
    zone: public
    service: http
    permanent: true
    state: enabled
    immediate: yes
  notify:
    - Reload firewall
  tags: firewalld_https

- name: Enable HTTPS at the Linux firewall
  firewalld:
    zone: public
    service: https
    permanent: true
    state: enabled
    immediate: yes
  notify:
    - Reload firewall
  tags: firewalld_https

Common tasks have nice replacements

So instead of running SUDO with a privileged command:

sudo dnf install -y nginx
sudo systemctl enable nginx.service --now

You can have something like this:

# oracle.yaml file, which tells which roles to call, included from site.yaml
---
- hosts: oracle
  serial: 2
  remote_user: opc
  become: true
  become_user: root
  roles:
  - oracle
# NGINX task (roles/oracle/tasks/nginx.yaml)
- name: Ensure nginx is at the latest version
  dnf:
    name: nginx >= 1.14.1
    state: present
    update_cache: true
  tags: install_nginx
# And a handler that will restart NGINX after it gets modified (handlers/main.yaml)
---
- name: Restart Nginx
  ansible.builtin.service:
    name: nginx
    state: restarted
- name: Reload firewall
  ansible.builtin.systemd:
    name: firewalld.service
    state: reloaded

How to Run the Playbooks

Normally you don't wait to have the whole playbook written, but you run the pieces you need in the proper order. At some point you will have your whole playbook finished and ready to go.

Make sure the playbook behaves properly with --check before making any changes

The very first step is to check your playbook file for errors. For that you can use yamllint:

yamllint roles/oracle/tasks/main.yaml

But doing this for every yaml file in your playbook can be tedious an error-prone. As an alternative, you can run the playbook in a 'dry-run' mode, to see what will happen without actually making any changes:

Another way to gradually test a complex playbook is by executing a specific task by using a tag or group of tags. That way you can do controlled execution of your playbook:

Keep in mind that this will not execute any dependencies that you may have defined on you playbook, tough:

Use Ansible-lint when ansible-playbook --check is not good enough

Some errors are more subtle and will not get caught with ansible-playbook --check. To get a more complete check on your playbooks before minor issues become a headache you can use ansible-lint. So let's get it installed:

python3 -m venv ~/virtualenv/ansiblelint && . ~/virtualenv/ansiblelint/bin/activate
pip install --upgrade pip
pip install --upgrade wheel
pip install ansible-lint

Now we can check the playbook:

(ansiblelint) [josevnz@dmaf5 OracleCloudHomeLab]$ ansible-lint site.yaml 
WARNING  Overriding detected file kind 'yaml' with 'playbook' for given positional argument: site.yaml
WARNING  Listing 1 violation(s) that are fatal
syntax-check[specific]: couldn't resolve module/action 'firewalld'. This often indicates a misspelling, missing collection, or incorrect module path.
roles/oracle/tasks/nginx.yaml:2:3

Strange, firewalld is available on our Ansible installation. What else was installed by ansible-lint?

(ansiblelint) [josevnz@dmaf5 OracleCloudHomeLab]$ ansible --version
ansible [core 2.14.0]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/josevnz/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/josevnz/virtualenv/ansiblelint/lib64/python3.9/site-packages/ansible
  ansible collection location = /home/josevnz/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/josevnz/virtualenv/ansiblelint/bin/ansible
  python version = 3.9.9 (main, Nov 19 2021, 00:00:00) [GCC 10.3.1 20210422 (Red Hat 10.3.1-1)] (/home/josevnz/virtualenv/ansiblelint/bin/python3)
  jinja version = 3.1.2
  libyaml = True

Ansible-lint installed its own ansible [core], and firewalld is part of ansible.posix collection. We will use Ansible Galaxy to install it:

(ansiblelint) [josevnz@dmaf5 OracleCloudHomeLab]$ which ansible-galaxy
~/virtualenv/ansiblelint/bin/ansible-galaxy
(ansiblelint) [josevnz@dmaf5 OracleCloudHomeLab]$ ansible-galaxy collection install ansible.posix
Starting galaxy collection install process
Process install dependency map
Starting collection install process
Downloading https://galaxy.ansible.com/download/ansible-posix-1.4.0.tar.gz to /home/josevnz/.ansible/tmp/ansible-local-18099xpw_8usc/tmp8msc9uf5/ansible-posix-1.4.0-_f17f525
Installing 'ansible.posix:1.4.0' to '/home/josevnz/.ansible/collections/ansible_collections/ansible/posix'
ansible.posix:1.4.0 was installed successfully

Running it again:

(ansiblelint) [josevnz@dmaf5 OracleCloudHomeLab]$ ansible-lint site.yaml 
WARNING  Overriding detected file kind 'yaml' with 'playbook' for given positional argument: site.yaml
WARNING  Listing 50 violation(s) that are fatal
name[play]: All plays should be named. (warning)
oracle.yaml:2

fqcn[action-core]: Use FQCN for builtin module actions (service).
roles/oracle/handlers/main.yaml:2 Use `ansible.builtin.service` or `ansible.legacy.service` instead.

fqcn[action-core]: Use FQCN for builtin module actions (command).
roles/oracle/handlers/main.yaml:6 Use `ansible.builtin.command` or `ansible.legacy.command` instead.

Some warnings are pedantic ('Use FQCN for builtin module actions (command)') and others require attention (Commands should not change things if nothing needs doing.).

Ansible-lint found many smells on the playbook, there is one option to re-write the files and correct some of these errors automatically:

There are some guidelines you can follow to correct these issues. Below are some that can be directly applied to the warnings we got earlier:

Note that all the errors are easy to solve. Some commands decide on their own if they should make changes or not but have a hard time communicating back to Ansible:

- name: Get SSL certificate
  ansible.builtin.shell:
    argv:
      - /opt/certbot/bin/certbot
      - --nginx
      - --agree-tos
      - -m "{{ ssl_maintainer_email }}"
      - -d "{{ inventory_hostname }}"
      - --non-interactive
  notify:
    - Restart Nginx
  tags: certbot_install

In our case, certboot prints a message if the certificate is not yet due for renewal. If that output is missing then we trigger the Nginx restart (see defining changed):

- name: Get SSL certificate
  ansible.builtin.shell:
    argv:
      - /opt/certbot/bin/certbot
      - --nginx
      - --agree-tos
      - -m {{ ssl_maintainer_email }}
      - -d {{ inventory_hostname }}
      - --non-interactive
  register: certbot_output # Registers the certbot output.
  changed_when: 
    - '"Certificate not yet due for renewal" not in certbot_output.stdout'
  notify:
    - Restart Nginx
  tags: certbot_install

I do want to use shell, as I need to expand the variable for certbot, but ansible-lint is still not happy:

(ansiblelint) [josevnz@dmaf5 OracleCloudHomeLab]$ ansible-lint site.yaml
WARNING  Overriding detected file kind 'yaml' with 'playbook' for given positional argument: site.yaml
WARNING  Listing 1 violation(s) that are fatal
command-instead-of-shell: Use shell only when shell functionality is required.
roles/oracle/tasks/nginx.yaml:47 Task/Handler: Get SSL certificate

You can skip specific rules or tags by adding them to your configuration file:
# .config/ansible-lint.yml
warn_list:  # or 'skip_list' to silence them completely
  - command-instead-of-shell  # Use shell only when shell functionality is required.

                   Rule Violation Summary                    
 count tag                      profile rule associated tags 
     1 command-instead-of-shell basic   command-shell, idiom 

Failed after min profile: 1 failure(s), 0 warning(s) on 8 files.

Time to treat this error as a warning, as I know they are not issues, by creating a .config/ansible-lint.yml:

(ansiblelint) [josevnz@dmaf5 OracleCloudHomeLab]$ ansible-lint site.yaml
WARNING  Overriding detected file kind 'yaml' with 'playbook' for given positional argument: site.yaml
WARNING  Listing 1 violation(s) that are fatal
command-instead-of-shell: Use shell only when shell functionality is required. (warning)
roles/oracle/tasks/nginx.yaml:47 Task/Handler: Get SSL certificate


                        Rule Violation Summary                         
 count tag                      profile rule associated tags           
     1 command-instead-of-shell basic   command-shell, idiom (warning) 

Passed with min profile: 0 failure(s), 1 warning(s) on 8 files.

Much better now, the warning is not treated as an error.

Jinja best practices

If you plan to use variables and Jinja templates, make sure you quote them (example: "dest: /opt/prometheus-{{ prometheus_version }}.linux-{{ architecture }}.tar.gz")

Constrain where the playbook runs with --limit and --tags

Say that you are only interested in running your playbook on a certain host. In that case, you can also do that by using the --limit flag:

ansible-playbook --inventory inventory --limit fido.yourcompany.com --tags certbot_renew site.yaml

Here we did run only a task tagged certbot_renew on the host fido.yourcompany.com.

How to deal with a real issue

Let's make this interesting: say that I was eager to update one of my requirements for certboot, and I changed versions if pip to '22.3.1':

pip==22.3.1
wheel==0.38.4
certbot==1.32.0
certbot-nginx==1.32.0

When I run the playbook we have a failure:

This is an issue with the versions if specified on the requirements_certboot.txt file. When you install a Python library using a virtual environment you can specify versions like this:

pip==22.3.1 wheel==0.38.1 certbot==1.23.0 certbot-nginx==1.23.0

To fix the issue, we will revert the versions used on the file and then re-run the requirements file and Certbot installation task:

- name: Setup Certbot
  pip:
    requirements: /opt/requirements_certboot.txt
    virtualenv: /opt/certbot/
    virtualenv_site_packages: true
    virtualenv_command: /usr/bin/python3 -m venv
    state: forcereinstall
  tags: certbot_env

ansible-playbook --inventory inventory --tags certbot_env site.yaml

See it in action:

How to run the whole playbook

ansible-playbook --inventory inventory site.yaml

It is time to run the whole playbook:

Wrapping up

This tutorial only touches the surface of what you can do with Ansible. So below are a few more resources you should explore to learn more:

• Improving inventories: How to create dynamic inventory files in Ansible, How to write a Python script to create dynamic Ansible inventories, How to write an Ansible plugin to create inventory files

• Sometimes your playbooks will run slow, and you may need to Assess resource consumption with Ansible callback plugins.

• And there will be a time when deeper debugging is needed.

The SSL and TLS protocols are frequently attacked. And understanding past attacks can inform your knowledge as a defender and help you secure current systems. It can also help you predict the direction of future attacks.

So here's a summary of some of the most famous attacks targeting these protocols:

Browser Exploit Against SSL/TLS (BEAST):

BEAST (disclosed in 2011) allowed a man-in-the-middle attacker to discover encrypted information from an SSL/TLS session. It impacted SSL 3.0 and TLS 1.0.

This attack depended on the implementation of the block cipher used by TLS. The implementation used CBC, Cipher Block Chaining mode. This involves XORing each block of plaintext (except the first) with the previous block of ciphertext, then using the encryption algorithm on the block.

The first block is XORed with an IV (initialization vector). Much of the mode's security depends on the IV being truly random. But TLS 1.0 didn't randomly generate IVs – it just used the last block of ciphertext from the previous message. That meant that anyone able to snoop on the encrypted traffic had a copy of the IV.

An attacker who could snoop on the encrypted traffic could launch a chosen plaintext attack by guessing a block of data, XORing it with the (known) IV and the previous block of ciphertext, and injecting the created block into the session. This allowed the attacker to check if the entire block was correct.

Given this, the flaw was seen as largely theoretical until BEAST was released. BEAST found a way to shift the cipher block boundaries to isolate one byte of a message at a time until it had been guessed.

That, and the fact that HTTP messages are typically standardized so that an attacker would know where in the message sensitive information (like the session cookie) was transmitted, allowed an attacker to brute force compromise a session cookie.

While the attack was theoretically extremely interesting, and generated a lot of interest, it only works if the attacker can insert malicious code into a page and violate the same-origin policy. If the attacker had this much access to your system, they would also have a number of attacks they could attempt, many of which are far less complicated to execute.

Mitigation Measures for SSL/TLS Attacks:

1. (Safest) Only allow TLS 1.1 or 1.2 since they addressed the vulnerability. However, at the time, most websites and browsers didn't support TLS 1.1. or 1.2.

2. As TLS supported both a block cipher and a stream cipher, switch to the stream cipher (RC4). However, in 2013 it was demonstrated that RC4 was insecure, and in 2015 it was officially banned (by the Internet Engineering Task Force, or IETF).

3. Use a different block cipher mode. Unfortunately, TLS 1.0 didn't support any other modes.

4. Insert packets of length 0. Essentially, as a packet of length 0 would be padded to the block size, it becomes a packet of just padding for the sender, but is immediately discarded by the recipient. These blocks would be used as IVs for the next message, solving the problem of insecure IVs. This option was largely unused as it caused interoperability issues with some SSL stacks, notably including Internet Explorer 6.0. OpenSSL implemented this, but disabled it by default.

5. Practice defense in depth to prevent attackers from getting man-in-the-middle access to a victim network.

Sources/Further Information:

• How the BEAST Attack Works

• An Illustrated Guide to the BEAST Attack

• Original Write Up of BEAST

• A Comprehensive Study of BEAST, CRIME, TIME, BREACH, LUCKY 13 & RC4 Biases

Heartbleed Vulnerability:

Heartbleed (introduced in 2012/disclosed in April 2014) was a vulnerability in the heartbeat extension of the OpenSSL library (this is used to keep a connection opened).

This library is used largely for servers running Apache and NGINX (at the time of disclosure, roughly 17% of the internet's 'secure' (using SSL/TLS) websites were vulnerable), but impacted any server which depended on the vulnerable version of OpenSSL.

Essentially, the client sends a message to the server which contains the response it requests and the size of the response. The server would respond with the requested data in the size it was requested (see below for an explanation from xkcd).

The vulnerability was that the server would not check that the request was actually the same size as its stated size. If the user sent a requested response (for example 'bird') which was shorter than the requested length (500 letters), the server would 'pad' the response with data from its memory to meet the length requirement, potentially leaking sensitive memory information.

https://xkcd.com/1354/

This leaked data would be unencrypted and could contain anything – sensitive credentials, documents, and so on.

But in order to successfully exploit this bug, several things needed to happen: the site had to have implemented SSL, it needed to be running a vulnerable version of OpenSSL (the vulnerable versions were between 1.0.1 and 1.0.1f), the attacker needed to have had access to the environment between finding out the bug exists and it being patched, and there needed to have been something useful in memory at the time the attacker carried out the attack. This is relatively more difficult, though far from impossible.

Unfortunately, since the exploitation doesn't leave any abnormal traces in logs, it's hard to know if or how often this bug was actually exploited.

Also, this doesn't account for the danger of an attacker having previous pcaps of traffic and pulling the site's private key in an attack. This means they could have decrypted large amounts of sensitive data, causing a serious breach.

If intelligence agencies were able to pull off this type of attack, it's likely it would never be publicly released.

Mitigation Measures for Heartbleed:

• Upgrade OpenSSL to the latest version, patching the vulnerability (the vulnerable versions were between 1.0.1 and 1.0.1f).

Sources/Further Reading:

• 5 Years Later, Heartbleed Vulnerability Still Unpatched

• XKCD Heartbleed

• Everything you need to know about the Heartbleed Bug

Padding Oracle On Downgraded Legacy Encryption (POODLE):

POODLE (disclosed in September 2014) took advantage of a flaw in SSL3.0. In order to support legacy systems, some systems continued to offer support for SSL 3.0, even though it had been replaced by newer versions.

For the attack to be successful, the attacker needed to be able to control parts of the client side of the SSL connection and needed to have visibility of the resulting ciphertext (the most common way to have this access is to act as a man-in-the-middle).

This attack, while powerful, does require a separate attack to gain this access.

During a typical handshake negotiation, the client and server will work together to find a protocol which works for both to communicate. They'll start with the highest protocol they both offer and work down (so the client may offer TLS 1.2, and the server may respond with TLS 1.1).

But if the connection fails (either due to an attacker or a network connection problem), the client server will downgrade to the lowest protocol they offer, likely SSL 3.0. This is a 'feature' offered so that servers and clients at different stages of advancement can communicate.

SSL3.0 uses either RC4 (stream cipher) or a block cipher in CBC mode for encryption. RC4 was well-known, even at the time, for having a number of flaws, including that if the same secret (like a cookie) was sent repeatedly, more and more information about it would leak.

CBC mode requires that the message length is a multiple of the block size or that padding is used to fulfill the length condition. This means that padding is frequently used, and in the SSL 3.0 implementation it isn't verified that it hasn't changed in transit.

Essentially, while the message is hashed before sending and at the receiving end, and the re-compiled hash is compared to ensure message integrity, the padding is not included.

The only specification for padding is that the last byte needs to be the padding length. Therefore, an attacker can replace the padding and it will still be accepted, as long as they get the last digit correct.

If an attacker knows the location of the data they're trying to decrypt (for example an HTTP session cookie, which is typically sent in the same part of the block each time and is therefore easy to locate), they can copy it over into the final block, replacing the padding.

Then, the receiver will XOR the padded block with the previous block's ciphertext (which the attacker can see) and will only accept the data if the last byte matches the padding's length.

Essentially this attack makes brute-forcing SSL feasible. Whereas it would be practically impossible to brute force SSL without any information, this attack allows for recovering each byte after a maximum of 256 attempts per byte. That means that, in a few minutes, an attacker could compromise a session cookie or other sensitive information.

Mitigation Measures for POODLE:

• Disabling SSL 3.0 is the only complete mitigation of POODLE. However, some sites only supported SSL 3.0.

• An alternative is to use the TLS_FALLBACK_SCV cipher suite. This suite allows a server to fall back to earlier protocols, but rather than dropping immediately to SSL 3.0, the client can specify a preference. One problem with this suite is that it wasn't broadly supported when it was introduced (limited to largely Google services). Additionally, if the only option the server supports is SSL 3.0, POODLE attacks are still possible. However, it means that an attacker can't force a downgrade on a connection with a server which supports alternative protocols.

• Practice defense in depth to prevent attackers from getting man-in-the-middle access to a victim network. While the attack is dangerous, it requires man-in-the-middle access, making it much harder to exploit than the remotely exploitable Heartbleed.

Sources/Further Information:

• Everything you need to know about the POODLE bug

• This POODLE Bites: Exploiting the SSL 3.0 Fallback

• What is the POODLE Attack?

• CISA SSL 3.0 Protocol Vulnerability

Wrapping Up

Understanding past attacks on TLS and SSL can both inform your knowledge as a defender and help you secure your systems.

Often systems are out of date, or legacy applications may require use of outdated protocols. This means that even older attacks may be successful if defenses aren't applied appropriately.

In many cases, working in corporate security requires having enough information to make informed decisions and recommendations. If, for example, a legacy app requires using an outdated protocol, knowledge of attacks like POODLE and Heartbleed can help you make effective recommendations about how to secure that application, rather than making general recommendations like 'update to a newer protocol' (which might be impossible).

Typically, as a security analyst, you're trying to balance business needs with technical capabilities, and making recommendations based on the organization's risk level. That might mean saying that a legacy app shouldn't be run anymore, or it may mean making recommendations on how to secure the application as much as possible, given that it needs to use an insecure protocol.

Understanding past TLS/SSL attacks can also help you predict the direction of future attacks. Since Heartbleed and POODLE (in 2014), we've seen attacks like FREAK and Logjam in 2015 and Sweet32 in 2016. We'll likely continue to see additional attacks.

Understanding the fundamentals of how TLS and SSL work, and how they've been attacked in the past can help you predict or understand future attacks.

Cover photo: "cybersecurity" by Christian Colen

In that case, the browser has run a check on the SSL (secure socket layer) certificate and found a problem with it – the SSL could have expired or might not have been installed at all.

In some cases, the problem could be because of your browser and not the website. So, in this article, I will show you how to fix the “Your connection is not private” error on a Chrome browser.

What We'll Cover

• How to Fix the “Your connection is not private” Error on a Chrome Browser
  • Reload the Web Page
  • Clear Chrome’s Cache
  • Make Sure your Computer’s Date and Time are Correct
  • Disable your Antivirus and VPN
• Final Thoughts

How to Fix the “Your connection is not private” Error on a Chrome Browser

Reload the Web Page

The first thing I would advise you do is to reload the page.

Reloading the web page is the old trick everyone tries if there is a problem with that web page.

In addition, there are chances that SSL-related work is going on with the website, so if you wait a while and reload the page, the issue could disappear.

If reloading doesn’t fix the issue for you, proceed to other solutions in this article.

Clear Chrome’s Cache

The SSL data of the website in your Chrome browser cache might have expired. So if you clear the cache, the error may go away.

Follow the steps below to clear your Chrome browser cache:

Step 1: Click the 3 vertical dots on the top-right corner and select Settings:

Step 2: Click the “Privacy and Security” tab on the left sidebar and select “Clear browsing data”:

Step 3: Select Cache and Cookies, then click “Clear data”:

Make Sure your Computer’s Date and Time are Correct

If your computer clock is behind or ahead, your browser will show you a “Your connection is not secure” error.

These days, the error message has become more accurate in Chrome:

In this case, you should set your date and time to the correct one and make it automatic, so nothing readjusts it again:

Disable your Antivirus and VPN

Some Antivirus programs with SSL scanning features can make your browser show the “Your connection is not private” error if they detect any irregularity with the SSL certificate of a website.

In the same vein, a VPN (virtual private network) conceals your IP address and other information. The problem is that the privacy a VPN gives you could have a negative effect on some sites' SSL.

Due to this, you should consider disabling your antivirus and VPN programs, at least temporarily, to see if that fixes the error.

Final Thoughts

Other fixes that might get rid of the “Your connection is not private” error for you include:

• Trying to access the web page in incognito mode. In Chrome, you can open an incognito tab by pressing CTRL + SHIFT + N.
• Restarting your Router
• Restarting your computer
• Updating your OS

If all the fixes fail to work, the problem could be from the website. This means there is a problem with the website’s SSL certificate. So, try to contact the site admin.

If you find no admin to contact, you should stay off that website and make sure you don’t share sensitive information with the site.

If you’re a site admin and your users complain about this error, I wrote an article on how you can restore your site’s SSL.

This error is mostly caused by an expired or bad SSL certificate. It also occurs when the browser can’t verify the legitimacy of a website’s SSL certificate.

That error message could be a giant one like this:

The browser could also show you a signal in the address bar like this:

In this article, I will show you what an SSL certificate is. I will also show you how to fix SSL errors as a site owner and as a user.

What We'll Cover

• What is SSL and Why is it Used?
• How to Fix SSL Error as a Site Owner
  • Purchase an SSL Certificate
  • Make sure you Turn on SSL on your Website
  • If your website is hosted on Github pages…
  • If your website is hosted on Netlify…
  • If your website is a WordPress website…
  • Contact your Hosting Provider
• How to Fix SSL Error as a User
  • Make Sure your Date and Time are Correct
  • Clear Saved SSLs on your Computer
  • Clear your Browser’s Cache and Cookies
• Final Thoughts

  What is SSL and Why is it Used?

SSL stands for secure socket layer. It is the international standard security technology for keeping the sharing of information safe between a website and its users.

In the Chrome browser, when a website has a valid SSL, a locked padlock is shown in the address bar – indicating that any information the user shares with that website is encrypted.

How to Fix SSL Error as a Site Owner

If you are a site owner and your users are complaining that your website shows SSL errors, you can fix the issue with any of the methods explained below:

Purchase an SSL Certificate

If your website doesn’t have an SSL certificate installed, any modern browser your user is using will alert them your site is not secure.

In this case, you should purchase an SSL certificate from any of the providers out there.

By the way, you can buy an SSL certificate from companies that sell domain names. You can also buy an SSL from Sectigo or SSLs.

Make sure you Turn on SSL on your Website

If you’ve purchased and installed an SSL but your website still shows SSL errors, it could happen because you unknowingly didn’t turn on SSL.

If your website is hosted on Github pages…

Step 1: Navigate to your site repo and click Settings:

Step 2: Click on Pages on the left sidebar:

Step 3: Tick “Enforce HTTPS”:

This is required for sites using GitHub’s default domain (example.github.io).

Even if you’re using a custom domain, make sure that box is ticked.

If your website is hosted on Netlify…

Step 1: Click on the site with an SSL error:

Step 2: Click on “Site Settings”:

Step 3: Click “Domain Management”, and then HTTPS on the left sidebar. Make sure there’s the message “Your site has HTTPS enabled”.

If your website is a WordPress website…

If you have a WordPress website with an SSL error, install the Force SSL plugin on your website

Contact your Hosting Provider

If every method discussed above fails to work for you, then you should contact the customer service of your hosting provider.

How to Fix SSL Error as a User

If you visit a website and you’re getting any SSL-related errors, there are some things you can do as a user. That’s because the problem is not always caused by the website – as long as an SSL certificate is installed on the website.

Make Sure your Date and Time are Correct

If your computer’s date and time are ahead or behind, the browser might show an SSL-related error.

This is because SSLs have expiration dates. So, when your computer’s date and time are behind or ahead, the check your browser runs to see if the SSL certificate of that website is valid will fail.

In this case, the browser will suggest that you change your date and time.

Clear Saved SSLs on your Computer

Clearing the SSL certificates stored by your computer can fix the issue for you.

When next you visit that website with the SSL error, your browser will run another check to revalidate the SSL installed on that website.

To clear the SSLs, hit the Windows button on your keyboard, search for “Internet Options” and click the Internet Options search result:

Switch to the content tab and click “Clear SSL state”:

Clear your Browser’s Cache and Cookies

The SSL info of a website in your browser’s cache and cookies might have expired, so if you clear both records, it could fix the issue for you.

To clear Chrome’s cache and cookies, click the 3 vertical dots on the top-right corner and select Settings:

Switch to the Privacy and Security tab on the left sidebar and select “Clear browsing data”:

Select Cache and Cookies, then click “Clear data”:

If you’re using another browser that is not Chrome, clear the browser’s cache and cookies.

Final Thoughts

As a web administrator or owner, it's very important to make sure SSL is installed and properly working on your website. Otherwise, it might not just affect your website alone, but also your business.

If you’re a user, too, make sure any website you’re visiting shows the padlock icon on the address bar. If that’s not the case, make sure you don’t share sensitive information like card details and passwords with the website.

Thank you for reading.

Recently, I came across a SaaS product that validates SSL certificates for your website. I wanted to try writing the same thing in Go – and it turns out that it's pretty straightforward (just 17 lines of code).

You will need to perform three major checks on your website.

1. First, check to see whether your site has an SSL certificate or not. You'll also need to know if your website is using a self-signed SSL certificate which is not considered a valid certificate (it needs to be signed by a certificate authority).
2. Second, to see if the SSL certificate has the correct hostname.
3. And third, you need to know the expiration date of the server certificate.

Prerequisites:

• You should have go setup on your computer.

Step 1: Check if your website has an SSL certificate

First, we will try to check if the website has an SSL certificate or not.

To do this we need to establish a TLS connection with the website. If that succeeds it means the website has a valid TLS certificate.

To establish a TLS connection we can use the Go crypto/tls package. We'll use the Dial method to connect to the website, like this:

package main

import (
    "crypto/tls"
)

func main() {
    conn, err := tls.Dial("tcp", "example.com:80", nil)
    if err != nil {
        panic("Server doesn't support SSL certificate err: " + err.Error())
    }
}

We will try to run our test on example.com. When you run the above code, you should get the following error:

$ go run main.go
panic: Server doesn't support SSL certificate err: tls: first record does not look like a TLS handshake

goroutine 1 [running]:
main.main()
        /Users/umesh/personal/spike/main.go:99 +0x2ca
exit status 2

Basically it tried to establish the connection but failed. The Dial method succeeds only when the website has a valid certificate (it will fail if the certificate is self-signed).

Now try the same code on a website that has SSL enabled. I am using my own website's URL as an example:

package main

import (
    "crypto/tls"
)

func main() {
    conn, err := tls.Dial("tcp", "blog.umesh.wtf:443", nil)
    if err != nil {
        panic("Server doesn't support SSL certificate err: " + err.Error())
    }
}

This time, we should be able to successfully establish the connection without the code panic.

In production you should not use panic, but rather should handle the error gracefully.

Step 2: Check whether the SSL certificate and website hostname match

To verify the hostname we will need to call VerifyHostname on the conn return by Dial. This method tries to match the common name or subject alt name specified in the certificate with the domain passed as a parameter.

package main

import (
    "crypto/tls"
)

func main() {
    conn, err := tls.Dial("tcp", "blog.umesh.wtf:443", nil)
    if err != nil {
        panic("Server doesn't support SSL certificate err: " + err.Error())
    }

    err = conn.VerifyHostname("blog.umesh.wtf")
    if err != nil {
        panic("Hostname doesn't match with certificate: " + err.Error())
    }
}

This will successfully execute without any error as the certificate's common name and hostname are the same.

Step 3: Verify the expiration date of the server's SSL certificate

We can get the certificate chain using conn.ConnectionState().PeerCertificates. We can then use this certificate to get the expiration date of the server certificate.

We will use the first certificate from the list of certificates and try to get the expiration date using the field NotAfter.

package main

import (
    "crypto/tls"
    "fmt"
    "time"
)

func main() {
    conn, err := tls.Dial("tcp", "blog.umesh.wtf:443", nil)
    if err != nil {
        panic("Server doesn't support SSL certificate err: " + err.Error())
    }

    err = conn.VerifyHostname("blog.umesh.wtf")
    if err != nil {
        panic("Hostname doesn't match with certificate: " + err.Error())
    }
    expiry := conn.ConnectionState().PeerCertificates[0].NotAfter
    fmt.Printf("Issuer: %s\nExpiry: %v\n", conn.ConnectionState().PeerCertificates[0].Issuer, expiry.Format(time.RFC850))
}

The output should contain the certificate's expiration date and Issuer Name.

$ go run main.go
Issuer: CN=Let's Encrypt Authority X3, O=Let's Encrypt, C=US
Expiry: Wednesday, 16-Dec-20 16:20:00 UTC

Now we have successfully validated the site's certificate.

Conclusion

You can also get detailed information like root CA, the date the certificate was issued, and all the chained certificates.

This tool was just written using the core libraries of Go (and no external libraries). Do let me know what do you think about it!

If you liked this article, you can head over to my personal blog to see more stuff I've written.

I regularly write about programming and software, so do subscribe to my newsletter and get the latest posts from me delivered straight into your inbox. You can also get in touch with me on Twitter.

In this article, I will walk you through how the world wide web works at a fundamental level.

The core technology is HTTP - Hypertext Transfer Protocol. It's the communication protocol you use when you browse the web.

At a fundamental level, when you visit a website, your browser makes an HTTP request to a server. Then that server responds with a resource (an image, video, or the HTML of a web page) - which your browser then displays for you.

This is HTTP's message-based model. Every HTTP interaction includes a request and a response.

By its nature, HTTP is stateless.

Stateless means that all requests are separate from each other. So each request from your browser must contain enough information on its own for the server to fulfill the request. That also means that each transaction of the message based model of HTTP is processed separately from the others.

URLs

The URL (Uniform Resource Locator) is probably the most known concept of the Web. It is also one of most important and useful concepts. A URL is a web address used to identify resources on the Web.

The idea of the web is structured around resources. From its beginnings the Web was the platform for sharing text/HTML files, documents, images etc, and as such it can be considered a collection of resources.

Example of an URL

Protocol — Most often they are HTTP (or HTTPS for a secure version of HTTP).

Other notable protocols are:

• File Transfer Protocol (FTP) — is a standard protocol used for transferring files between a client and a server over a network.
• Simple Mail Transfer Protocol (SMTP) is a standard for email transmission.

Domain — Name that is used to identify one or more IP addresses where the resource is located.

Path —Specifies the resource location on the server. It uses the same logic as a resource location used on the device where you are reading this article (i.e. /search/cars/VWBeetle.pdf or C:/my cars/VWBeetle.pdf).

Parameters — Additional data used to identify or filter the resource on the server.

Note: When searching for articles and more information about HTTP, you may encounter the term URI (or uniform resource identifier). URI is sometimes being used instead of URL but mostly in formal specifications and by people who want to show off. :)

HTTP Requests

In HTTP, every request must have an URL address. Additionally, the request needs a method. The four main HTTP methods are:

• GET
• PUT
• POST
• DELETE

I will explain these methods, and more, in the HTTP Methods section of this article.

And these methods directly correspond to actions:

• read
• update
• create
• delete

All HTTP messages have one or more headers, followed by an optional message body. The body contains the data that will be sent with the request or the data received with the response.

The first part of every HTTP request holds three items:

Example:

• GET /adds/search-result?item=vw+beetle HTTP/1.1

When a URL contains a “?” sign, it means it contains a query. That means it sends parameters of the requested resource.

1. The first part is a method which tells which HTTP method is used. Most commonly used is the GET method. GET method retrieves a resource from the web server and since GET doesn’t have a message body nothing after the header is needed.
2. The second part is a requested URL.
3. The third part is a HTTP version being used. Version 1.1. is the most common version for most browsers, however, version 2.0 is taking over.

There are also some other interesting things in an HTTP request:

Referer header — tells the URL from where the request has originated.

User-Agent header — additional information about the browser being used to generate the request.

Host header — uniquely identifies a host name; it is necessary when multiple web pages are hosted on the same server.

Cookie header — submits additional parameters to the client.

HTTP Responses

Just like in HTTP requests, HTTP responses also consist of three items:

Example:

HTTP/1.1 200 OK

1. The first part is the HTTP version being used.
2. The second part is the numeric code of the result for the request.
3. The third part is a textual description of the second part.

There are some other interesting things in an HTTP response:

Server header — information about which web server software is being used.

Set-Cookie header — issues the cookie to the browser.

Message body — it is common for an HTTP response to hold a message body.

Content-Length header — tells the size of the message body in bytes.

HTTP Methods

The most common methods are GET and POST. But there are a few others, too.

GET —  You use this method to request data from a specified resource where data is not modified it in any way. GET requests do not change the state of resource.

POST — You use this method to send data to a server to create a resource.

PUT — You use this method to update the existing resource on a server by using the content in the body of the request. Think of this as a way to "edit" something.

HEAD —  You use this method the same way you use GET, but with the distinction that the return of a HEAD method should not contain body in the response. But the return will contain same headers as if GET was used. You use the HEAD method to check whether the resource is present prior of making a GET request.

TRACE — You use this method for diagnostic purposes. The response will contain in its body the exact content of the request message.

OPTIONS — You use this method to describe the communication options (HTTP methods) that are available for the target resource.

PATCH —  You use this method to apply partial modifications to a resource.

DELETE —You use this method to delete the specified resource.

REST

Representational state transfer (REST) is an architecture style where requests and responses contain representations of the current state of the systems resource.

“Regular” way:

• http://carapp.com/search?make=wv&model=beetle

REST-style:

• http://carapp.com/search/vw/beetle

You can learn more about REST here if you're curious.

HTTP Headers

There are three main components that make up the request/response structure. These include:

• First line
• Headers
• Body/Content

We already talked about the first line in HTTP requests and responses, and body function was mentioned too. Now we'll talk about HTTP headers.

The HTTP headers are added after the first line and are defined as name:value pairs separated by a colon. HTTP headers are used to send additional parameters along with the request or response.

As I already said, the body of the message includes the data to be sent with the request or the data received along with the response.

There are different types of headers that are grouped based on their usage into 4 broad categories:

• General header — Headers that can be used in both requests and response messages and that are independent of the data being exchanged.
• Request header — These headers define parameters for the data requested or parameters that give important information about the client making the request.
• Response header — These headers contain information about the incoming response.
• Entity header — The entity headers describe the content that makes up the body of the message.

Types of headers

HTTP status codes

Browsing the web, you may have encountered "404 error: not found" pages or "500 errors: server is not responding" pages.

These are HTTP status codes.

Every HTTP response message must contain an HTTP status code in its first line, telling us the result of the request.

There are five groups of status codes which are grouped by the first digit:

• 1xx — Informational.
• 2xx — The request was successful.
• 3xx — The client is redirected to a different resource.
• 4xx — The request contains an error of some kind.
• 5xx — The server encountered an error fulfilling the request.

Here's a full list of HTTP Status Response Codes and their explanation.

HTTPS (Hypertext Transfer Protocol Secure)

The secure version of HTTP protocol is HyperText Transfer Protocol Secure (HTTPS). HTTPS provides encrypted communication between a browser (client) and the website (server).

In HTTPS, the communication protocol is encrypted using Transport Layer Security (TLS) or Secure Sockets Layer (SSL).

The protocol is therefore also often called HTTP over TLS, or HTTP over SSL.

Both the TLS and SSL protocols use an asymmetric encryption system. Asymmetric encryption systems use a public key (encryption key) and a private key (decryption keys) to encrypt a message.

Anyone can use the public key to encrypt a message. However, private keys are secret, and that means that only the intended receiver can decrypt the message.

Example of asymmetric encryption system

SSL/TLS handshake

When you request a HTTPS connection to a website, the website sends its SSL certificate to your browser. That process where your browser and website initiate communication is called the “SSL/TLS handshake.”

The SSL/TLS handshake involves a series of steps where browser and website validate each other and start communication through the SSL/TLS tunnel.

As you probably noticed, when a trusted secure tunnel is used during in a HTTPS connection, the green padlock icon is displayed in the browsers address bar.

Example of one of my secure pages

Benefits of HTTPS

The major benefits of a HTTPS are:

• Customer information, like credit card numbers and other sensitive information, is encrypted and cannot be intercepted.
• Visitors can verify you are a registered business and that you own the domain.
• Customers know they are not suppose to visit sites without HTTPS, and therefore, they are more likely to trust and complete purchases from sites that use HTTPS.

Thank you for reading! Check out more articles like this on my freeCodeCamp profile. And check out other fun stuff I build on my GitHub page.

At that point, I had only viewed my portfolio on localhost by viewing the files on my local computer. I didn’t know anything about how to host a website online.

Learning how to host my first website wasn’t easy but it was a great learning experience. If you're an aspiring web developer or just interested in launching your own static website, then I hope you will find this guide useful.

Who is this guide for?

This guide is aimed at complete beginners who want to host a static website (a site with fixed content). I will provide a how-to-guide for the following:

1. How to buy a domain.

2. How to configure your domain for an external hosting provider.

3. How to host your website with Amazon Web Services (AWS).

4. How to make your website secure (SSL certification) using Amazon Certification Manager.

There might be some terminology that’s new to you. Please go ahead and look up terms that are not familiar. I’ve put in useful links and explanations where I thought appropriate.

What is a domain name and DNS (Domain Name System)?

A domain name is your website address. For example thecodinghamster.com. But for a computer, a domain name is actually a series of numbers (an IP address). An IP address looks like this: 123.321.0.1

It’s not easy for us to remember a long string of numbers. So your computer refers to a DNS to translate a text based website address into an IP address which it can then understand. A DNS is like a directory.

I watched this great video which explains domain name, DNS and how this works in under five minutes. Please watch the first five minutes of the video if you're interested:

Where can you buy your domain name?

You can buy a domain name from a domain name registrar. Prices start from a few dollars. Your domain name is unique. Each domain name registrar offers different levels of services/support. But you can register your domain with any registrar.

What is a hosting provider?

“An Internet hosting service is a service that runs Internet servers, allowing organizations and individuals to serve content to the Internet. There are various levels of service and various kinds of services offered.”

When I was looking for a hosting provider for my website, I explored different options. The prices ranged from £2.00 — £5.00 per month with various storage options from 0.5GB to 10GB. The prices seemed reasonable but all I wanted to do was host a static website. It had a few images, HTML, CSS and JavaScript files. No dynamic content.

Why AWS?

After some further research, I found AWS. AWS offers a free tier option. Essentially, you get lots of free products. Some of them expire after 12 months and others are free in perpetuity. The only cost that you will incur for hosting a static website is the cost of setting up a hosted zone. This cost $0.50 per month. So I went with AWS and set up my account.

The great thing about AWS is the price and it’s a reliable hosting provider. But one thing to bear in mind is that you’re reliant on their documentation. As I started reading about what services AWS offered, it quickly became confusing! I used the official AWS guide provided for setting up static websites. But I found myself getting lost with clicking on a link to another link and so forth. I started researching other guides to make up for knowledge gaps.

I found this this excellent guide by Victoria Drake.

I followed Victoria Drake’s guide alongside the AWS one and managed to muddle through. But there were still a few things that were not explained which I hope to flesh out.

Before we proceed, here is your to do list:

• Do some research on domain registrars and buy your domain name.

• Sign up for a free account with AWS.

• Have both the documentation from AWS and Victoria Drake’s guide open. Use my guide to guide you through the documentation (hope that makes sense!).

Here we go!

AWS: Create a Hosted Zone on Route 53.

Route 53 is where all your DNS requests are handled.

The first thing you must set up is your hosted zone with Route 53. This is really easy if you bought your domain through AWS. A hosted zone is created automatically once you’ve purchase it. If you’ve done this then just skip to the next section (Set up your S3 Buckets).

However, if you were like me and bought your domain name via another registrar then please do the following.

This next part is how to create a Hosted Zone on Route 53 if you haven’t bought your domain name from AWS:

1. Go to Route 53 in your console and click on “Create Hosted Zone”. Fill in your domain address, comment is optional and choose a “Public Hosted Zone”. Click on “Create”.

2. Once your hosted zone is created, you need your NS (Name Servers) records:

3. Go to your domain name registrar and login. Depending on your registrar, you should find a section in your settings called "Nameservers" which you can edit. You need to copy across the AWS NS records and change the existing NS records in your domain settings.

Please note, don't copy the full stop/period at the end of the NS record. For example, is should be “ns-63.awsdns-07.com”, not “ns-63.awsdns-07.com.”

It will take up to 24 hours to propagate.

Set up your S3 Buckets

In the meantime you can set up your S3 Buckets. The S3 Bucket is the storage for your files such as your index.html.

You must configure two buckets for your website: 1) yourdomainname.com and 2) www.yourdomainname.com.

The first bucket is your main bucket where you will upload all your documents, such as your index.html. The second bucket redirects to the first bucket. To set up your S3 Buckets, please follow the AWS documentation on how to set up your S3 bucket ( 2: Create and Configure Buckets and Upload Data).

In addition to the documentation, there are a few things to note:

• In section 2.1 (part 2): click on the link How Do I Create an S3 Bucket? This is a step by step guide and explain all the settings you need to choose.

• In section 2.1 (part 3): you don’t have to upload your website files yet. You can add a test index.html in the meantime.

Take note of your endpoint. You can find this in your S3 bucket > “Properties” tab > “Static Web Hosting” box. It should look something like this: http://yourdomainname.com.s3-website.eu-west-2.amazonaws.com

Add the Alias/“A” records in Route 53

Finally go back to Route 53 and open your hosted zone to set up your Alias records. You can follow the documentation on “Step 3: Add Alias Records for example.com and www.example.com”. It is quite straightforward.

Once the NS settings have propagated, your site is live! You’ll be able to visit your site at the domain address e.g. yourdomainname.com

However, please note that it won’t be secure and you’ll see a http:// prefix in the address bar. I’ll get to that in the next section.

How to make your website secure and what is a SSL certificate?

It’s very important to make your website secure and to do this you’ll need to get an SSL certificate. SSL stands for Secure Sockets Layer and it uses encryption to securely transfer data between a user and site. Google will also give a rankings boost for websites with HTTPS.

If you secure the website with an SSL certificate, you’ll see https:// and a padlock symbol in your address bar.

There are different types of SSL certificates: Extended Validation Certificate and a Domain validated certificate. For a personal website or blog, only a domain validated certificate is required. You also won’t see the name of your company on the left of the bar like the example above. You only get this if you have an Extended Validation Certificate which is more for large companies/enterprises.

How much does it cost for an SSL certificate?

I’ve seen a range of ways of getting an SSL certificate. You can pay a premium for a service that will do this on your behalf or you can do it for free with Let’s Encrypt. Let’s Encrypt is an official Certificate Authority (CA). But you have to renew your certificate every three months and the process is quite complicated.

I didn’t want to pay a premium or want the hassle of renewing every three months. Conveniently, AWS can issue SSL certificates for a very small fee. You pay $0.75 for each certificate issued and it lasts for one year.

If you choose not to go with AWS, make sure you do your research and choose a trusted CA!

How do you get an SSL certificate with AWS?

Log into your AWS console and navigate to the AWS Certificate Manager (ACM).

Make sure you change the region from the default (Ohio) to N. Virginia. This is not explicit in the guides and only the N. Virginia region can issue certificates. I learned the hard way and wasted a lot of time!

Then click on “Get Started” under “Provision Certificates”.

Follow the documentation with AWS (“Requesting a public certificate using the console”) and use Victoria Drake’s guide (under “SSL certificate”).

In addition to the guides there are few things that weren’t fully explained:

• You’ll need to validate your domain ownership by email or directly with DNS. I would suggest to always verify ownership by DNS validation.

• Once you’ve requested your certificate you’ll get something like this (except the status will be pending). Click on the “Export DNS Configuration file”:

It’s an Excel spreadsheet which will contain something like this:

• You’ll need to add these records to your DNS settings with your registrar. Log in and go to DNS settings. The interface varies with different registrars but you’re looking for your Host records under your DNS settings.

• Click on “Add Record” > record type is CNAME:

You need to add two records: 1) Hostname should be “@” and Target Name should be the Record value from the DNS configuration file.

2. Hostname should be * (asterisk) and Target Name should be the Record value from the DNS configuration file.

If you want more information about the CNAME and record types, I found this helpful article.

It's quite long, but I've pulled out the useful part:

“Note: Hostname refers to the prefix before the domain name. To create a Blank record, use an @ in the Hostname field. This represents an empty prefix (so the name exactly matches the domain name; for example divapirate.com). The @ hostname is also referred to as the the root of the domain. An * (asterisk) in the Hostname is a wildcard, and represents any prefix. For example, creating a record for *.divapirate.com will point .divapirate.com at the IP address provided.”

You just need to wait for the verification. For me, this took about an hour.

How do you add your SSL certificate?

With AWS you can add SSL certificate to your website through setting up CloudFront. CloudFront is great for speeding up your website. I used the AWS documentation and Victoria Drake’s guide (look out for her useful tips).

Please note, when you create your CloudFront distribution, there’s a drop down menu to add your SSL certificate. If you have been issued with an SSL certificate already, it will be pre-populated in the drop down menu.

Your state in your CloudFront dashboard should changed to “Enabled”. This isn’t instant and takes a little time.

Almost there…

Finally, you need to get your Domain Name from the CloudFront distribution. It should be something like this dsfdser83543.cloudfront.net.

Go back to Route 53 > hosted zone > change both Alias records (Alias Target) to the CloudFront Domain Name.

Voila! You have hosted your first secure static website with AWS.

Hope you found this useful. If you have any questions or just want to say hello, find me on Twitter @PhoebeVF

A big thank you to Victoria Drake for her guide. For a more advanced tutorial on this topic, please check out Victoria's article: "Hosting your static site with AWS S3, Route 53, and CloudFront".

Illustrations courtesy of https://undraw.co

What is SSL?

SSL (Secure Sockets Layer) is a standard security protocol for establishing encrypted links between a web server and a browser in an online communication.

The usage of SSL technology ensures that all data transmitted between the web server and browser remains encrypted.

An SSL certificate is necessary to create SSL connection. You would need to give all details about the identity of your website and your company as and when you choose to activate SSL on your web server. Following this, two cryptographic keys are created — a Private Key and a Public Key.

Learn More: Why SSL is Critical?

In order to force your web traffic to use HTTPS, edit the codes in the .htaccess file.

Before we move onto redirecting HTTP to HTTPS, here’s how you can edit .htaccess file. If you already know skip to Redirection steps.

Editing .htaccess File

There are instructions/directives in the .htaccess file that tell the server how to act in certain scenarios and directly affects how your website functions. Common directives in .htaccess file:

• Redirects

• Rewriting URLs

Ways to edit an .htaccess file:

1. Edit the file on your computer and upload it to the server using FTP.

2. Use “Edit” mode in FTP program that allows you to edit a file remotely.

3. Use a text editor and SSH to edit the file.

4. Use the File Manager in cPanel to edit the file.

Editing .htaccess in cPanel File Manager

Note: Backup your website in case something goes wrong.

1. Login to cPanel

2. Files > File Manager > Document Root for:

3. Now select the domain name you want to access

4. Check “Show Hidden Files (dotfiles)”

5. Click “Go”

6. After a new tab or window opens, look for the .htaccess file.

7. Right click on the .htaccess file and click on “Code Edit” on the menu.

8. A dialogue box may pop up asking about encoding. Click “Edit” button to continue.

9. Edit the file

10. “Save Changes” when done.

11. Test your website to make sure it is done correctly. In case, there is an error, restore to the previous version and try again.

12. Once you are done, click “Close” to close the window.

Redirecting HTTP to HTTPS

1. Redirect All Web Traffic

If you have existing code in your .htaccess, add the following:

RewriteEngine On
RewriteCond %{SERVER_PORT} 80
RewriteRule ^(.*)$ https://www.yourdomain.com/$1 [R,L]

2. Redirect Only a Specific Domain

For redirecting a specific domain to use HTTPS, add the following:

RewriteEngine On
RewriteCond %{HTTP_HOST} ^yourdomain\.com [NC]
RewriteCond %{SERVER_PORT} 80
RewriteRule ^(.*)$ https://www.yourdomain.com/$1 [R,L]

3. Redirect Only a Specific Folder

Redirecting to HTTPS on a specific folder, add the following:

RewriteEngine On
RewriteCond %{SERVER_PORT} 80
RewriteCond %{REQUEST_URI} folder
RewriteRule ^(.*)$ https://www.yourdomain.com/folder/$1 [R,L]

Note: Replace “yourdomain” with your actual domain name wherever required. Also, in case of the folder, replace /folder with the actual folder name.

Think it was helpful? Share this article to help others come on HTTPS.

Adding an SSL certificate and custom Namecheap domain to a GitLab Pages site can be a bit more challenging than it seems.

Crucial pieces of the setup information live in sometimes dense documentation across different sites. It can be hard to tell if you’ve set things up correctly given that you have to wait hours to confirm your changes have propagated.

Even when you know something is wrong, you can’t always tell what. This makes debugging the problem frustrating and challenging to fix.

This guide aims to make the process a bit more straightforward and less frustrating. It assumes that you’ve:

• Already set up your project on GitLab Pages and are able to access it by entering <your-username>.gitlab.io/<your-project-name> in your browser
• Have purchased a custom domain name along with an SSL certificate through Namecheap

Step 1: Activate the SSL certificate

In Namecheap, go to the ‘Product List’ > ‘SSL Certificates’ page. You should see a list of SSL certificates that you have purchased, but have not yet activated. Click ‘Activate’ on the SSL certificate that you wish to activate for your site.

Step 2: Generate the SSL certificate request

You should have been brought to a page that looks like the following:

In order to generate a CSR, you’ll need to run the following command in your terminal: openssl req -new -newkey rsa:2048 -nodes -keyout <your-domain-name>.key -out <your-domain-name>.csr.

A private key will be generated as a result of that command. DO NOT LOSE THIS KEY. You will need it later on when you go to install your certificate on GitLab. Should you lose it, you will have to submit another CSR request.

You can read the nitty-gritty details here about generating a CSR if you wish, but the TL;DR is:

• It’s strongly encouraged that you fill out all the required fields. Your CSR could be rejected during activation of you do not. If you are filling this CSR out for a personal or hobby site, you can enter NA for the ‘Organization’ and ‘Organization Unit’ fields.
• If the certificate is being issued for a specific subdomain, you need to specify the subdomain in the ‘Common Name’ field. Example: subdomain.ssl-certificate-host.com
• If the certificate is meant to be a wildcard certificate, the domain should start with an asterisk. Example: *.ssl-certificate-host.com

For the purposes of this guide, the assumption will be made that you are getting the certificate for something like <example-domain&gt;.com .

Once you’ve run the command, you should have a .csr and .key file in your working directory. Open the .csr file, and copy the contents in it. It should have the header ----- BEGIN CERTIFICATE REQUEST -----.

Paste the contents of the file into the Enter CSR field. The page will automatically fill out the domain field on the form based on the information in the CSR.

Once you click ‘Next’, you should see the following page:

Check that the information is correct, and then click ‘Next’ again to go to the ‘Confirm you own the domain’ step.

Step 3: Confirm you own the domain

There are a few different options that are available to you in order to do this:

• Email
• HTTP-based
• DNS-based

I personally have had issues validating through email, so for the purposes of this guide, select ‘DNS-based’. This requires you to set up a CNAME value in your domain’s DNS settings, which we will cover later on in this guide.

For now, click ‘Next’ after selecting ‘DNS-based’, but if you change your mind about this form of validation later on, it’s possible to change it.

Step 4: Specify who will receive the SSL file

Confirm that the email in the field is correct. This is the email that will receive the certificate once it’s been activated.

Step 5: Review and Submit

Confirm the information shown is correct, and then click ‘Submit’.

Step 6: Set up the CNAME record for validating ownership of the domain

Once you submit the form, you will be redirected to a page showing the SSL certificate details with a helpful notification window that looks like the following:

Click on the link for the DNS-based DCV method. You’ll be brought to a page that shows information that you entered earlier, such as:

• The domain name
• The type of web server that will have the certificate installed (should be Apache, Nginx, cPanel, or other)
• DCV Methods In Use

Access the dropdown options for the ‘Edit Methods’ button to the right of ‘DCV Methods in Use’ in order to access and click the ‘Get Record’ option.

A popover will appear showing the CNAME record you need to set up in order to confirm ownership of the domain. Copy these values to an empty text file as you’ll need to go to the ‘Advanced DNS’ page for your domain. This is accessible through ‘Dashboard’ or ‘Domain List’ > ‘Manage’ (besides your domain in the list) > ‘Advanced DNS’.

Under the ‘Host Records’ section:

• Click ‘Add New Record’
• Select ‘CNAME Record’.
• Paste the values that you copied earlier from the ‘Get Record’ popover into the corresponding fields.

Before you save those values though, there’s a bit of a ‘gotcha’.

As Namecheap points out in their documentation, they “add the domain name automatically to the values submitted during record creation”. This means that the domain name that appears in the ‘host’ value is a duplicated value. Remove <your-custom-domain&gt;.com at the end of the ‘host’ value and you’ll be good to go.

After you save that record, it’ll take a bit of time before the certificate is issued. Once you receive the certificate in your email, proceed to step 8. If you haven’t already though, let’s set up the additional records needed in order to send people to <your-username>.gitlab.io/<your-project> when they enter <your-custom-domain>.com.

Step 7: Set up your host records in Namecheap

As outlined in GitLab’s docs, you’ll also need to prove on GitLab’s end of things that you own the custom domain that you want to serve your GitLab Pages site on.

As mentioned earlier, this guide assumes that you are just looking to use example.com (or www.example.com), so you’ll want to add the following host records:

• Type A Record, Host @, Value 35.185.44.232 (this is the current GitLab Pages IP at the time of writing)
• Type CNAME Record, Host www , Value example.com (this ensures that people who enter the 'www’ subdomain (i.e: www.example.com) still reach your site)
• Note: You won’t be able to enter this one until you’ve added the domain through the ‘New Pages Domain’ flow outlined in Step 8. Type TXT Record, Host @ , Value gitlab-pages-verification-code=11112222aaaabbbb

Step 8: Install the certificate in GitLab

Head on over to the ‘Pages’ page of your GitLab project that you’re trying to set up (under ‘Settings’ > ‘Pages’ in the sidebar).

To add your custom domain that GitLab serves your Pages site on, click on the ‘New Domain’ button on the top right. You should see something like the following:

Enter your custom domain (example.com) in the domain field, and then the next part is where it gets interesting.

If you try just to enter your certificate (example_com.crt) and your private key (generated when you initially sent the certificate request) in the fields, you’ll likely get a ‘Certificate is missing intermediates’ error.

This is because GitLab is using something like NGINX to receive requests on it’s Pages IP before routing the request to the correct site. Namecheap, in their documentation, calls out that “it is required to combine your certificate with CA certificates in a single file”.

What this means for you is that you need to combine the text found in your example_com.crt and example_com.ca-bundle files in the ‘certificate field’. In the end you should have something like:

Add the private key to the last field, and you’re done. It will take time for the changes to propagate. If you check back in a few hours, you should see an indication beside your address in the URL bar showing that your connection to your site is now secure.

Resources/References

• https://about.gitlab.com/features/pages/
• https://docs.gitlab.com/ee/user/project/pages/getting_started_part_three.html#dns-txt-record
• https://www.namecheap.com/support/knowledgebase/article.aspx/9474/69/how-do-i-create-a-pem-file-from-the-certificates-i-received-from-you
• https://www.namecheap.com/support/knowledgebase/article.aspx/9637/68/how-can-i-complete-the-domain-control-validation-dcv-for-my-SSL-certificate#dns
• https://stackoverflow.com/a/49124195/2719852

A few weeks ago I bumped into mkcert, a tool written by Filippo. He is the same guy behind the popular heartbleed test tool.

The tool in question answers one simple need:

By creating a local root CA file that gets installed in your system, it makes all certificates issued by mkcert trusted.

After downloading the latest release from GitHub, you can simply “install” it by running mkcert -install. Once that is done, you can create your first, trusted (by your own system) certificate:

$ mkcert somedomain.local

Using the local CA at "/home/alex/.local/share/mkcert" ✨

Created a new certificate valid for the following names ?
 - "somedomain.local"

The certificate is at "./somedomain.local.pem" and the key at "./somedomain.local-key.pem" ✅

For example, here’s how it would look like if you had to boot a Node server with SSL support:

const fs = require('fs')

const options = {
  key: fs.readFileSync(__dirname + '/somedomain.local-key.pem'),
  cert: fs.readFileSync(__dirname + '/somedomain.local.pem')
};

require('https').createServer(options, (req, res) => {
  res.writeHead(200)
  res.end(`Got SSL?`)
}).listen(443)

Pretty neat, eh?

What mkcert does is to simply add another CA file in your system.(I guess under /etc/ssl/certs/ca-certificates.crt, but I’m not entirely sure). Browsers consider these certificates trusted — a nice workaround to trick any HTTP client.

That’s it for today! Adios!

_Originally published at odino.org (1st September 2018)._
_You can follow me on Twitter — rants are welcome!_ ?

Almost any website you visit today is protected by HTTPS. If yours isn’t yet, it should be. Securing your server with HTTPS also means that you can’t send requests to this server from one that isn’t protected by HTTPS. This poses a problem for developers who use a local development environment because all of them run on http://localhost out-of-the-box.

At the startup I’m a part of, we decided to secure our AWS Elastic Load Balancer endpoints with HTTPS as part of a move to enhance security. I ran into a situation where my local development environment’s requests to the server started getting rejected.

A quick Google search later, I found several articles like this, this or this one with detailed instructions on how I could implement HTTPS on localhost. None of these instructions seemed to work even after I followed them religiously. Chrome always threw a NET::ERR_CERT_COMMON_NAME_INVALID error at me.

The problem

All the detailed instructions I had found were correct for the time they were written. Not anymore.

After a ton of Googling, I discovered that the reason for my local certificate getting rejected was that Chrome had deprecated support for commonName matching in certificates, in effect, requiring a subjectAltName since January 2017.

The solution

We’ll be using OpenSSL to generate all of our certificates.

Step 1: Root SSL certificate

The first step is to create a Root Secure Sockets Layer (SSL) certificate. This root certificate can then be used to sign any number of certificates you might generate for individual domains. If you aren’t familiar with the SSL ecosystem, this article from DNSimple does a good job of introducing Root SSL certificates.

Generate a RSA-2048 key and save it to a file rootCA.key. This file will be used as the key to generate the Root SSL certificate. You will be prompted for a pass phrase which you’ll need to enter each time you use this particular key to generate a certificate.

openssl genrsa -des3 -out rootCA.key 2048

You can use the key you generated to create a new Root SSL certificate. Save it to a file namedrootCA.pem. This certificate will have a validity of 1,024 days. Feel free to change it to any number of days you want. You’ll also be prompted for other optional information.

openssl req -x509 -new -nodes -key rootCA.key -sha256 -days 1024 -out rootCA.pem

Step 2: Trust the root SSL certificate

Before you can use the newly created Root SSL certificate to start issuing domain certificates, there’s one more step. You need to to tell your Mac to trust your root certificate so all individual certificates issued by it are also trusted.

Open Keychain Access on your Mac and go to the Certificates category in your System keychain. Once there, import the rootCA.pem using File > Import Items. Double click the imported certificate and change the “When using this certificate:” dropdown to Always Trust in the Trust section.

Your certificate should look something like this inside Keychain Access if you’ve correctly followed the instructions till now.

Step 2: Domain SSL certificate

The root SSL certificate can now be used to issue a certificate specifically for your local development environment located at localhost.

Create a new OpenSSL configuration file server.csr.cnf so you can import these settings when creating a certificate instead of entering them on the command line.

[req]
default_bits = 2048
prompt = no
default_md = sha256
distinguished_name = dn

[dn]
C=US
ST=RandomState
L=RandomCity
O=RandomOrganization
OU=RandomOrganizationUnit
emailAddress=hello@example.com
CN = localhost

Create a v3.ext file in order to create a X509 v3 certificate. Notice how we’re specifying subjectAltName here.

authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
subjectAltName = @alt_names

[alt_names]
DNS.1 = localhost

Create a certificate key for localhost using the configuration settings stored in server.csr.cnf. This key is stored in server.key.

openssl req -new -sha256 -nodes -out server.csr -newkey rsa:2048 -keyout server.key -config <( cat server.csr.cnf )

A certificate signing request is issued via the root SSL certificate we created earlier to create a domain certificate for localhost. The output is a certificate file called server.crt.

openssl x509 -req -in server.csr -CA rootCA.pem -CAkey rootCA.key -CAcreateserial -out server.crt -days 500 -sha256 -extfile v3.ext

Use your new SSL certificate

You’re now ready to secure your localhost with HTTPS. Move the server.key and server.crt files to an accessible location on your server and include them when starting your server.

In an Express app written in Node.js, here’s how you would do it. Make sure you do this only for your local environment. Do not use this in production.

I hope you found this tutorial useful. If you’re not comfortable with running the commands given here by yourself, I’ve created a set of handy scripts you can run quickly to generate the certificates for you. More details can be found on the GitHub repo.

I love helping fellow web developers. Follow me on Twitter and let me know if you have any suggestions or feedback. If you’d like to show your appreciation towards any of the work I’ve done, be it a blog post, an open source project or just a funny tweet, you can buy me a cup of coffee.

When it comes to security-related tasks, like generating keys, CSRs, certificates, calculating digests, debugging TLS connections and other tasks related to PKI and HTTPS, you’d most likely end up using the OpenSSL tool.

OpenSSL includes tonnes of features covering a broad range of use cases, and it’s difficult to remember its syntax for all of them and quite easy to get lost. man pages are not so helpful here, so often we just Google “openssl how to [use case here]” or look for some kind of “openssl cheatsheet” to recall the usage of a command and see examples.

This post is my personal collection of openssl command snippets and examples, grouped by use case.

Use cases

Here is a list of use cases, that I’ll be covering:

1. Working with RSA and ECDSA keys
2. Create certificate signing requests (CSR)
3. Create X.509 certificates
4. Verify CSRs or certificates
5. Calculate message digests and base64 encoding
6. TLS client to connect to a remote server
7. Measure TLS connection and handshake time
8. Convert between encoding (PEM, DER) and container formats (PKCS12, PKCS7)
9. List ciphers suites
10. Manually check certificate revocation status from OCSP responder

Surely, this is not a complete list, but it covers the most common use cases and includes those I’ve been working with. For example, I skip encryption and decryption, or using openssl for CA management. openssl is like a universe. You never know where it ends. ?

Working with RSA and ECDSA keys

In the commands below, replace [bits] with the key size (For example, 2048, 4096, 8192).

Generate an RSA key:
openssl genrsa -out example.key [bits]

Print public key or modulus only:
openssl rsa -in example.key -pubout
openssl rsa -in example.key -noout -modulus

Print textual representation of RSA key:
openssl rsa -in example.key -text -noout

Generate new RSA key and encrypt with a pass phrase based on AES CBC 256 encryption:
openssl genrsa -aes256 -out example.key [bits]

Check your private key. If the key has a pass phrase, you’ll be prompted for it:
openssl rsa -check -in example.key

Remove passphrase from the key:
openssl rsa -in example.key -out example.key

Encrypt existing private key with a pass phrase:
openssl rsa -des3 -in example.key -out example_with_pass.key

Generate ECDSA key. curve is to be replaced with: prime256v1, secp384r1, secp521r1, or any other supported elliptic curve:
openssl ecparam -genkey -name [curve] | openssl ec -out example.ec.key

Print ECDSA key textual representation:
openssl ec -in example.ec.key -text -noout

List available EC curves, that OpenSSL library supports:
openssl ecparam -list_curves

Generate DH params with a given length:
openssl dhparam -out dhparams.pem [bits]

Create certificate signing requests (CSR)

In the commands below, replace [digest] with the name of the supported hash function: md5, sha1, sha224, sha256, sha384 or sha512, etc. It’s better to avoid weak functions like md5 and sha1, and stick to sha256 and above.

Create a CSR from existing private key.
openssl req -new -key example.key -out example.csr -[digest]

Create a CSR and a private key without a pass phrase in a single command:
openssl req -nodes -newkey rsa:[bits] -keyout example.key -out example.csr

Provide CSR subject info on a command line, rather than through interactive prompt.
openssl req -nodes -newkey rsa:[bits] -keyout example.key -out example.csr -subj "/C=UA/ST=Kharkov/L=Kharkov/O=Super Secure Company/OU=IT Department/CN=example.com"

Create a CSR from existing certificate and private key:
openssl x509 -x509toreq -in cert.pem -out example.csr -signkey example.key

Generate a CSR for multi-domain SAN certificate by supplying an openssl config file:
openssl req -new -key example.key -out example.csr -config req.conf

where req.conf:

[req]prompt=nodefault_md = sha256distinguished_name = dnreq_extensions = req_ext

[dn]CN=example.com

[req_ext]subjectAltName=@alt_names

[alt_names]DNS.1=example.comDNS.2=www.example.comDNS.3=ftp.example.com

Create X.509 certificates

Create self-signed certificate and new private key from scratch:
openssl req -nodes -newkey rsa:2048 -keyout example.key -out example.crt -x509 -days 365

Create a self signed certificate using existing CSR and private key:
openssl x509 -req -in example.csr -signkey example.key -out example.crt -days 365

Sign child certificate using your own “CA” certificate and it’s private key. If you were a CA company, this shows a very naive example of how you could issue new certificates.
openssl x509 -req -in child.csr -days 365 -CA ca.crt -CAkey ca.key -set_serial 01 -out child.crt

Print textual representation of the certificate
openssl x509 -in example.crt -text -noout

Print certificate’s fingerprint as md5, sha1, sha256 digest:
openssl x509 -in cert.pem -fingerprint -sha256 -noout

Verify CSRs or certificates

Verify a CSR signature:
openssl req -in example.csr -verify

Verify that private key matches a certificate and CSR:
openssl rsa -noout -modulus -in example.key | openssl sha256
openssl x509 -noout -modulus -in example.crt | openssl sha256
openssl req -noout -modulus -in example.csr | openssl sha256

Verify certificate, provided that you have root and any intemediate certificates configured as trusted on your machine:
openssl verify example.crt

Verify certificate, when you have intermediate certificate chain. Root certificate is not a part of bundle, and should be configured as a trusted on your machine.
openssl verify -untrusted intermediate-ca-chain.pem example.crt

Verify certificate, when you have intermediate certificate chain and root certificate, that is not configured as a trusted one.
openssl verify -CAFile root.crt -untrusted intermediate-ca-chain.pem child.crt

Verify that certificate served by a remote server covers given host name. Useful to check your mutlidomain certificate properly covers all the host names.
openssl s_client -verify_hostname [www.example.com](http://www.example.com) -connect example.com:443

Calculate message digests and base64 encoding

Calculate md5, sha1, sha256, sha384, sha512digests:
openssl dgst -[hash_function] <input.file
cat input.file | openssl [hash_function]

Base64 encoding and decoding:
cat /dev/urandom | head -c 50 | openssl base64 | openssl base64 -d

TLS client to connect to a remote server

Connect to a server supporting TLS:
openssl s_client -connect [example.com:443](http://www.google.com:443)
openssl s_client -host example.com -port 443

Connect to a server and show full certificate chain:
openssl s_client -showcerts -host example.com -port 443 </dev/null

Extract the certificate:
openssl s_client -connect example.com:443 2>&1 < /dev/null | sed -n '/-----BEGIN/,/-----END/p' > certificate.pem

Override SNI (Server Name Indication) extension with another server name. Useful for testing when multiple secure sites are hosted on same IP address:
openssl s_client -servername [www.](http://www.foobbz.site)example.com -host example.com -port 443

Test TLS connection by forcibly using specific cipher suite, e.g. ECDHE-RSA-AES128-GCM-SHA256. Useful to check if a server can properly talk via different configured cipher suites, not one it prefers.
openssl s_client -host example.com -port 443 -cipher ECDHE-RSA-AES128-GCM-SHA256 2>&1 </dev/null

Measure TLS connection and handshake time

Measure SSL connection time without/with session reuse:
openssl s_time -connect example.com:443 -new
openssl s_time -connect example.com:443 -reuse

Roughly examine TCP and SSL handshake times using curl:
curl -kso /dev/null -w "tcp:%{time_connect}, ssldone:%{time_appconnect}\n" https://example.com

Measure speed of various security algorithms:
openssl speed rsa2048
openssl speed ecdsap256

Convert between encoding and container formats

Convert certificate between DER and PEM formats:
openssl x509 -in example.pem -outform der -out example.der
openssl x509 -in example.der -inform der -out example.pem

Combine several certificates in PKCS7 (P7B) file:
openssl crl2pkcs7 -nocrl -certfile child.crt -certfile ca.crt -out example.p7b

Convert from PKCS7 back to PEM. If PKCS7 file has multiple certificates, the PEM file will contain all of the items in it.
openssl pkcs7 -in example.p7b -print_certs -out example.crt

Combine a PEM certificate file and a private key to PKCS#12 (.pfx .p12). Also, you can add a chain of certificates to PKCS12 file.
openssl pkcs12 -export -out certificate.pfx -inkey privkey.pem -in certificate.pem -certfile ca-chain.pem

Convert a PKCS#12 file (.pfx .p12) containing a private key and certificates back to PEM:
openssl pkcs12 -in keystore.pfx -out keystore.pem -nodes

List cipher suites

List available TLS cipher suites, openssl client is capable of:
openssl ciphers -v

Enumerate all individual cipher suites, which are described by a short-hand OpenSSL cipher list string. This is useful when you’re configuring server (like Nginx), and you need to test your ssl_ciphers string.
openssl ciphers -v 'EECDH+ECDSA+AESGCM:EECDH+aRSA+SHA256:EECDH:DHE+AESGCM:DHE:!RSA!aNULL:!eNULL:!LOW:!RC4'

Manually check certificate revocation status from OCSP responder

This is a multi-step process:

1. Retrieve the certificate from a remote server
2. Obtain the intermediate CA certificate chain
3. Read OCSP endpoint URI from the certificate
4. Request a remote OCSP responder for certificate revocation status

First, retrieve the certificate from a remote server:
openssl s_client -connect example.com:443 2>&1 < /dev/null | sed -n '/-----BEGIN/,/-----END/p' > cert.pem

You’d also need to obtain intermediate CA certificate chain. Use -showcerts flag to show full certificate chain, and manually save all intermediate certificates to chain.pem file:
openssl s_client -showcerts -host example.com -port 443 </dev/null

Read OCSP endpoint URI from the certificate:
openssl x509 -in cert.pem -noout -ocsp_uri

Request a remote OCSP responder for certificate revocation status using the URI from the above step (e.g. http://ocsp.stg-int-x1.letsencrypt.org).
openssl ocsp -header "Host" "ocsp.stg-int-x1.letsencrypt.org" -issuer chain.pem -VAfile chain.pem -cert cert.pem -text -url [http://ocsp.stg-int-x1.letsencrypt.org](http://ocsp.stg-int-x1.letsencrypt.org)

Resources

I’ve put together a few resources about OpenSSL that you may find useful.

OpenSSL Essentials: Working with SSL Certificates, Private Keys and CSRs | DigitalOcean — https://www.digitalocean.com/community/tutorials/openssl-essentials-working-with-ssl-certificates-private-keys-and-csrs

The Most Common OpenSSL Commands — https://www.sslshopper.com/article-most-common-openssl-commands.html

OpenSSL: Working with SSL Certificates, Private Keys and CSRs — https://www.dynacont.net/documentation/linux/openssl/