Companies like LinkedIn, Uber, and Netflix use Kafka to process trillions of events and petabtyes of data each day.

Kafka was originally developed at LinkedIn, to help handle their real-time data feeds. It's now maintained by the Apache Software Foundation, and is widely adopted in industry (being used by 80% of Fortune 100 companies).

Why Should You Learn Apache Kafka?

Kafka lets you:

• Publish and subscribe to streams of events

• Store streams of events in the same order they happened

• Process streams of events in real time

The main thing Kafka does is help you efficiently connect diverse data sources with the many different systems that might need to use that data.

Kafka helps you connect data sources to the systems using that data

Some of the things you can use Kafka for include:

• Personalizing recommendations for customers

• Notifying passengers of flight delays

• Payment processing in banking

• Online fraud detection

• Managing inventory and supply chains

• Tracking order shipments

• Collecting telemetry data from Internet of Things (IoT) devices

What all these uses have in common is that they need to take in and process data in real time, often at huge scales. This is something Kafka excels at. To give one example, Pinterest uses Kafka to handle up to 40 million events per second.

Kafka is distributed, which means it runs as a cluster of nodes spread across multiple servers. It's also replicated, meaning that data is copied in multiple locations to protect it from a single point of failure. This makes Kafka both scalable and fault-tolerant.

Kafka is also fast. It's optimized for high throughput, making effective use of disk storage and batched network requests.

This article will:

• Introduce you to the core concepts behind Kafka

• Show you how to install Kafka on your own computer

• Get you started with the Kafka Command Line Interface (CLI)

• Help you build a simple Java application that produces and consumes events via Kafka

Things the article won't cover:

• More advanced Kafka topics, such as security, performance, and monitoring

• Deploying a Kafka cluster to a server

• Using managed Kafka services like Amazon MSK or Confluent Cloud

Table of Contents

1. Event Streaming and Event-Driven Architectures

2. Core Kafka Concepts
   a. Event Messages in Kafka
   b. Topics in Kafka
   c. Partitions in Kafka
   d. Offsets in Kafka
   e. Brokers in Kafka
   f. Replication in Kafka
   g. Producers in Kafka
   h. Consumers in Kafka
   i. Consumer Groups in Kafka
   j. Kafka Zookeeper

3. How to Install Kafka on Your Computer

4. How to Start Zookeeper and Kafka

5. The Kafka CLI
   a. How to List Topics
   b. How to Create a Topic
   c. How to Describe Topics
   d. How to Partition a Topic
   e. How to Set a Replication Factor
   f. How to Delete a Topic
   g. How to use kafka-console-producer
   h. How to use kafka-console-consumer
   i. How to use kafka-consumer-groups

6. How to Build a Kafka Client App with Java
   a. How to Set Up the Project
   b. How to Install the Dependencies
   c. How to Create a Kafka Producer
   d. How to Send Multiple Messages and Use Callbacks
   e. How to Create a Kafka Consumer
   f. How to Shut Down the Consumer

7. Where to Take it From Here

Before we dive into Kafka, we need some context on event streaming and event-driven architectures.

Event Streaming and Event-Driven Architectures

An event is a record that something happened, as well as information about what happened. For example: a customer placed an order, a bank approved a transaction, inventory management updated stock levels.

Events can triggers one or more processes to respond to them. For example: sending an email receipt, transmitting funds to an account, updating a real-time dashboard.

Event streaming is the process of capturing events in real-time from sources (such as web applications, databases, or sensors) to create streams of events. These streams are potentially unending sequences of records.

The event stream can be stored, processed, and sent to different destinations, also called sinks. The destinations that consume the streams could be other applications, databases, or data pipelines for further processing.

As applications have become more complex, often being broken up into different microservices distributed across multiple data centers, many organizations have adopted an event-driven architecture for their applications.

This means that instead of parts of your application directly asking each other for updates about what happened, they each publish events to event streams. Other parts of the application continuously subscribe to these streams and only act when they receive an event that they are interested in.

This architecture helps ensure that if part of your application goes down, other parts won't also fail. Additionally, you can add new features by adding new subscribers to the event stream, without having to rewrite the existing codebase.

Core Kafka Concepts

Kafka has become one of the most popular ways to implement event streaming and event-driven architectures. But it does have a bit of a learning curve and you need to understand a couple of concepts before you can make effective use of it.

These core concepts are:

• event messages

• topics

• partitions

• offsets

• brokers

• producers

• consumers

• consumer groups

• Zookeeper

Event Messages in Kafka

When you write data to Kafka, or read data from it, you do this in the form of messages. You'll also see them called events or records.

A message consists of:

• a key

• a value

• a timestamp

• a compression type

• headers for metadata (optional)

• partition and offset id (once the message is written to a topic)

A Kafka message consisting of key, value, timestamp, compression type, and headers

Every event in Kafka is, at its simplest, a key-value pair. These are serialized into binary, since Kafka itself handles arrays of bytes rather than complex language-specific objects.

Keys are usually strings or integers and aren't unique for every message. Instead, they point to a particular entity in the system, such as a specific user, order, or device. Keys can be null, but when they are included they are used for dividing topics into partitions (more on partitions below).

The message value contains details about the event that happened. This could be as simple as a string or as complex as an object with many nested properties. Values can be null, but usually aren't.

By default, the timestamp records when the message was created. You can overwrite this if your event actually occurred earlier and you want to record that time instead.

Messages are usually small (less than 1 MB) and sent in a standard data format, such as JSON, Avro, or Protobuf. Even so, they can be compressed to save on data. The compression type can be set to gzip, lz4, snappy, zstd, or none.

Events can also optionally have headers, which are key-value pairs of strings containing metadata, such as where the event originated from or where you want it routed to.

Once a message is sent into a Kafka topic, it also receives a partition number and offset id (more about these later).

Topics in Kafka

Kafka stores messages in a topic, an ordered sequence of events, also called an event log.

A Kafka topic containing messages, each with a unique offset

Different topics are identified by their names and will store different kinds of events. For example a social media application might have posts, likes, and comments topics to record every time a user creates a post, likes a post, or leaves a comment.

Multiple applications can write to and read from the same topic. An application might also read messages from one topic, filter or transform the data, and then write the result to another topic.

One important feature of topics is that they are append-only. When you write a message to a topic, it's added to the end of the log. Events in a topic are immutable. Once they're written to a topic, you can't change them.

A Producer writing events to topics and a Consumer reading events from topics

Unlike with messaging queues, reading an event from a topic doesn't delete it. Events can be read as often as needed, perhaps several times by multiple different applications.

Topics are also durable, holding onto messages for a specific period (by default 7 days) by saving them to physical storage on disk.

You can configure topics so that messages expire after a certain amount of time, or when a certain amount of storage is exceeded. You can even store messages indefinitely as long as you can pay for the storage costs.

Partitions in Kafka

In order to help Kafka to scale, topics can be divided into partitions. This breaks up the event log into multiple logs, each of which lives on a separate node in the Kafka cluster. This means that the work of writing and storing messages can be spread across multiple machines.

When you create a topic, you specify the amount of partitions it has. The partitions are themselves numbered, starting at 0. When a new event is written to a topic, it's appended to one of the topic's partitions.

A topic divided into three partitions

If messages have no key, they will be evenly distributed among partitions in a round robin manner: partition 0, then partition 1, then partition 2, and so on. This way, all partitions get an even share of the data but there's no guarantee about the ordering of messages.

Messages that have the same key will always be sent to the same partition, and in the same order. The key is run through a hashing function which turns it into an integer. This output is then used to select a partition.

Messages without keys are sent across partitions, while messages with the same keys are sent to the same partition

Messages within each partition are guaranteed to be ordered. For example, all messages with the same customer_id as their key will be sent to the same partition in the order in which Kafka received them.

Offsets in Kafka

Each message in a partition gets an id that is an incrementing integer, called an offset. Offsets start at 0 and are incremented every time Kafka writes a message to a partition. This means that each message in a given partition has a unique offset.

Offsets are unique within a partition but not between partitions

Offsets are not reused, even when older messages get deleted. They continue to increment, giving each new message in the partition a unique id.

When data is read from a partition, it is read in order from the lowest existing offset upwards. We'll see more about offsets when we cover Kafka consumers.

Brokers in Kafka

A single "server" running Kafka is called a broker. In reality, this might be a Docker container running in a virtual machine. But it can be a helpful mental image to think of brokers as individual servers.

A Kafka cluster made up of three brokers

Multiple brokers working together make up a Kafka cluster. There might be a handful of brokers in a cluster, or more than 100. When a client application connects to one broker, Kafka automatically connects it to every broker in the cluster.

By running as a cluster, Kafka becomes more scalable and fault-tolerant. If one broker fails, the others will take over its work to ensure there is no downtime or data loss.

Each broker manages a set of partitions and handles requests to write data to or read data from these partitions. Partitions for a given topic will be spread evenly across the brokers in a cluster to help with load balancing. Brokers also manage replicating partitions to keep their data backed up.

Partitions spread across brokers

Replication in Kafka

To protect against data loss if a broker fails, Kafka writes the same data to copies of a partition on multiple brokers. This is called replication.

The main copy of a partition is called the leader, while the replicas are called followers.

The data from the leader partition is copied to follower partitions on different brokers

When a topic is created, you set a replication factor for it. This controls how many replicas get written to. A replication factor of three is common, meaning data gets written to one leader and replicated to two followers. So even if two brokers failed, your data would still be safe.

Whenever you write messages to a partition, you're writing to the leader partition. Kafka then automatically copies these messages to the followers. As such, the logs on the followers will have the same messages and offsets as on the leader.

Followers that are up to date with the leader are called In-Sync Replicas (ISRs). Kafka considers a message to be committed once a minimum number of replicas have saved it to their logs. You can configure this to get higher throughput at the expense of less certainty that a message has been backed up.

Producers in Kafka

Producers are client applications that write events to Kafka topics. These apps aren't themselves part of Kafka – you write them.

Usually you will use a library to help manage writing events to Kafka. There is an official client library for Java as well as dozens of community-supported libraries for languages such as Scala, JavaScript, Go, Rust, Python, C#, and C++.

A Producer application writing to multiple topics

Producers are totally decoupled from consumers, which read from Kafka. They don't know about each other and their speed doesn't affect each other. Producers aren't affected if consumers fail, and the same is true for consumers.

If you need to, you could write an application that writes certain events to Kafka and reads other events from Kafka, making it both a producer and a consumer.

Producers take a key-value pair, generate a Kafka message, and then serialize it into binary for transmission across the network. You can adjust the configuration of producers to batch messages together based on their size or some fixed time limit to optimize writing messages to the Kafka brokers.

It's the producer that decides which partition of a topic to send each message to. Again, messages without keys will be distributed evenly among partitions, while messages with keys are all sent to the same partition.

Consumers in Kafka

Consumers are client applications that read messages from topics in a Kafka cluster. Like with producers, you write these applications yourself and can make use of client libraries to support the programming language your application is built with.

A Consumer reading messages from multiple topics

Consumers can read from one or more partitions within a topic, and from one or more topics. Messages are read in order within a partition, from the lowest available offset to the highest. But if a consumer reads data from several partitions in the same topic, the message order between these partitions is not guaranteed.

For example, a consumer might read messages from partition 0, then partition 2, then partition 1, then back to partition 0. The messages from partition 0 will be read in order, but there might be messages from the other partitions mixed among them.

It's important to remember that reading a message does not delete it. The message is still available to be read by any other consumer that needs to access it. It's normal for multiple consumers to read from the same topic if they each have uses for the data in it.

By default, when a consumer starts up it will read from the current offset in a partition. But consumers can also be configured to go back and read from the oldest existing offset.

Consumers deserialize messages, converting them from binary into a collection of key-value pairs that your application can then work with. The format of a message should not change during a topic's lifetime or your producers and consumers won't be able to serialize and deserialize it correctly.

One thing to be aware of is that consumers request messages from Kafka, it doesn't push messages to them. This protects consumers from becoming overwhelmed if Kafka is handling a high volume of messages. If you want to scale consumers, you can run multiple instances of a consumer together in a consumer group.

Consumer Groups in Kafka

An application that reads from Kafka can create multiple instances of the same consumer to split up the work of reading from different partitions in a topic. These consumers work together as a consumer group.

When you create a consumer, you can assign it a group id. All consumers in a group will have the same group id.

You can create consumer instances in a group up to the number of partitions in a topic. So if you have a topic with 5 partitions, you can create up to 5 instances of the same consumer in a consumer group. If you ever have more consumers in a group than partitions, the extra consumer will remain idle.

Consumers in a consumer group reading messages from a topic's partitions

If you add another consumer instance to a consumer group, Kafka will automatically redistribute the partitions among the consumers in a process called rebalancing.

Each partition is only assigned to one consumer in a group, but a consumer can read from multiple partitions. Also, multiple different consumer groups (meaning different applications) can read from the same topic at the same time.

Kafka brokers use an internal topic called __consumer_offsets to keep track of which messages a specific consumer group has successfully processed.

As a consumer reads from a partition, it regularly saves the offset it has read up to and sends this data to the broker it is reading from. This is called the consumer offset and is handled automatically by most client libraries.

A Consumer committing the offsets it has read up to

If a consumer crashes, the consumer offset helps the remaining consumers to know where to start from when they take over reading from the partition.

The same thing happens if a new consumer is added to the group. The consumer group rebalances, the new consumer is assigned a partition, and it picks up reading from the consumer offset of that partition.

Kafka Zookeeper

One other topic that we briefly need to cover here is how Kafka clusters are managed. Currently this is usually done using Zookeeper, a service for managing and synchronizing distributed systems. Like Kafka, it's maintained by the Apache Foundation.

Kafka uses Zookeeper to manage the brokers in a cluster, and requires Zookeeper even if you're running a Kafka cluster with only one broker.

Recently, a proposal has been accepted to remove Zookeeper and have Kafka manage itself (KIP-500), but this is not yet widely used in production.

Zookeeper keeps track of things like:

• Which brokers are part of a Kafka cluster

• Which broker is the leader for a given partition

• How topics are configured, such as the number of partitions and the location of replicas

• Consumer groups and their members

• Access Control Lists – who is allowed to write to and read from each topic

A Zookeeper ensemble managing the brokers in a Kafka cluster

Zookeeper itself runs as a cluster called an ensemble. This means that Zookeeper can keep working even if one node in the cluster fails. New data gets written to the ensemble's leader and replicated to the followers. Your Kafka brokers can read this data from any of the Zookeeper nodes in the ensemble.

Now that you understand the main concepts behind Kafka, let's get some hands-on practice working with Kafka.

You're going to install Kafka on your own computer, practice interacting with Kafka brokers from the command line, and then build a simple producer and consumer application with Java.

How to Install Kafka on Your Computer

At the time of writing this guide, the latest stable version of Kafka is 3.3.1. Check kafka.apache.org/downloads to see if there is a more recent stable version. If there is, you can replace "3.3.1" with the latest stable version in all of the following instructions.

Install Kafka on macOS

If you're using macOS, I recommend using Homebrew to install Kafka. It will make sure you have Java installed before it installs Kafka.

If you don't already have Homebrew installed, install it by following the instructions at brew.sh.

Next, run brew install kafka in a terminal. This will install Kafka's binaries at usr/local/bin.

Finally, run kafka-topics --version in a terminal and you should see 3.3.1. If you do, you're all set.

To make it easier to work with Kafka, you can add Kafka to the PATH environment variable. Open your ~/.bashrc (if using Bash) or ~/.zshrc (if using Zsh) and add the following line, replacing USERNAME with your username:

PATH="$PATH:/Users/USERNAME/kafka_2.13-3.3.1/bin"

You'll need to close your terminal for this change to take effect.

Now, if you run echo $PATH you should see that the Kafka bin directory has been added to your path.

Install Kafka on Windows (WSL2) and Linux

Kafka isn't natively supported on Windows, so you will need to use either WSL2 or Docker. I'm going to show you WSL2 since it's the same steps as Linux.

To set up WSL2 on Widows, follow the instructions in the official docs.

From here on, the instructions are the same for both WSL2 and Linux.

First, install Java 11 by running the following commands:

wget -O- https://apt.corretto.aws/corretto.key | sudo apt-key add - 

sudo add-apt-repository 'deb https://apt.corretto.aws stable main'

sudo apt-get update; sudo apt-get install -y java-11-amazon-corretto-jdk

Once this has finished, run java -version and you should see something like:

openjdk version "11.0.17" 2022-10-18 LTS
OpenJDK Runtime Environment Corretto-11.0.17.8.1 (build 11.0.17+8-LTS)
OpenJDK 64-Bit Server VM Corretto-11.0.17.8.1 (build 11.0.17+8-LTS, mixed mode)

From your root directory, download Kafka with the following command:

wget https://archive.apache.org/dist/kafka/3.3.1/kafka_2.13-3.3.1.tgz

The 2.13 means it is using version 2.13 of Scala, while 3.3.1 refers to the Kafka version.

Extract the contents of the download with:

tar xzf kafka_2.13-3.3.1.tgz

If you run ls, you'll now see kafka_2.13-3.3.1 in your root directory.

To make it easier to work with Kafka, you can add Kafka to the PATH environment variable. Open your ~/.bashrc (if using Bash) or ~/.zshrc (if using Zsh) and add the following line, replacing USERNAME with your username:

PATH="$PATH:home/USERNAME/kafka_2.13-3.3.1/bin"

You'll need to close your terminal for this change to take effect.

Now, if you run echo $PATH you should see that the Kafka bin directory has been added to your path.

Run kafka-topics.sh --version in a terminal and you should see 3.3.1. If you do, you're all set.

How to Start Zookeeper and Kafka

Since Kafka uses Zookeeper to manage clusters, you need to start Zookeeper before you start Kafka.

How to Start Kafka on macOS

In one terminal window, start Zookeeper with:

/usr/local/bin/zookeeper-server-start /usr/local/etc/zookeeper/zoo.cfg

In another terminal window, start Kafka with:

/usr/local/bin/kafka-server-start /usr/local/etc/kafka/server.properties

While using Kafka, you need to keep both these terminal windows open. Closing them will shut down Kafka.

How to Start Kafka on Windows (WSL2) and Linux

In one terminal window, start Zookeeper with:

~/kafka_2.13-3.3.1/bin/zookeeper-server-start.sh ~/kafka_2.13-3.3.1/config/zookeeper.properties

In another terminal window, start Kafka with:

~/kafka_2.13-3.3.1/bin/kafka-server-start.sh ~/kafka_2.13-3.3.1/config/server.properties

While using Kafka, you need to keep both these terminal windows open. Closing them will shut down Kafka.

Now that you have Kafka installed and running on your machine, it's time to get some hands-on practice.

The Kafka CLI

When you install Kafka, it comes with a Command Line Interface (CLI) that lets you create and manage topics, as well as produce and consume events.

First, make sure Zookeeper and Kafka are running in two terminal windows.

In a third terminal window, run kafka-topics.sh (on WSL2 or Linux) or kafka-topics (on macOS) to make sure the CLI is working. You'll see a list of all the options you can pass to the CLI.

kafka-topics options

Note: When working with the Kafka CLI, the command will be kafka-topics.sh on WSL2 and Linux. It will be kafka-topics.sh on macOS if you directly installed the Kafka binaries and kafka-topics if you used Homebrew. So if you're using Homebrew, remove the .sh extension from the example commands in this section.

How to List Topics

To see the topics available on the Kafka broker on your local machine, use:

kafka-topics.sh --bootstrap-server localhost:9092 --list

This means "Connect to the Kafka broker running on localhost:9092 and list all topics there". --bootstrap-server refers to the Kafka broker you are trying to connect to and localhost:9092 is the IP address it's running at. You won't see any output since you haven't created any topics yet.

How to Create a Topic

To create a topic (with the default replication factor and number of partitions), use the --create and --topic options and pass them a topic name:

kafka-topics.sh --bootstrap-server localhost:9092 --create --topic my_first_topic

If you use an _ or . in your topic name, you will see the following warning:

WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.

Since Kafka could confuse my.first.topic with my_first_topic, it's best to only use either underscores or periods when naming topics.

How to Describe Topics

To describe the topics on a broker, use the --describe option:

kafka-topics.sh --bootstrap-server localhost:9092 --describe

This will print the details of all the topics on this broker, including the number of partitions and their replication factor. By default, these will both be set to 1.

If you add the --topic option and the name of a topic, it will describe only that topic:

kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic my_first_topic

How to Partition a Topic

To create a topic with multiple partitions, use the --partitions option and pass it a number:

kafka-topics.sh --bootstrap-server localhost:9092 --create --topic my_second_topic --partitions 3

How to Set a Replication Factor

To create a topic with a replication factor higher than the default, use the --replication-factor option and pass it a number:

kafka-topics.sh --bootstrap-server localhost:9092 --create --topic my_third_topic --partitions 3 --replication-factor 3

You should get the following error:

ERROR org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 2 larger than available brokers: 1.

Since you're only running one Kafka broker on your machine, you can't set a replication factor higher than one. If you were running a cluster with multiple brokers, you could set a replication factor as high as the total number of brokers.

How to Delete a Topic

To delete a topic, use the --delete option and specify a topic with the --topic option:

kafka-topics.sh --bootstrap-server localhost:9092 --delete --topic my_first_topic

You won't get any output to say the topic was deleted but you can check using --list or --describe.

How to Use kafka-console-producer

You can produce messages to a topic from the command line using kafka-console-producer.

Run kafka-console-producer.sh to see the options you can pass to it.

kafka-console-producer options

To create a producer connected to a specific topic, run:

kafka-console-producer.sh --bootstrap-server localhost:9092 --topic TOPIC_NAME

Let's produce messages to the my_first_topic topic.

kafka-console-producer.sh --bootstrap-server localhost:9092 --topic my_first_topic

Your prompt will change and you will be able to type text. Press enter to send that message. You can keep sending messages until you press ctrl + c.

Sending messages using kafka-console-producer

If you produce messages to a topic that doesn't exist, you'll get a warning, but the topic will be created and the messages will still get sent. It's better to create a topic in advance, however, so you can specify partitions and replication.

By default, the messages sent from kafka-console-producer have their keys set to null, and so they will be evenly distributed to all partitions.

You can set a key by using the --property option to set parse.key to be true and providing a key separator, such as :

For example, we can create a books topic and use the books' genre as a key.

kafka-topics.sh --bootstrap-server localhost:9092 --topic books --create

kafka-console-producer.sh --bootstrap-server localhost:9092 --topic books --property parse.key=true --property key.separator=:

Now you can enter keys and values in the format key:value. Anything to the left of the key separator will be interpreted as a message key, anything to the right as a message value.

science_fiction:All Systems Red
fantasy:Uprooted
horror:Mexican Gothic

Producing messages with keys and values

Now that you've produced messages to a topic from the command line, it's time to consume those messages from the command line.

How to Use kafka-console-consumer

You can consumer messages from a topic from the command line using kafka-console-consumer.

Run kafka-console-consumer.sh to see the options you can pass to it.

kafka-console-consumer options

To create a consumer, run:

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic TOPIC_NAME

When you start a consumer, by default it will read messages as they are written to the end of the topic. It won't read messages that were previously sent to the topic.

If you want to read the messages you already sent to a topic, use the --from-beginning option to read from the beginning of the topic:

kafka-console-consumer --bootstrap-server localhost:9092 --topic my_first_topic --from-beginning

The messages might appear "out of order". Remember, messages are ordered within a partition but ordering can't be guaranteed between partitions. If you don't set a key, they will be sent round robin between partitions and ordering isn't guaranteed.

You can display additional information about messages, such as their key and timestamp, by using the --property option and setting the print property to true.

Use the --formatter option to set the message formatter and the --property option to select which message properties to print.

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic my_first_topic --from-beginning --formatter kafka.tools.DefaultMessageFormatter --property print.timestamp=true --property print.key=true --property print.value=true

Consuming messages from a topic

We get the messages' timestamp, key, and value. Since we didn't assign any keys when we sent these messages to my_first_topic, their key is null.

How to Use kafka-consumer-groups

You can run consumers in a consumer group using the Kafka CLI. To view the documentation for this, run:

kafka-consumer-groups.sh

kafka-consumer-groups options

First, create a topic with three partitions. Each consumer in a group will consume from one partition. If there are more consumers than partitions, any extra consumers will be idle.

kafka-topics.sh --bootstrap-server localhost:9092 --topic fantasy_novels --create --partitions 3

You add a consumer to a group when you create it using the --group option. If you run the same command multiple times with the same group name, each new consumer will be added to the group.

To create the first consumer in your consumer group, run:

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic fantasy_novels --group fantasy_consumer_group

Next, open two new terminal windows and run the same command again to add a second and third consumer to the consumer group.

Three consumers running in a consumer group

In a different terminal window, create a producer and send a few messages with keys to the topic.

Note: Since Kafka 2.4, Kafka will send messages in batches to one "sticky" partition for better performance. In order to demonstrate messages being sent round robin between partitions (without sending a large volume of messages), we can set the partitioner to RoundRobinPartitioner.

kafka-console-producer.sh --bootstrap-server localhost:9092 --topic fantasy_novels --property parse.key=true --property key.separator=: --property partitioner.class=org.apache.kafka.clients.producer.RoundRobinPartitioner

tolkien:The Lord of the Rings
le_guin:A Wizard of Earthsea
leckie:The Raven Tower
de_bodard:The House of Shattered Wings
okorafor:Who Fears Death
liu:The Grace of Kings

Messages spread between consumers in a consumer group

If you stop one of the consumers, the consumer group will rebalance and future messages will be sent to the remaining consumers.

Now that you have some experience working with Kafka from the command line, the next step is to build a small application that connects to Kafka.

How to Build a Kafka Client App with Java

We're going to build a simple Java app that both produces messages to and consumes messages from Kafka. For this we'll use the official Kafka Java client.

If at any point you get stuck, the full code for this project is available on GitHub.

Preliminaries

First of all, make sure you have Java (at least JDK 11) and Kafka installed.

We're going to send messages about characters from The Lord of the Rings. So let's create a topic for these messages with three partitions.

From the command line, run:

kafka-topics.sh --bootstrap-server localhost:9092 --create --topic lotr_characters --partitions 3

How to Set Up the Project

I recommend using IntelliJ for Java projects, so go ahead and install the Community Edition if you don't already have it. You can download it from jetbrains.com/idea

In Intellij, select File, New, and Project.

Give your project a name and select a location for it on your computer. Make sure you have selected Java as the language, Maven as the build system, and that the JDK is at least Java 11. Then click Create.

Setting up a Maven project in IntelliJ

Note: If you're on Windows, IntelliJ can't use a JDK installed on WSL. To install Java on the Windows side of things, go to docs.aws.amazon.com/corretto/latest/corretto-11-ug/downloads-list and download the Windows installer. Follow the installation steps, open a command prompt, and run java -version. You should see something like:

openjdk version "11.0.18" 2023-01-17 LTS
OpenJDK Runtime Environment Corretto-11.0.18.10.1 (build 11.0.18+10-LTS)
OpenJDK 64-Bit Server VM Corretto-11.0.18.10.1 (build 11.0.18+10-LTS, mixed mode)

Once your Maven project finishes setting up, run the Main class to see "Hello world!" and make sure everything worked.

How to Install the Dependencies

Next, we're going to install our dependencies. Open up pom.xml and inside the <project> element, create a <dependencies> element.

We're going to use the Java Kafka client for interacting with Kafka and SLF4J for logging, so add the following inside your <dependencies> element:

<!-- https://mvnrepository.com/artifact/org.apache.kafka/kafka-clients -->  
<dependency>  
    <groupId>org.apache.kafka</groupId>  
    <artifactId>kafka-clients</artifactId>  
    <version>3.3.1</version>  
</dependency>  
<!-- https://mvnrepository.com/artifact/org.slf4j/slf4j-api -->  
<dependency>  
    <groupId>org.slf4j</groupId>  
    <artifactId>slf4j-api</artifactId>  
    <version>2.0.6</version>  
</dependency>  
<!-- https://mvnrepository.com/artifact/org.slf4j/slf4j-simple -->  
<dependency>  
    <groupId>org.slf4j</groupId>  
    <artifactId>slf4j-simple</artifactId>  
    <version>2.0.6</version>  
</dependency>

The package names and version numbers might be red, meaning you haven't downloaded them yet. If this happens, click on View, Tool Windows, and Maven to open the Maven menu. Click on the Reload All Maven Projects icon and Maven will install these dependencies.

Reloading Maven dependencies in IntelliJ

Create a HelloKafka class in the same directory as your Main class and give it the following contents:

package org.example;

import org.slf4j.Logger;  
import org.slf4j.LoggerFactory;  

public class HelloKafka {  
    private static final Logger log = LoggerFactory.getLogger(HelloKafka.class);  

    public static void main(String[] args) {  
        log.info("Hello Kafka");  
    }  
}

To make sure your dependencies are installed, run this class and you should see [main] INFO org.example.HelloKafka - Hello Kafka printed to the IntelliJ console.

How to Create a Kafka Producer

Next, we're going to create a Producer class. You can call this whatever you want as long as it doesn't clash with another class. So don't use KafkaProducer as you'll need that class in a minute.

package org.example;  

import org.slf4j.Logger;  
import org.slf4j.LoggerFactory;  

public class Producer {  
    private static final Logger log = LoggerFactory.getLogger(KafkaProducer.class);  

    public static void main(String[] args) {  
        log.info("This class will produce messages to Kafka");  
    }  
}

All of our Kafka-specific code is going to go inside this class's main() method.

The first thing we need to do is configure a few properties for the producer. Add the following inside the main() method:

Properties properties = new Properties(); 

properties.setProperty(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");  
properties.setProperty(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());  
properties.setProperty(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());

Properties stores a set of properties as pairs of strings. The ones we're using are:

• ProducerConfig.BOOTSTRAP_SERVERS_CONFIG which specifies the IP address to use to access the Kafka cluster

• ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG which specifies the serializer to use for message keys

• ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG which specifies the serializer to use for message values

We're going to connect to our local Kafka cluster running on localhost:9092, and use the StringSerializer since both our keys and values will be strings.

Now we can create our producer and pass it the configuration properties.

KafkaProducer<String, String> producer = new KafkaProducer<>(properties);

To send a message, we need to create a ProducerRecord and pass it to our producer. ProducerRecord contains a topic name, and optionally a key, value, and partition number.

We're going to create the ProducerRecord with the topic to use, the message's key, and the message's value.

ProducerRecord<String, String> producerRecord = new ProducerRecord<>("lotr_characters", "hobbits", "Bilbo");

We can now use the producer's send() method to send the message to Kafka.

producer.send(producerRecord);

Finally, we need to call the close() method to stop the producer. This method handles any messages currently being processed by send() and then closes the producer.

producer.close();

Now it's time to run our producer. Make sure you have Zookeeper and Kafka running. Then run the main() method of the Producer class.

Sending a message from a producer in a Java Kafka client app

Note: On Windows, your producer might not be able to connect to a Kafka broker running on WSL. To fix this, you're going to need to do the following:

• In a WSL terminal, navigate to Kafka's config folder: cd ~/kafka_2.13-3.3.1/config/

• Open server.properties, for example with Nano: nano server.properties

• Uncomment #listeners=PLAINTEXT//:9092

• Replace it with listeners=PLAINTEXT//[::1]:9092

• In your Producer class, replace "localhost:9092" with "[::1]:9092"

[::1], or 0:0:0:0:0:0:0:1, refers to the loopback address (or localhost) in IPv6. This is equivalent to 127.0.0.1 in IPv4.

If you change listeners, when you try to access the Kafka broker from the command line you'll also have to use the new IP address, so use --bootstrap-server ::1:9092 instead of --bootstrap-server localhost:9092 and it should work.

We can now check that Producer worked by using kafka-console-consumer in another terminal window to read from the lotr_characters topic and see the message printed to the console.

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic lotr_characters --from-beginning

kafka-console-consumer reading the message sent by the producer in our Java app

How to Send Multiple Messages and Use Callbacks

So far we're only sending one message. If we update Producer to send multiple messages, we'll be able to see how keys are used to divide messages between partitions. We can also take this opportunity to use a callback to view the sent message's metadata.

To do this, we're going to loop over a collection of characters to generate our messages.

So replace this:

ProducerRecord<String, String> producerRecord = new ProducerRecord<>("lotr_characters", "hobbits", "Bilbo");  

producer.send(producerRecord);

with this:

HashMap<String, String> characters = new HashMap<String, String>();  
characters.put("hobbits", "Frodo");  
characters.put("hobbits", "Sam");  
characters.put("elves", "Galadriel");  
characters.put("elves", "Arwen");
characters.put("humans", "Éowyn");  
characters.put("humans", "Faramir");

for (HashMap.Entry<String, String> character : characters.entrySet()) {  
    ProducerRecord<String, String> producerRecord = new ProducerRecord<>("lotr_characters", character.getKey(), character.getValue());  

    producer.send(producerRecord, (RecordMetadata recordMetadata, Exception err) -> {  
        if (err == null) {  
            log.info("Message received. \n" +  
                    "topic [" + recordMetadata.topic() + "]\n" +  
                    "partition [" + recordMetadata.partition() + "]\n" +  
                    "offset [" + recordMetadata.offset() + "]\n" +  
                    "timestamp [" + recordMetadata.timestamp() + "]");  
        } else {  
            log.error("An error occurred while producing messages", err);  
        }  
    });  
}

Here, we're iterating over the collection, creating a ProducerRecord for each entry, and passing the record to send(). Behind the scenes, Kafka will batch these messages together to make fewer network requests. send() can also take a callback as a second argument. We're going to pass it a lambda which will run code when the send() request completes.

If the request completed successfully, we get back a RecordMetadata object with metadata about the message, which we can use to see things such as the partition and offset the message ended up in.

If we get back an exception, we could handle it by retrying to send the message, or alerting our application. In this case, we're just going to log the exception.

Run the main() method of the Producer class and you should see the message metadata get logged.

The full code for the Producer class should now be:

package org.example;  

import org.apache.kafka.clients.producer.KafkaProducer;  
import org.apache.kafka.clients.producer.ProducerConfig;  
import org.apache.kafka.clients.producer.ProducerRecord;  
import org.apache.kafka.clients.producer.RecordMetadata;  
import org.apache.kafka.common.serialization.StringSerializer;  
import org.slf4j.Logger;  
import org.slf4j.LoggerFactory;  

import java.util.HashMap;  
import java.util.Properties;  

public class Producer {  
    private static final Logger log = LoggerFactory.getLogger(Producer.class);  

    public static void main(String[] args) {  
        log.info("This class produces messages to Kafka");  

        Properties properties = new Properties();
        properties.setProperty(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092"); 
        properties.setProperty(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());  
        properties.setProperty(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());  

        KafkaProducer<String, String> producer = new KafkaProducer<>(properties);  

        HashMap<String, String> characters = new HashMap<String, String>();  
        characters.put("hobbits", "Frodo");  
        characters.put("hobbits", "Sam");  
        characters.put("elves", "Galadriel");  
        characters.put("elves", "Arwen");
        characters.put("humans", "Éowyn");  
        characters.put("humans", "Faramir"); 

        for (HashMap.Entry<String, String> character : characters.entrySet()) {  
            ProducerRecord<String, String> producerRecord = new ProducerRecord<>("lotr_characters", character.getKey(), character.getValue());  

            producer.send(producerRecord, (RecordMetadata recordMetadata, Exception err) -> {  
                if (err == null) {  
                    log.info("Message received. \n" +  
                            "topic [" + recordMetadata.topic() + "]\n" +  
                            "partition [" + recordMetadata.partition() + "]\n" +  
                            "offset [" + recordMetadata.offset() + "]\n" +  
                            "timestamp [" + recordMetadata.timestamp() + "]");  
                } else {  
                    log.error("An error occurred while producing messages", err);  
                }  
            });  
        }
        producer.close();  
    }  
}

Next, we're going to create a consumer to read these messages from Kafka.

How to Create a Kafka Consumer

First, create a Consumer class. Again, you can call it whatever you want, but don't call it KafkaConsumer as you will need that class in a moment.

All the Kafka-specific code will go in Consumer's main() method.

package org.example;  

import org.slf4j.Logger;  
import org.slf4j.LoggerFactory;  

public class Consumer {  
    private static final Logger log = LoggerFactory.getLogger(Consumer.class);  

    public static void main(String[] args) {  
        log.info("This class consumes messages from Kafka");  
    }  
}

Next, configure the consumer properties.

Properties properties = new Properties();  
properties.setProperty(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");  
properties.setProperty(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());  
properties.setProperty(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());  
properties.setProperty(ConsumerConfig.GROUP_ID_CONFIG, "lotr_consumer_group");  
properties.setProperty(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");

Just like with Producer, these properties are a set of string pairs. The ones we're using are:

• ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG which specifies the IP address to use to access the Kafka cluster

• ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG which specifies the deserializer to use for message keys

• ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG which specifies the deserializer to use for message values

• ConsumerConfig.GROUP_ID_CONFIG which specifies the consumer group this consumer belongs to

• ConsumerConfig.AUTO_OFFSET_RESET_CONFIG which specifies the offset to start reading from

We're connecting to the Kafka cluster on localhost:9092, using string deserializers since our keys and values are strings, setting a group id for our consumer, and telling the consumer to read from the start of the topic.

Note: If you're running the consumer on Windows and accessing a Kafka broker running on WSL, you'll need to change "localhost:9091" to "[::1]:9092" or "0:0:0:0:0:0:0:1:9092", like you did in Producer.

Next, we create a KafkaConsumer and pass it the configuration properties.

KafkaConsumer<String, String> consumer = new KafkaConsumer<>(properties);

We need to tell the consumer which topic, or topics, to subscribe to. The subscribe() method takes in a collection of one or more strings, naming the topics you want to read from. Remember, consumers can subscribe to more than one topic at the same time. For this example, we'll use one topic, the lotr_characters topic.

String topic = "lotr_characters";  

consumer.subscribe(Arrays.asList(topic));

The consumer is now ready to start reading messages from the topic. It does this by regularly polling for new messages.

We'll use a while loop to repeatedly call the poll() method to check for new messages.

poll() takes in a duration for how long it should read for at a time. It then batches these messages into an iterable called ConsumerRecords. We can then iterate over ConsumerRecords and do something with each individual ConsumerRecord.

In a real-world application, we would process this data or send it to some further destination, like a database or data pipeline. Here, we're just going to log the key, value, partition, and offset for each message we receive.

while(true){  
    ConsumerRecords<String, String> messages = consumer.poll(Duration.ofMillis(100));  

    for (ConsumerRecord<String, String> message : messages){  
        log.info("key [" + message.key() + "] value [" + message.value() +"]");  
        log.info("partition [" + message.partition() + "] offset [" + message.offset() + "]");  
    }  
}

Now it's time to run our consumer. Make sure you have Zookeeper and Kafka running. Run the Consumer class and you'll see the messages that Producer previously sent to the lotr_characters topic in Kafka.

The Kafka client app consuming messages that were previously produced to Kafka

How to Shut Down the Consumer

Right now, our consumer is running in an infinite loop and polling for new messages every 100 ms. This isn't a problem, but we should add safeguards to handle shutting down the consumer if an exception occurs.

We're going to wrap our code in a try-catch-finally block. If an exception occurs, we can handle it in the catch block.

The finally block will then call the consumer's close() method. This will close the socket the consumer is using, commit the offsets it has processed, and trigger a consumer group rebalance so any other consumers in the group can take over reading the partitions this consumer was handling.

try {
            // subscribe to topic(s)
            String topic = "lotr_characters";
            consumer.subscribe(Arrays.asList(topic));

            while (true) {
                // poll for new messages
                ConsumerRecords<String, String> messages = consumer.poll(Duration.ofMillis(100));

                // handle message contents
                for (ConsumerRecord<String, String> message : messages) {
                    log.info("key [" + message.key() + "] value [" + message.value() + "]");
                    log.info("partition [" + message.partition() + "] offset [" + message.offset() + "]");
                }
            }
        } catch (Exception err) {
            // catch and handle exceptions
            log.error("Error: ", err);
        } finally {
            // close consumer and commit offsets
            consumer.close();
            log.info("consumer is now closed");
        }

Consumer will continuously poll its assigned topics for new messages and shut down safely if it experiences an exception.

The full code for the Consumer class should now be:

package org.example;

import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.common.serialization.StringDeserializer;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.time.Duration;
import java.util.Arrays;
import java.util.Properties;

public class Consumer {
    private static final Logger log = LoggerFactory.getLogger(Consumer.class);

    public static void main(String[] args) {
        log.info("This class consumes messages from Kafka");

        Properties properties = new Properties();
        properties.setProperty(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        properties.setProperty(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
        properties.setProperty(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
        properties.setProperty(ConsumerConfig.GROUP_ID_CONFIG, "lotr_consumer_group");
        properties.setProperty(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");

        KafkaConsumer<String, String> consumer = new KafkaConsumer<>(properties);

        try {
            String topic = "lotr_characters";
            consumer.subscribe(Arrays.asList(topic));

            while (true) {
                ConsumerRecords<String, String> messages = consumer.poll(Duration.ofMillis(100));

                for (ConsumerRecord<String, String> message : messages) {
                    log.info("key [" + message.key() + "] value [" + message.value() + "]");
                    log.info("partition [" + message.partition() + "] offset [" + message.offset() + "]");
                }
            }
        } catch (Exception err) {
            log.error("Error: ", err);
        } finally {
            consumer.close();
            log.info("The consumer is now closed");
        }
    }
}

You now have a basic Java application that can send messages to and read messages from Kafka. If you got stuck at any point, the full code is available on GitHub.

Where to Take it from Here

Congratulations on making it this far. You've learned:

• the main concepts behind Kafka

• how to communicate with Kafka from the command line

• how to build a Java app that produces to and consumes from Kafka

There's plenty more to learn about Kafka, whether that's Kafka Connect for connecting Kafka to common data systems or the Kafka Streams API for processing and transforming your data.

Some resources you might find useful as you continue your journey with Kafka are:

• the official Kafka docs

• courses from Confluent

• Conduktor's kafkademy

I hope this guide has been helpful and made you excited to learn more about Kafka, event streaming, and real-time data processing.

As a result, a lot of modern sites and apps use the same format to display large numbers in a compact way. For example, displaying 123,000 as 123K.

freeCodeCamp's YouTube and Instagram profiles using compact number format.

You can do this by writing a custom format function, using a third-party library, or, best of all, using a built-in JavaScript API.

You can of course write your own formatter function (and there are several available on Stack Overflow) but you will end up having to check for a lot of conditions.

As an aside, since ES2021, JavaScript supports using underscores as numeric separators to make large numbers easier to read in your code.

function formatCompactNumber(number) {
  if (number < 1000) {
    return number;
  } else if (number >= 1000 && number < 1_000_000) {
    return (number / 1000).toFixed(1) + "K";
  } else if (number >= 1_000_000 && number < 1_000_000_000) {
    return (number / 1_000_000).toFixed(1) + "M";
  } else if (number >= 1_000_000_000 && number < 1_000_000_000_000) {
    return (number / 1_000_000_000).toFixed(1) + "B";
  } else if (number >= 1_000_000_000_000 && number < 1_000_000_000_000_000) {
    return (number / 1_000_000_000_000).toFixed(1) + "T";
  }
}

formatCompactNumber(12_000);        // 12.0K
formatCompactNumber(2_000_000);     // 2.0M
formatCompactNumber(2_500_000);     // 2.5M
formatCompactNumber(6_000_000_000); // 6.0B
formatCompactNumber(6_900_000_000); // 6.9B

This implementation still leaves a .0 after an even thousand, million, billion, or trillion. You could fix this using the replace method and a regular expression.

function formatCompactNumber(number) {
  if (number < 1000) {
    return number;
  } else if (number >= 1000 && number < 1_000_000) {
    return (number / 1000).toFixed(1).replace(/\.0$/, "") + "K";
  } else if (number >= 1_000_000 && number < 1_000_000_000) {
    return (number / 1_000_000).toFixed(1).replace(/\.0$/, "") + "M";
  } else if (number >= 1_000_000_000 && number < 1_000_000_000_000) {
    return (number / 1_000_000_000).toFixed(1).replace(/\.0$/, "") + "B";
  } else if (number >= 1_000_000_000_000 && number < 1_000_000_000_000_000) {
    return (number / 1_000_000_000_000).toFixed(1).replace(/\.0$/, "") + "T";
  }
}

But what if you need to handle negative numbers? You could add another conditional.

function formatCompactNumber(number) {
  if (number < 0) {
    return "-" + formatCompactNumber(-1 * number);
  }
  if (number < 1000) {
    return number;
  } else if (number >= 1000 && number < 1_000_000) {
    return (number / 1000).toFixed(1).replace(/\.0$/, "") + "K";
  } else if (number >= 1_000_000 && number < 1_000_000_000) {
    return (number / 1_000_000).toFixed(1).replace(/\.0$/, "") + "M";
  } else if (number >= 1_000_000_000 && number < 1_000_000_000_000) {
    return (number / 1_000_000_000).toFixed(1).replace(/\.0$/, "") + "B";
  } else if (number >= 1_000_000_000_000 && number < 1_000_000_000_000_000) {
    return (number / 1_000_000_000_000).toFixed(1).replace(/\.0$/, "") + "T";
  }
}

As you can probably see by now, this only scratches the surface of things you would need to consider when writing your own function for displaying compact numbers.

There are a handful of npm packages for formatting numbers compactly. For example, you could install cldr-compact-number, but this would also add 3 kilobytes (or 1.2 kilobytes gzipped) to your JavaScript bundle, adding slightly to your pageload time.

Thankfully, you don't need to use any third-party libraries to format compact numbers, since there is a relatively simple solution that is natively supported in JavaScript.

How to Use the JavaScript Internationalization API

The JavaScript Internationalization API helps you to support different languages and formatting conventions when working in JavaScript. This could be anything from formatting dates and times to knowing whether a or ä comes first when comparing strings.

The API has excellent browser support, with 98% support worldwide. It works by using the Intl object to create a namespace for language-sensitive string comparison, number formatting, and date and time formatting.

As well as Intl.DateTimeFormat for formatting dates and times, and Intl.Collator for doing language-sensitive string comparison, there is Intl.NumberFormat. This lets you format numbers in a language-specific manner.

To get started, create a formatter using the Intl.NumberFormat constructor. Optionally, you can pass it one or more locales and an options object.

new Intl.NumberFormat(locales, options);

A locale is a parameter that defines the user's language, region, or other localization preference. In this case it's an IETF language tag, such as "en-US" for US English, "zh-CH" for Mandarin, or "uk" for Ukrainian.

In the browser, you can access the user's locale using navigator.language. If you don't provide a locale, the API will attempt to use the user's locale from their browser.

The options object can contain values to control things like how to format currency, whether to use l or liters, or whether to display + or - for positive or negative values.

Once you have created a formatter, you can call its format method and pass it the number you want to format. It will return the number formatted according to whatever configuration you provided.

For example, you could use the Internationalization API to format financial values according to different currencies:

const number = 12345678.99

const germanCurrencyFormatter = new Intl.NumberFormat("de-DE", { style: "currency", currency: "EUR" });

const chineseCurrencyFormatter = new Intl.NumberFormat("zh-CH", { style: "currency", currency: "CNY" });

germanCurrencyFormatter.format(number); // 12.345.678,99 €
chineseCurrencyFormatter.format(number); // ¥12,345,678.99

One of the options that is available is notation. This controls the formatting when displaying numbers. The possible values for notation are:

• "standard" – plain number formatting according to the conventions of the locale (the default)

• "scientific" – returns the order of magnitude for a number

• "engineering" – returns the exponent of ten when the number is divisible by three

• "compact" – returns a string representing the exponent, such as K for thousands

So, for example, if you format the number 123456789 with the locale set to "en", you will get:

• standard: 123,456,789

• scientific: 1.235E8

• engineering: 123.457E6

• compact: 123M

If you want a formatter to display large numbers in a compact manner, set the locale to your desired locale and set notation to "compact" .

const formatter = Intl.NumberFormat("en", { notation: "compact" });

This can be used to create a much shorter formatting function:

function formatCompactNumber(number) {
  const formatter = Intl.NumberFormat("en", { notation: "compact" });
  return formatter.format(number);
}

formatCompactNumber(-57);               // -57
formatCompactNumber(999);               // 999
formatCompactNumber(8_554);             // 8.5K
formatCompactNumber(150_000);           // 150K
formatCompactNumber(3_237_512);         // 3.2M
formatCompactNumber(9_782_716_897);     // 9.8B
formatCompactNumber(7_899_693_036_970); // 7.9T

Now your site or app can display even the largest numbers compactly, making your layout that little bit neater and tidier.

Thanks for reading!

I hope this quick guide helps you when you're working with large numbers in JavaScript and perhaps encourages you to explore the Internationalization API further.

Eleventy (also called 11ty) is a simple yet powerful static site generator. It uses JavaScript to transform data and templates into HTML pages.

It’s beginner-friendly, has fast build times, and generates fast sites by default. It also has a very active and friendly community.

Eleventy excels at content-driven sites and is used by Google, Netlify, MIT, CERN, the A11y Project, ESLint, and more.

Since pages are generated ahead of time, they can be served as fast as possible from a Content Delivery Network (CDN). Eleventy also generates no client-side JavaScript, which helps your site to load faster.

In this tutorial, we're going to build a simple developer portfolio site to demonstrate some of the main features of Eleventy.

Eleventy portfolio homepage

We’ll learn about:

1. Setting up and configuring an Eleventy project

2. Templating and layouts

3. Handling CSS and images

4. Working with collections and data files

5. Shortcodes and Eleventy plugins

6. Deploying the site to Netlify

The portfolio site will have:

• A Homepage

• An About page

• A Contact page (with contact form)

• A Projects page

• A page for each project (with case study)

Eleventy can pull in data from APIs, a Content Management System (CMS), or from local files. To keep things simple, we’ll store our project data in Markdown files.

The full code for the finished portfolio is available on GitHub. If you get stuck at any stage, please check your code against the finished site.

Table of Contents:

1. Prerequisite - Install Node.js

2. Initial Project Setup

3. How to Configure the Project

4. How to Add a Template

5. How to Use Templates in Eleventy

6. How to Use Layouts in Eleventy

7. How to Configure the CSS and Images

8. How to Use Partials in Eleventy

9. How to Use Collections in Eleventy

10. How to Use Directory Data Files

11. How to Use Collections in Templates

12. How to Use Shortcodes

13. How to Use the Eleventy Image Plugin

14. How to Build a Contact Form with Netlify Forms

15. How to Deploy to Netlify

16. Where to Take it from Here

Prerequisite - Install Node.js

If you don’t already have Node.js installed, go to nodejs.org and follow the instructions for your operating system.

Open a terminal and use node --version to make sure it’s installed. As long as it’s version 12 or newer, you’re good to go.

Initial Project Setup

First, create a directory for your portfolio. You can call it eleventy-portfolio or whatever you want.

Open this directory in a terminal and run npm init -y to create a package.json file with the default settings.

Next, install Eleventy using npm install --save-dev @11ty/eleventy.

In the root directory of the project, create a .gitignore file with the following contents so that Git doesn’t track any unwanted files:

node_modules
/public

How to Configure the Project

Eleventy is “zero-config” by default. If you don’t change anything, Eleventy will take all the files in your root directory, run a build process, and output the resulting files to a _site directory.

But Eleventy also has flexible configuration options that let you customize your build process, watch for changes in certain file types, and manipulate content with filters and shortcodes.

Your Eleventy configuration goes in a .eleventy.js file in the root of your project.

For example, the default input directory is the root directory of your project, while the default output directory is _site. Some people prefer to change this, with src and public being common choices.

If you’d like this structure, create src and public directories in the root of your project and then set them as the input and output directories in .eleventy.js.

module.exports = function (eleventyConfig) {
  return {
    dir: {
      input: "src",
      output: "public"
    }
  };
};

In case you’re wondering, the eleventyConfig argument that is passed to the function is the default configuration object that Eleventy provides. Soon we’ll use this object to customize our Eleventy build process.

How to Add a Template

Let’s add our first template. We’ll keep things as simple as possible by using a Markdown file.

In the src directory, create an index.md with # Hello World from Eleventy as its contents. This is your first Eleventy template.

To build and view the site we can use the development server that comes with Eleventy.

In your terminal, make sure you’re in the root directory of your project and run eleventy --serve. This starts the development server, which will watch your src directory and automatically reload your site whenever you change your code.

After a moment you’ll see:

[Browsersync] Access URLs:
 ----------------------------------
    Local: http://localhost:8080
 External: http://your_ip_address:8080
 ----------------------------------
[Browsersync] Serving files from: public

Open a web browser and go to http://localhost:8080. Congratulations, you’ve made a (very simple) Eleventy site! 🥳🎉

At this stage, your project will have the following structure:

node_modules/
public/
src/
.eleventy.js
.gitignore
package.lock.json
package.json

Most sites need more than one page, so we’re going to need to learn more about templates.

Before we do that, we can customize our build commands if we want to. This step is entirely optional.

Optional Step – How to Create Custom Build Commands

The default command to run the development server is eleventy --serve, while the default command to build the site is eleventy.

If you’d like to replace these with different commands, such as start and build, open package.json and under scripts replace the “test” command with your preferred commands:

"scripts": {
    "start": "eleventy --serve",
    "build": "eleventy"
  },

Now we can use npm start in the terminal to start the development server and npm run build to generate a build of our site.

You can use ctrl/cmd + c to stop the development server whenever you need to.

How to Use Templates in Eleventy

Turning Markdown files into HTML is neat, but so far you aren’t really getting much benefit over just writing your site in plain HTML. This is where templates come in.

First, we need to clarify some terms:

• Template – A content file that Eleventy will transform into a page, or pages, in the built site

• Layout – A template that wraps around another template, usually to provide a structure to present content in

• Partial – A template that makes up part of another template

Templates let you combine content and data to generate whatever HTML your site needs.

Layouts let you give multiple templates the same basic structure.

Partials let you build small reusable components that you can use in larger templates.

Eleventy supports ten different languages for templating, including: HTML, Markdown, JavaScript, Liquid, Nunjucks, Handlebars, Mustache, EJS, Haml, and Pug. (In version 1.0 Eleventy added support for custom templates using any arbitrary file extension, but this is probably better reserved for more custom/advanced use cases).

You can even mix different templating languages in the same file, such as Markdown and Nunjucks, if you want to.

In this project, we’ll use Nunjucks. It’s a templating language for JavaScript created by Mozilla, and is pretty popular in the Eleventy community.

In the src directory, delete index.md and create an index.njk file. If you’re using VS Code, type ! + tab to generate the basic HTML structure for the page. Change the title to "Eleventy Portfolio" and in the <body> element, add <h1>Home Page</h1>.

Your page should look like this:

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <meta http-equiv="X-UA-Compatible" content="IE=edge">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <title>Eleventy Portfolio</title>
</head>
<body>
  <h1>Home Page</h1>
</body>
</html>

Next, still in src, create about.njk and contact.njk files. You can copy the contents of index.njk into them and replace the <h1> with <h1>About Page</h1> and <h1>Contact Page</h1> respectively.

Start your dev server. if it isn’t already running. Go to http://localhost:8080 to see the homepage, http://localhost:8080/about for the About page, and http://localhost:8080/contact for the Contact page.

In our portfolio site, each of these pages is going to have the same basic layout. So instead of writing the same code in each page template, we’re going to use Eleventy layouts.

How to Use Layouts in Eleventy

Layouts are templates that wrap around other templates, presenting the content in a consistent way.

Inside the src directory, create a _includes directory. This is going to contain all our layouts and partials.

Inside _includes, create a base.njk file. This will provide a standard layout for every page of our site.

Copy the following code into base.njk:

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <meta http-equiv="X-UA-Compatible" content="IE=edge">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <meta name="description" content="I'm a Frontend software developer who builds sites and apps that help people reach their personal and professional goals."/>
  <title>{{ title }}</title>
</head>
<body>
  <div class="content">
    <header class="header container">
    <h1 class="header__title">
      <a href="/">Marie Jackson</a>
    </h1>
    <ul class="header__links">
      <li>
        <a class="header__link" href="/about">About</a>
      </li>
      <li>
        <a class="header__link" href="/projects">Projects</a>
      </li>
      <li>
        <a class="header__link" href="/contact">Contact</a>
      </li>
    </ul>
    </header>
    <main class="main container">
      {{ content | safe }}
    </main>
  </div>
  <footer class="footer">
      <p>&copy; Marie Jackson 2022</p>
  </footer>
</body>
</html>

The content value will be the main content of whichever template we use with base.njk as its layout. safe is a filter that prevents this content from being escaped (having potentially unsafe characters replaced).

Now, change index.njk to be:

---
title: "Eleventy Portfolio"
layout: "base.njk"
---

<h1>{{ title }} Home Page</h1>

Notice how the template has frontmatter data at the top of the file. By default this is written in YAML, but you can use other languages too.

This frontmatter lets you set values for your templates. In this case, the layout value tells the template to use the base.njk layout and the title value provides a title that we are using in our template's <h1> tag.

Next, delete everything from about.njk and paste in the following content:

---
title: "Eleventy Portfolio"
layout: "base.njk"
---

<section class="bio prose">
  <h2 class="heading--main">My story</h2>
  <p>Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Aliquet risus feugiat in ante metus dictum.</p>

  <p>Tellus pellentesque eu tincidunt tortor aliquam nulla facilisi cras fermentum. Turpis egestas integer eget aliquet. Vestibulum morbi blandit cursus risus at ultrices mi tempus. Ut lectus arcu bibendum at. Integer enim neque volutpat ac tincidunt.</p>

  <p>Commodo ullamcorper a lacus vestibulum sed arcu. Et tortor consequat id porta nibh venenatis cras sed. Nulla pharetra diam sit amet nisl. Ipsum nunc aliquet bibendum enim facilisis gravida neque convallis a. Nec sagittis aliquam malesuada bibendum.</p>

  <p>Tellus pellentesque eu tincidunt tortor aliquam nulla facilisi cras fermentum. Turpis egestas integer eget aliquet. Vestibulum morbi blandit cursus risus at ultrices mi tempus. Ut lectus arcu bibendum at. Integer enim neque volutpat ac tincidunt.</p>

  <p>Commodo ullamcorper a lacus vestibulum sed arcu. Et tortor consequat id porta nibh venenatis cras sed. Nulla pharetra diam sit amet nisl. Ipsum nunc aliquet bibendum enim facilisis gravida neque convallis a. Nec sagittis aliquam malesuada bibendum.</p>
</section>

Now delete everything from contact.njk and paste in this content:

---
title: "Eleventy Portfolio"
layout: "base.njk"
---

<h2 class="heading--main text-center">Want to get in touch?</h2>
<p class="contact__sub-heading text-center">I'm always open to new opportunities and projects. </p>

<form class="form" name="contact" action="/success" method="POST" data-netlify="true">
  <div class="form__section">
    <label class="form__label" for="yourName">Name</label>
    <input class="form__input" name="name" type="text" id="yourName" required="true">
  </div>
  <div class="form__section">
    <label class="form__label" for="yourEmail">Email</label>
    <input class="form__input" name="email" type="email"  id="yourEmail" required="true">
  </div>
  <div class="form__section">
    <label class="form__label" for="message">Message</label>
    <textarea class="form__input" name="message" id="message" rows="4" required="true"></textarea>
  </div>
    <button class="form__button" type="submit">Let's talk</button>
</form>

We’re going to learn about how this contact form will work later in the tutorial.

Our portfolio is starting to take shape, even though things still look really bare. Let's get our CSS and images working next.

How to Configure the CSS and Images

While Eleventy can understand supported templating languages out of the box, it needs to be configured to process CSS and image files. Fortunately, this doesn’t require much configuration. While we’re at it, we’re also going to add a favicon to the site.

Inside the src directory, create three folders: css, images, and favicons.

src directory structure

Inside the css directory, make a style.css file. Since this isn’t a CSS tutorial, I’m going to provide the CSS in the GitHub repo for the project. You can copy and paste it from there, but I’m not going to cover the CSS in any depth.

The images for this portfolio are also available in the images directory in the GitHub repo. Copy these images into the images directory of your project.

Finally, copy the files from the favicons directory in the GitHub repo into the favicons directory of your project.

In base.njk add these lines to the <head> element:

<link rel="icon" href="/favicon.ico" sizes="any">
<link rel="apple-touch-icon" href="/apple-touch-icon.png">
<link rel="preconnect" href="https://fonts.googleapis.com">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link href="https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600&display=swap" rel="stylesheet">
<link rel="stylesheet" href="{{'/css/style.css' | url | safe}}">

The first two links include the favicon, the next three will fetch the Inter font from Google fonts, and the last one will connect style.css to base.njk.

Now, if we reload our homepage we’ll see that… absolutely nothing has changed.

By default, Eleventy will only process template files. To tell it to include CSS files and images, we need to add a few lines of configuration.

Add these lines to the configuration function in .eleventy.js:

module.exports = function (eleventyConfig) {
  eleventyConfig.addPassthroughCopy("./src/css/");
  eleventyConfig.addWatchTarget("./src/css/");
  eleventyConfig.addPassthroughCopy("./src/images/");
  eleventyConfig.addPassthroughCopy({ "./src/favicons": "/" });

  return {
    dir: {
      input: "src",
      output: "public"
    }
  };
};

addPassthroughCopy tells Eleventy to pass the CSS, favicons, and image files through to the final build.

addWatchTarget tells the Eleventy dev server to watch the css directory and reload the site if the files in this directory change.

With the favicons, we’re also telling Eleventy to output these files to the root of the generated content so that the links in base.njk will work.

Restart the server and you’ll see that the CSS is finally being applied and the favicon is showing up. We'll include the images shortly.

Homepage with styles applied

About page with styles applied

Contact page with styles applied

The About page and Contact page are fairly self-contained. But the homepage for our site is going to have a couple of parts to it. It will consist of a header and footer, as well as a profile section, technologies section, and projects section. Each of these parts is going to use a partial.

Layout of portfolio homepage

How to Use Partials in Eleventy

Partials are templates that make up part of another template. Partials help us to think about our site in terms of reusable components that we can include whenever we need them.

In the _includes directory, create a header.njk and footer.njk file.

Cut the header element out of base.njk and paste it into header.njk.

Cut the footer element out of base.njk and paste it into footer.njk.

Back in base.njk add {% include "header.njk" %} where the header element used to be and {% include "footer.njk" %} where the footer element used to be.

base.njk should now have this content inside its <body> tag:

<div class="content">
    {% include "header.njk" %}
    <main class="main container">
      {{ content | safe }}
    </main>
</div>
{% include "footer.njk" %}

The site won’t look any different but our base layout is already becoming more modular.

Next, still in the _includes directory, create a profile.njk file with the following content:

<section class="profile">
  <div class="profile__image-wrapper">
    <img class="profile__image" src="/images/profile.jpg" alt="Marie Jackson, Software Developer">
  </div>
  <div class="profile__card">
    <p class="profile__text">Hi! I'm <span class="profile__text--highlight">Marie</span>, a mathematician turned software developer from Hampton, Virginia.</p>
    <p class="profile__text">As a <span class="profile__text--highlight">Frontend Developer</span>, I love building sites and apps that help people reach their personal and professional goals.</p>
    <p class="profile__text">I focus on speed, security and scalability, using React.js and Firebase to create rich, dynamic experiences.</p>
    <p class="profile__text">I'm always open to new opportunities and projects. So don't hesitate to <a class="profile__link" href="/contact">get in touch</a>.<p>
  </div>
</section>

Next, create a technologies.njk file with this content:

<section class="technologies">
  <h2 class="technologies__heading">Technologies I love to work with</h2>
  <ul class="technologies__list">
    <li class="technologies__item">
      <div class="technologies__logo">
      <img src="/images/javascript.svg" alt="JavaScript logo">
      </div>
      <h3 class="technologies__title">JavaScript</h3>
    </li>
    <li class="technologies__item">
      <div class="technologies__logo">
      <img src="/images/react.svg" alt="React.js logo">
      </div>
      <h3 class="technologies__title">React.js</h3>
    </li>
    <li class="technologies__item">
      <div class="technologies__logo">
        <img src="/images/tailwindcss.svg" alt="Tailwind CSS logo">
      </div>
      <h3 class="technologies__title">Tailwind CSS</h3>
    </li>
    <li class="technologies__item">
      <div class="technologies__logo">
        <img src="/images/firebase.svg" alt="Firebase logo">
      </div>
      <h3 class="technologies__title">Firebase</h3>
    </li>
  </ul>
</section>

In index.njk replace the <h1> tag with:

{% include "profile.njk" %}
{% include "technologies.njk" %}

Portfolio homepage with profile and technologies sections

Our homepage is starting to take shape but the site still needs the most important part of any portfolio: projects.

To keep the project data organized, we’re going to use collections.

How to Use Collections in Eleventy

Collections let you group related content together. In our portfolio we’re going to create a projects collection using Markdown files to store information about each individual project.

Inside the src directory, create a projects directory. We’re going to need a Markdown file for each project. As placeholders, we’ll use three projects that I’ve been meaning to build.

Josh W Comeau has some great advice on building an effective developer portfolio and he strongly recommends describing your personal projects with detailed case studies. So for each of our projects we’re going to have a case study laying out:

• What problem we solved

• Why we chose these specific technologies

• What challenges we faced

• What lessons we learned

Copy the following three sample projects into the projects directory:

catch-up.md

---
title: "Catch Up"
summary: "Sometimes it's hard to keep in touch with friends and family. I made this app to remind me to schedule a call if we haven't talked in a while."
image: /images/catch-up.jpg
imageAlt: "Screenshots of catch up app"
tech:
  - "Next.js"
  - "Firebase"
  - "Tailwind CSS"
siteUrl: "#"
repoUrl: "#"
---

### Problem Solved

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Tincidunt tortor aliquam nulla facilisi. Feugiat scelerisque varius morbi enim nunc faucibus a pellentesque sit. Condimentum lacinia quis vel eros donec ac odio tempor orci.

### Technologies Used

Scelerisque eleifend donec pretium vulputate sapien nec sagittis aliquam. Diam sit amet nisl suscipit adipiscing bibendum est ultricies. Consequat ac felis donec et odio pellentesque diam volutpat commodo.

### Challenges Faced

Eget mauris pharetra et ultrices. Molestie nunc non blandit massa enim nec. Ut tortor pretium viverra suspendisse potenti nullam ac tortor vitae. Nulla at volutpat diam ut venenatis. Volutpat ac tincidunt vitae semper quis lectus nulla at.

### Lessons Learned

Non blandit massa enim nec. Tempor commodo ullamcorper a lacus vestibulum sed. Et netus et malesuada fames ac turpis egestas integer eget. In ante metus dictum at tempor commodo. Eu scelerisque felis imperdiet proin fermentum leo.

sourdough-sensei.md

---
title: "Sourdough Sensei"
summary: "Like a lot of people, I got really into sourdough in 2020. I made this app to help me bake delicious bread by putting all my recipes and schedules in one place."
image: /images/sourdough-sensei.jpg
imageAlt: "Screenshots of sourdough bread app"
tech:
  - "React.js"
  - "Firebase"
  - "Tailwind CSS"
siteUrl: "#"
repoUrl: "#"
---

### Problem Solved

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Tincidunt tortor aliquam nulla facilisi. Feugiat scelerisque varius morbi enim nunc faucibus a pellentesque sit. Condimentum lacinia quis vel eros donec ac odio tempor orci.

### Technologies Used

Scelerisque eleifend donec pretium vulputate sapien nec sagittis aliquam. Diam sit amet nisl suscipit adipiscing bibendum est ultricies. Consequat ac felis donec et odio pellentesque diam volutpat commodo.

### Challenges Faced

Eget mauris pharetra et ultrices. Molestie nunc non blandit massa enim nec. Ut tortor pretium viverra suspendisse potenti nullam ac tortor vitae. Nulla at volutpat diam ut venenatis. Volutpat ac tincidunt vitae semper quis lectus nulla at.

### Lessons Learned

Non blandit massa enim nec. Tempor commodo ullamcorper a lacus vestibulum sed. Et netus et malesuada fames ac turpis egestas integer eget. In ante metus dictum at tempor commodo. Eu scelerisque felis imperdiet proin fermentum leo.

spellbook.md

---
title: "Spellbook"
summary: "I'm a huge Dungeons and Dragons fan, but keeping my spells straight has always been a challenge. I built this app to put all the information I need at my fingertips."
image: /images/spellbook.jpg
imageAlt: "Screenshots of DnD project"
tech:
  - "Next.js"
  - "Firebase"
  - "Tailwind CSS"
siteUrl: "#"
repoUrl: "#"
---

### Problem Solved

Yes, I could have just used DnD Beyond. But where's the fun in that? Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Tincidunt tortor aliquam nulla facilisi. Feugiat scelerisque varius morbi enim nunc faucibus a pellentesque sit. Condimentum lacinia quis vel eros donec ac odio tempor orci.

### Technologies Used

Scelerisque eleifend donec pretium vulputate sapien nec sagittis aliquam. Diam sit amet nisl suscipit adipiscing bibendum est ultricies. Consequat ac felis donec et odio pellentesque diam volutpat commodo.

### Challenges Faced

Eget mauris pharetra et ultrices. Molestie nunc non blandit massa enim nec. Ut tortor pretium viverra suspendisse potenti nullam ac tortor vitae. Nulla at volutpat diam ut venenatis. Volutpat ac tincidunt vitae semper quis lectus nulla at.

### Lessons Learned

Non blandit massa enim nec. Tempor commodo ullamcorper a lacus vestibulum sed. Et netus et malesuada fames ac turpis egestas integer eget. In ante metus dictum at tempor commodo. Eu scelerisque felis imperdiet proin fermentum leo.

Just like with templates, the frontmatter at the top of these files makes values available that you can inject into your templates.

Since these Markdown files are ultimately inside the src directory, Eleventy will treat them as templates and create a HTML page from each file. Their URL will be in the format /subdirectory_name/filename, for example /projects/sourdough-sensei.

But Eleventy won't know what layout to use for these pages since they don't yet have a layout value in their frontmatter.

sourdough-sensei page without layout or frontmatter data

Right now these files aren’t a collection. Collections are defined by sharing a tags value, such as “tags”: “projects”.

Every file with the projects tag will be included in the projects collection.

Since we only have three projects, we could include a tags value in the frontmatter of our three Markdown files.

But if we had a site with a lot of content – for example dozens of blog posts, recorded talks, and tutorials that all shared dozens of tags between them – this could become difficult to manage. This is where directory data files are useful.

How to Use Directory Data Files

If you have certain values that are shared by every file in a folder, you can put these values in a directory data file.

Inside the projects directory, create a projects.json file. A directory data file should have the same name as the collection it’s attached to.

Any frontmatter fields that are shared by all the projects files should go in the projects.json directory data file:

{
  "layout": "project.njk",
  "tags": "projects"
}

The layout value means that every project will use the same layout (we’re going to create this project.njk file in a moment). The tags value is what turns them into the projects collection that we can use in our templates.

How to Use Collections in Templates

We’re now going to use the projects collection to:

• Add a projects section to our homepage

• Create a Projects page

• Create a case study page for each project

To include data from a collection on a page of your site, you need to reference the collections object in a template.

We can use Nunjucks to loop over the collection and output its contents. To access a frontmatter value from a project in the projects collection, we use project.data.

For example:

{% for project in collections.projects %}
{{ project.data.title }}
{% endfor %}

In the _includes directory, create project.njk, project-card.njk and project-grid.njk files.

_includes directory structure

We’ll use project.njk to create a page for each of our projects.

Since these pages are generated from templates, we can access their frontmatter values directly, such as title, image, imageAlt, and content for the main content of the Markdown file.

---
layout: "base.njk"
---

<div class="project">
  <h2 class="project__heading">{{ title }}</h2>
  <div class="project__image-wrapper">
      <img class="project__image" src="{{ image }}" alt="{{ imageAlt }}">
  </div>
  <div class="project__content prose">
    {{ content | safe }}
  </div>
</div>

project-grid.njk and project-card.njk will form the list of projects on our portfolio homepage and Projects page.

project-grid.njk will loop over the projects collection and insert a project-card partial for each project in the collection.

Add the following content to project-grid.njk:

<section class="projects">
  <h2 class="project__heading">Recent projects</h2>
  <div class="project-grid">
    {% for project in collections.projects %}
      {% include "project-card.njk" %}
    {% endfor %}
  </div>
</section>

Add the following content to project-card.njk:

<article class="project-card">
  <div class="project-card__image-wrapper">
    <img class="project__image" src="{{ project.data.image }}" alt="{{ project.data.imageAlt }}">
  </div>
  <div class="project-card__body">
    <div class="project-card__tags">
      {% for tag in project.data.tech %}
        <span class="project-card__tag">{{ tag }}</span>
      {% endfor %}
    </div>
    <h3 class="project-card__title">
      <a href="{{ project.url }}">{{ project.data.title }}</a>
    </h3>
    <p class="project-card__summary">{{ project.data.summary }}</p>
    <a class="project-card__link" href="{{ project.url }}">Read project case study 
      <svg xmlns="http://www.w3.org/2000/svg" class="project-card__link-icon" viewBox="0 0 20 20" fill="currentColor">
        <path fill-rule="evenodd" d="M7.293 14.707a1 1 0 010-1.414L10.586 10 7.293 6.707a1 1 0 011.414-1.414l4 4a1 1 0 010 1.414l-4 4a1 1 0 01-1.414 0z" clip-rule="evenodd" />
      </svg>
    </a>
  </div>
</article>

Since project-card.njk is accessing frontmatter values from a member of a collection, we need to use project.data to access these values in the template. Eleventy also generates a project.url value that we can use to link to the project's generated page.

In index.njk, add {% include "project-grid.njk" %} underneath the profile and technologies partials.

---
title: "Eleventy Portfolio"
layout: "base.njk"
---

{% include "profile.njk" %}
{% include "technologies.njk" %}
{% include "project-grid.njk" %}

Grid of project cards on homepage

Next we're going to create a Projects page. In the src directory, create a projects.njk file with the following contents:

---
title: "Eleventy Portfolio"
layout: "base.njk"
---

<h2 class="projects__heading">Recent projects</h2>
<div class="project-list">
  {% for project in collections.projects %}
    {% include "project-card.njk" %}
  {% endfor %}
</div>

Projects page

The projects are now being displayed on our homepage as well as the projects page, and each project now has its own page with its case study.

We could stop here, but there are a few more Eleventy features that will make our portfolio site even better, namely shortcodes and plugins.

How to Use Shortcodes

A shortcode is a way to inject reusable content (often a JavaScript string template literal) into your templates.

We’re going to create a simple year shortcode that outputs the current year so that the footer in our portfolio site is always up to date.

Add the following line to the configuration function in .eleventy.js.

eleventyConfig.addShortcode("year", () => `${new Date().getFullYear()}`);

When you use the shortcode in a template the function will be run and the year value will be injected into the template.

In footer.njk use {% year %} to access the year shortcode.

<footer class="footer">
  <p>&copy; Marie Jackson {% year %}</p>
</footer>

You might need to restart your development server for it to recognize the shortcode.

Now whenever you trigger a build of your site in the future, your footer will always show the correct year.

Shortcodes can do a lot more than this. Next, we’re going to use the Eleventy Image plugin, which uses shortcodes, to optimize our site’s images and improve our page load speed.

How to Use the Eleventy Image Plugin

Eleventy has a number of official plugins, from ones to check your writing for inclusive language to others that let you leverage Serverless functions.

The Image plugin is particularly useful since images are often the largest resource your site loads. It optimizes your images so your site uses the appropriate size and format for the user's browser, saving bandwidth for your user and making your site load faster.

First we need to install the Image plugin from npm. In the root of your project, run:

npm install @11ty/eleventy-img

At the top of .eleventy.js, we’re going to import the Image plugin and configure the shortcode that the plugin will use to optimize our images.

const Image = require("@11ty/eleventy-img");

async function imageShortcode(src, alt, sizes) {
  let metadata = await Image(`./src${src}`, {
    widths: [300, 800, null],
    formats: ["avif", "jpeg"],
    urlPath: "/images/",
    outputDir: "./public/images/"
  });

  let imageAttributes = {
    alt,
    sizes,
    loading: "lazy",
    decoding: "async"
  };

  return Image.generateHTML(metadata, imageAttributes);
}

The image shortcode takes in arguments for src, alt and sizes. These will be the URL for the image, the text for the image’s alt tag, and the sizes that are used to display different sized images at different screen sizes.

The widths property specifies what size images the plugin will generate. In this case, 300px, 800px, and the image's original size.

The formats property specifies which image formats to generate. Here we're using avif (which produces high quality images at low file sizes) with jpeg as a fallback for browsers that don't support avif.

urlPath and outputDir tell the plugin where to get the images from and where to output the optimized images to.

The plugin adds loading and decoding attributes to the generated HTML to lazy-load the images and decode them asynchronously, both of which will help with page load times.

Next, we’re going to include the shortcode in our configuration function. We'll call it EleventyImage for the sake of clarity.

eleventyConfig.addNunjucksAsyncShortcode("EleventyImage", imageShortcode);

Notice that we’re using addNunjucksAsyncShortcode rather than addShortcode. This is because the image generation process is asynchronous. It will take some amount of time to generate different image sizes and formats and we want our shortcode to wait until these have all been generated before it injects the finished HTML into our templates.

Since our shortcode is asynchronous, we’re going to run into an issue with using this shortcode inside a Nunjucks for loop. We need to use asyncEach, the asynchronous version of Nunjuck’s for.

In projects.njk and project-grid.njk, replace this:

{% for project in collections.projects %}
{% include "project-card.njk" %}
{% endfor %}

with this:

{% asyncEach project in collections.projects %}
{% include "project-card.njk" %}
{% endeach %}

Now, in project.njk we can replace this:

<img class="project__image" src="{{ image }}" alt="{{ imageAlt }}">

with this:

{% EleventyImage image, imageAlt, "(min-width: 30em) 50vw, 100vw" %}

The image, imageAlt and "(min-width: 30em) 50vw, 100vw" values are the src, alt and sizes parameters for the Image shortcode.

Next, in project-card.njk, we can replace this:

<img class="project-card__image" src="{{ project.data.image }}" alt="{{ project.data.imageAlt }}">

with this:

{% EleventyImage project.data.image, project.data.imageAlt, "(min-width: 30em) 50vw, 100vw" %}

Finally, in profile.njk we can replace this:

<img class="profile__image" src="/images/profile.jpg" alt="Marie Jackson, Software Developer">

with this:

{% EleventyImage "/images/profile.jpg", "Marie Jackson, Software Developer", "(min-width: 16em) 50vw, 100vw" %}

When our site builds, the Eleventy Image plugin will do a couple of things:

• there will be multiple formats and sizes for every image in public/images

• our generated HTML will now use the <picture> element

• the <img> tags will have loading="lazy" and decode="async" attributes

Now our site will serve the optimal image format and size depending on the browser and screen size of the site visitor. And images will be lazy loaded as they are about to enter the viewport.

If we use the network tab in a browser's developer tools, we can test the difference. On an iPhone 12, the unoptimized image on one of our project pages would be 30.37KB, while the image optimized by the Image plugin is just 6.01KB, an 80% saving!

Unoptimized image on mobile - 30.37KB

Optimized image on mobile 6.01KB

We’re almost ready to deploy our site. But before we do that, we need to complete our contact form.

How to a Build Contact Form with Netlify Forms

Contact page

Eleventy is a static site generator. But Eleventy works really well with the Jamstack architecture, where you statically generate as much of a site as possible in advance and use APIs and third-party services to add dynamic content and functionality.

In the past, if you wanted to have a contact form on your website, you would need some sort of server, such as a PHP app, to process the form submission.

We’re going to use Netlify Forms to add a contact form to our portfolio without needing to manage a server to handle the submitted forms.

To make this work, we need to make sure our form has two attributes. The most important one is data-netlify="true". The other is action="/success".

<form class="form" name="contact" action="/success" method="POST" data-netlify="true">
  <div class="form__section">
    <label class="form__label" for="yourName">Name</label>
    <input class="form__input" name="name" type="text" id="yourName" required="true">
  </div>
  <div class="form__section">
    <label class="form__label" for="yourEmail">Email</label>
    <input class="form__input" name="email" type="email"  id="yourEmail" required="true">
  </div>
  <div class="form__section">
    <label class="form__label" for="message">Message</label>
    <textarea class="form__input" name="message" id="message" rows="4" required="true"></textarea>
  </div>
    <button class="form__button" type="submit">Let's talk</button>
</form>

By having a data-netlify="true" attribute on our contact form, when the site is deployed to Netlify, Netlify will recognize this and take over handling the form submission.

By default, when someone completes a Netlify form, they will get a generically styled success message with a link back to the form page. But we can direct them to a custom page by including an action attribute on our form.

The action="/success" attribute means that when the form submits, the user will be redirected to a "success" page on your site (you can give this page a different name if you like). So we had better build that page now.

In the src directory, create a success.njk file with the following content:

---
title: "Eleventy Portfolio"
layout: "base.njk"
---

<div class="container text-center">
  <h2 class="heading--main">Thanks for getting in touch!</h2>
  <p>I'll respond as soon as I can.<p>
</div>

Once we deploy the site to Netlify, any submitted forms will show up in the Netlify interface. So let’s finally deploy our portfolio site.

How to Deploy to Netlify

You can deploy an Eleventy site on any static hosting platform: Netlify, Vercel, GitHub Pages, even an AWS S3 bucket.

I'll show you how to deploy to Netlify since we're using Netlify Forms for our contact form. On another hosting platform, you could use a Serverless function to handle submitting the form and sending an email.

If you don't already have a Netlify account, go to netlify.com and create a free one.

Netlify will give you the option to:

1. Import an existing project

2. Start from a template

3. Deploy manually

We already have our portfolio site, so we don't need a template.

I'll walk you through the two other options.

Netlify project start screen

Option 1 – How to Deploy Manually

If you're not comfortable with Git and GitHub, Netlify lets you drag and drop to upload a project into their interface.

On your command line, run npm run build or eleventy to build your site.

Now upload the site's public directory to Netlify's file upload interface. In a few moments Netlify will have the site live on a URL that you can visit.

If you want to make future changes to your deployed site, click on "Deploys" and scroll down to find the file uploader.

You can rebuild your site locally and upload the new version of your public folder to Netlify whenever you want.

Option 2 – How to Import a Project from Git

If you are familiar with Git and GitHub, commit your code and push it to GitHub. Then click on the "Import from Git" button.

Netlify will ask you to connect a Git provider. Choose GitHub and authorize Netlify to access your GitHub repositories.

Choose the repository that holds your portfolio site. You can search for "eleventy", or whatever name you gave it.

Netlify import project interface

Netlify will detect that this is an Eleventy project and will ask you to confirm the basic build settings.

Make sure the build command is either npm run build or eleventy.

Under "Publish directory", enter public instead of _site.

Now click the "Deploy site" button.

Netlify build settings page

In a few moments Netlify will tell you that your site is live and give you a URL for it.

Once your site is live, if go to the Contact page, fill out the form, and submit it. You'll be redirected to the custom success page that you made.

If you click on "Forms" in the Netlify interface, you'll be brought to the Netlify Forms dashboard.

The form will have whatever name you used in the name attribute on your contact form, in this case "contact".

Netlify forms dashboard

Congratulations, you've built and deployed an Eleventy portfolio site. 🥳🎉🎉🎉

Feel free to use this project as a template for your own portfolio and customize it any way you'd like. So many portfolios look similar, so it's always good when a portfolio shows off your personality and passions.

Where to Take it from Here

This tutorial has hopefully taught you the basics of Eleventy, and how to combine data and templates to make fast sites without a whole lot of tooling or configuration.

If you'd like to take your Eleventy journey further, the Eleventy docs are very good. There's lots more to learn about manipulating data, not to mention adding personalized content and dynamic interactivity with Serverless and Edge Functions.

11ty.rocks by Stephanie Eckles is also a great resource, with practical tips and helpful tutorials on all sorts of Eleventy features.

I hope this guide has been helpful and made you excited to learn more about Eleventy, static site generators, and the Jamstack.

In the Stack Overflow 2021 Survey, 4 out of the top 5 database technologies used by professional developers were relational database management systems.

PostgreSQL is an excellent choice as a first relational database management system to learn.

1. It’s widely used in industry, including at Uber, Netflix, Instagram, Spotify, and Twitch.

2. It’s open source, so you won’t be locked into a particular vendor.

3. It's more than 25 years old, and in that time it has earned a reputation for stability and reliability.

Whether you’re learning from the freeCodeCamp Relational Database Certification or trying out PostgreSQL on your own computer, you need a way to create and manage databases, insert data into them, and query data from them.

While there are several graphical applications for interacting with PostgreSQL, using psql and the command line is probably the most direct way to communicate with your database.

What is psql?

psql is a tool that lets you interact with PostgreSQL databases through a terminal interface. When you install PostgreSQL on a machine, psql is automatically included.

psql lets you write SQL queries, send them to PostgreSQL, and view the results. It also lets you use meta-commands (which start with a backslash) for administering the databases. You can even write scripts and automate tasks relating to your databases.

Now, running a database on your local computer and using the command line can seem intimidating at first. I’m here to tell you it’s really not so bad. This guide will teach you the basics of managing PostgreSQL databases from the command line, including how to create, manage, back up, and restore databases.

Prerequisite – Install PostgreSQL

If you haven’t already installed PostgreSQL on your computer, follow the instructions for your operating system on the official PostgreSQL documentation.

When you install PostgreSQL, you will be asked for a password. Keep this in a safe place as you’ll need it to connect to any databases you create.

How to Connect to a Database

You have two options when using psql to connect to a database: you can connect via the command line or by using the psql application. Both provide pretty much the same experience.

Option 1 – Connect to a database with the command line

Open a terminal. You can make sure psql is installed by typing psql --version. You should see psql (PostgreSQL) version_number, where version_number is the version of PostgreSQL that’s installed on your machine. In my case, it's 14.1.

Checking psql version via the command line

The pattern for connecting to a database is:

psql -d database_name -U username

The -d flag is shorter alternative for --dbname while -U is an alternative for --username.

When you installed PostgreSQL, a default database and user were created, both called postgres. So enter psql -d postgres -U postgres to connect to the postgres database as the postgres superuser.

psql -d postgres -U postgres

You will be prompted for a password. Enter the password you chose when you installed PostgreSQL on your computer. Your terminal prompt will change to show that you’re now connected to the postgres database.

Connecting to a database from the command line with psql

If you want to directly connect to a database as yourself (rather than as the postgres superuser), enter your system username as the username value.

Option 2 – Connect to a database with the psql application

Launch the psql application – it'll be called "SQL Shell (psql)". You will be prompted for a server, a database, a port and a username. You can just press enter to select the default values, which are localhost, postgres, 5432, and postgres.

Next, you’ll be prompted for the password you chose when you installed PostgreSQL. Once you enter this, your terminal prompt will change to show that you’re connected to the postgres database.

Connecting to a database with the psql application

Note: If you’re on Windows you might see a warning like “Console code page (850) differs from Windows code page (1252) 8-bit characters might not work correctly. See psql reference page 'Notes for Windows users' for details.” You don’t need to worry about this at this stage. If you want to read more about it, see the psql documentation.

How to Get Help in psql

To see a list of all psql meta-commands, and a brief summary of what they do, use the \? command.

\?

psql's help command

If you want help with a PostgreSQL command, use \h or \help and the command.

\h COMMAND

This will give you a description of the command, its syntax (with optional parts in square brackets), and a URL for the relevant part of the PostgreSQL documentation.

psql describing the DROP TABLE statement

How to Quit a Command in psql

If you’ve run a command that’s taking a long time or printing too much information to the console, you can quit it by typing q.

q

How to Create a Database

Before you can manage any databases, you’ll need to create one.

Note: SQL commands should end with a semicolon, while meta-commands (which start with a backslash) don’t need to.

The SQL command to create a database is:

CREATE DATABASE database_name;

For this guide, we’re going to be working with book data, so let’s create a database called books_db.

CREATE DATABASE books_db;

How to List Databases

You can view a list of all available databases with the list command.

\l

Listing all databases

You should see books_db, as well as postgres, template0, and template1. (The CREATE DATABASE command actually works by copying the standard database, called template1. You can read more about this in the PostgreSQL documentation.)

Using \l+ will display additional information, such as the size of the databases and their tablespaces (the location in the filesystem where the files representing the database will be stored).

\l+

Listing all databases with additional information

How to Switch Databases

You’re currently still connected to the default postgres database. To connect to a database or to switch between databases, use the \c command.

\c database_name

So \c books_db will connect you to the books_db database. Note that your terminal prompt changes to reflect the database you’re currently connected to.

Switching databases

How to Delete a Database

If you want to delete a database, use the DROP DATABASE command.

DROP DATABASE database_name;

You will only be allowed to delete a database if you are a superuser, such as postgres, or if you are the database’s owner.

If you try to delete a database that doesn’t exist, you will get an error. Use IF EXISTS to get a notice instead.

DROP DATABASE IF EXISTS database_name;

Deleting a database

You can’t delete a database that has active connections. So if you want to delete the database you are currently connected to, you’ll need to switch to another database.

How to Create Tables

Before we can manage tables, we need to create a few and populate them with some sample data.

The command to create a table is:

CREATE TABLE table_name();

This will create an empty table. You can also pass column values into the parentheses to create a table with columns. At the very least, a basic table should have a Primary Key (a unique identifier to tell each row apart) and a column with some data in it.

For our books_db, we’ll create a table for authors and another for books. For authors, we’ll record their first name and last name. For books, we’ll record the title and the year they were published.

We’ll make sure that the authors’ first_name and last_name and the books’ title aren’t null, since this is pretty vital information to know about them. To do this we include the NOT NULL constraint.

CREATE TABLE authors(
    author_id SERIAL PRIMARY KEY, 
    first_name VARCHAR(100) NOT NULL, 
    last_name VARCHAR(100) NOT NULL
);

CREATE TABLE books(
    book_id SERIAL PRIMARY KEY, 
    title VARCHAR(100) NOT NULL, 
    published_year INT
);

You will see CREATE TABLE printed to the terminal if the table was created successfully.

Now let's connect the two tables by adding a Foreign Key to books. Foreign Keys are unique identifiers that reference the Primary Key of another table. Books can, of course, have multiple authors but we’re not going to get into the complexities of many to many relationships right now.

Add a Foreign Key to books with the following command:

ALTER TABLE books ADD COLUMN author_id INT REFERENCES authors(author_id);

Next, let’s insert some sample data into the tables. We’ll start with authors.

INSERT INTO authors (first_name, last_name) 
VALUES (‘Tamsyn’, ‘Muir’), (‘Ann’, ‘Leckie’), (‘Zen’, ‘Cho’);

Select everything from authors to make sure the insert command worked.

SELECT * FROM authors;

Querying all data from the authors table

Next, we’ll insert some books data into books.

INSERT INTO books(title, published_year, author_id) 
VALUES (‘Gideon the Ninth’, 2019, 1), (‘Ancillary Justice’, 2013, 2), (‘Black Water Sister’, 2021, 3);

If you run SELECT * FROM books; you’ll see the book data.

Querying all data from the books table

How to List All Tables

You can use the \dt command to list all the tables in a database.

\dt

For books_db you will see books and authors. You'll also see books_book_id_seq and authors_author_id_seq. These keep track of the sequence of integers used as ids by the tables because we used SERIAL to generate their Primary Keys.

Listing all tables in a database

How to Describe a Table

To see more information about a particular table, you can use the describe table command: \d table_name. This will list the columns, indexes, and any references to other tables.

\d table_name

Describing the authors table

Using \dt+ table_name will provide more information, such as about storage and compression.

How to Rename a Table

If you ever need to change the name of a table, you can rename it with the ALTER TABLE command.

ALTER TABLE table_name RENAME TO new_table_name;

How to Delete a Table

If you want to delete a table, you can use the DROP TABLE command.

DROP TABLE table_name;

If you try to delete a table that doesn’t exist, you will get an error. You can avoid this by including the IF EXISTS option in the statement. This way you’ll get a notice instead.

DROP TABLE IF EXISTS table_name;

How to Manage Longer Commands and Queries

If you’re writing longer SQL queries, the command line isn’t the most ergonomic way to do it. It's probably better to write your SQL in a file and then have psql execute it.

If you are working with psql and think your next query will be long, you can open a text editor from psql and write it there. If you have an existing query, or maybe want to run several queries to load sample data, you can execute commands from a file that is already written.

Option 1 – Open a text editor from psql

If you enter the \e command, psql will open a text editor. When you save and close the editor, psql will run the command you just wrote.

\e

Writing commands in a text editor

On Windows, the default text editor for psql is Notepad, while on MacOs and Linux it's vi. You can change this to another editor by setting the EDITOR value in your computer’s environment variables.

Option 2 – Execute commands and queries from a file

If you have particularly long commands or multiple commands that you want to run, it would be better to write the SQL in a file ahead of time and have psql execute that file once you’re ready.

The \i command lets you read input from a file as if you had typed it into the terminal.

\i path_to_file/file_name.sql

Note: If you're executing this command on Windows, you still need to use forward slashes in the file path.

If you don’t specify a path, psql will look for the file in the last directory that you were in before you connected to PostgreSQL.

Executing SQL commands from a file

How to Time Queries

If you want to see how long your queries are taking, you can turn on query execution timing.

\timing

This will display in milliseconds the time that the query took to complete.

If you run the \timing command again, it will turn off query execution timing.

Using query execution timing

How to Import Data from a CSV File

If you have a CSV file with data and you want to load this into a PostgreSQL database, you can do this from the command line with psql.

First, create a CSV file called films.csv with the following structure (It doesn’t matter if you use Excel, Google Sheets, Numbers, or any other program).

A spreadsheet with Pixar film data

Open psql and create a films_db database, connect to it, and create a films table.

CREATE DATABASE films_db;

\c films_db

CREATE TABLE films(
    id SERIAL PRIMARY KEY,
    title VARCHAR(100),
    year INT,
    running_time INT
);

You can then use the \copy command to import the CSV file into films. You need to provide an absolute path to where the CSV file is on your computer.

\copy films(title, year, running_time) FROM 'path_to_file' DELIMITER ‘,’ CSV HEADER;

The DELIMITER option specifies the character that separates the columns in each row of the file being imported, CSV specifies that it is a CSV file, and HEADER specifies that the file contains a header line with the names of the columns.

Note: The column names of the films table don't need to match the column names of films.csv but they do need to be in the same order.

Use SELECT * FROM films; to see if the process was successful.

Importing data from a .csv file

How to Back Up a Database with pg_dump

If you need to backup a database, pg_dump is a utility that lets you extract a database into a SQL script file or other type of archive file.

First, on the command line (not in psql), navigate to the PostgreSQL bin folder.

cd "C:\Program Files\PostgreSQL\14\bin"

Then run the following command, using postgres as the username, and filling in the database and output file that you want to use.

pg_dump -U username database_name > path_to_file/filename.sql

Use postgres for the username and you will be prompted for the postgres superuser's password. pg_dump will then create a .sql file containing the SQL commands needed to recreate the database.

Backing up a database to a .sql file

If you don’t specify a path for the output file, pg_dump will save the file in the last directory that you were in before you connected to PostgreSQL.

Contents of films.sql backup file

You can pass the -v or --verbose flag to pg_dump to see what pg_dump is doing at each step.

Running pg_dump in verbose mode

You can also backup a database to other file formats, such as .tar (an archive format).

pg_dump -U username -F t database_name > path_to_file/filename.tar

Here the -F flag tells pg_dump that you're going to specify an output format, while t tells it it's going to be in the .tar format.

How to Restore a Database

You can restore a database from a backup file using either psql or the pg_restore utility. Which one you choose depends on the type of file you are restoring the database from.

1. If you backed up the database to a plaintext format, such as .sql, use psql.

2. If you backed up the database to an archive format, such as .tar, use pg_restore.

Option 1 – Restore a database using psql

To restore a database from a .sql file, on the command line (so not in psql), use psql -U username -d database_name -f filename.sql.

You can use the films_db database and films.sql file you used earlier, or create a new backup file.

Create an empty database for the file to restore the data into. If you're using films.sql to restore films_db, the easiest thing might be to delete films_db and recreate it.

DROP DATABASE films_db;

CREATE DATABASE films_db;

In a separate terminal (not in psql), run the following command, passing in postgres as the username, and the names of the database and backup file you are using.

psql -U username -d database_name -f path_to_file/filename.sql

The -d flag points psql to a specific database, while the -f flag tells psql to read from the specified file.

If you don’t specify a path for the backup file, psql will look for the file in the last directory that you were in before you connected to PostgreSQL.

You will be prompted for the postgres superuser's password and then will see a series of commands get printed to the command line while psql recreates the database.

Restoring a database using psql

This command ignores any errors that occur during the restore. If you want to stop restoring the database if an error occurs, pass in --set ON_ERROR_STOP=on.

psql -U username -d database_name --set ON_ERROR_STOP=on -f filename.sql

Option 2 – Restore a database using pg_restore

To restore a database using pg_restore, use pg_restore -U username -d database_name path_to_file/filename.tar.

Create an empty database for the file to restore the data into. If you're restoring films_db from a films.tar file, the easiest thing might be to delete films_db and recreate it.

DROP DATABASE films_db;

CREATE DATABASE films_db;

On the command line (not in psql), run the following command, passing in postgres as the username, and the names of the database and backup file you are using.

pg_restore -U username -d database_name path_to_file/filename.tar

Restoring a database using pg_restore

You can also pass in the -v or --verbose flag to see what pg_restore is doing at each step.

Using pg_restore in verbose mode

How to Quit psql

If you’ve finished with psql and want to exit from it, enter quit or \q.

\q

This will close the psql application if you were using it, or return you to your regular command prompt if you were using psql from the command line.

Where to Take it from Here

There are lots more things you can do with psql, such as managing schemas, roles, and tablespaces. But this guide should be enough to get you started with managing PostgreSQL databases from the command line.

If you want to learn more about PostgreSQL and psql, you could try out freeCodeCamp’s Relational Database Certificate . The official PostgreSQL documentation is comprehensive, and PostgreSQL Tutorial offers several in-depth tutorials.

I hope you find this guide helpful as you continue to learn about PostgreSQL and relational databases.