We'll implement core features like file replication, metadata management, and versioning, all while demonstrating how to achieve eventual consistency in a distributed environment. By the end, you'll have a fully functional distributed file storage system that can handle high traffic, optimize storage, and ensure data integrity.

What You Will Learn

• How to set up MinIO for distributed object storage.

• How to use gRPC for efficient client-server communication.

• How to implement file replication and metadata management.

• How to understand data consistency in a distributed system.

• How to use Docker to deploy a scalable, distributed architecture.

Prerequisites

Before starting, ensure you have the following installed:

• Node.js (v14 or higher)

• MinIO

• gRPC and gRPC-tools

• Docker

You’ll also need to have a basic understanding of Node.js, object storage, and distributed systems.

Table of Contents

• Project Overview

• Step 1: Setting Up the Project

• Step 2: Setting Up MinIO Distributed Storage Nodes

• Step 3: Defining the gRPC Protocol

• Step 4: Implementing the gRPC Server

• Step 5: Creating the Client

• Step 6: Running the System

• Conclusion: What You’ve Learned

Project Overview

We'll build a distributed file storage system where:

1. Users can upload and download files.

2. Files are replicated across multiple storage nodes to ensure high availability.

3. Metadata (like file names, upload times, and versions) is managed centrally.

4. The system handles eventual consistency by syncing file updates across nodes.

System Architecture

Our system will consist of:

1. gRPC Server: Manages file uploads, downloads, and metadata.

2. MinIO Distributed Storage Nodes: Handles object storage and replication.

3. Client Interface: Allows users to interact with the system via HTTP.

Step 1: Setting Up the Project

Create a new directory for the project and initialize a Node.js application:

mkdir distributed-file-storage
cd distributed-file-storage
npm init -y

Now, install the necessary dependencies:

npm install grpc @grpc/grpc-js @grpc/proto-loader express multer dotenv minio

• grpc: For building gRPC server and client.

• @grpc/proto-loader: Loads gRPC protocol files.

• express: For the client-side HTTP server.

• multer: For handling file uploads.

• dotenv: For managing environment variables.

• minio: MinIO client for interacting with storage nodes.

Create a .env file with the following content:

MINIO_ENDPOINT_1=localhost:9001
MINIO_ACCESS_KEY=minioadmin
MINIO_SECRET_KEY=minioadmin
PORT=5000

Step 2: Setting Up MinIO Distributed Storage Nodes

We'll use Docker to run multiple MinIO instances, simulating a distributed environment. Run the following commands to set up three MinIO containers:

docker run -p 9001:9000 --name minio1 -e "MINIO_ACCESS_KEY=minioadmin" -e "MINIO_SECRET_KEY=minioadmin" -d minio/minio server /data
docker run -p 9002:9000 --name minio2 -e "MINIO_ACCESS_KEY=minioadmin" -e "MINIO_SECRET_KEY=minioadmin" -d minio/minio server /data
docker run -p 9003:9000 --name minio3 -e "MINIO_ACCESS_KEY=minioadmin" -e "MINIO_SECRET_KEY=minioadmin" -d minio/minio server /data

These commands will start three MinIO nodes, each listening on a different port.

Step 3: Defining the gRPC Protocol

Create a new folder named protos and inside it, create a file called storage.proto:

syntax = "proto3";

service FileStorage {
  rpc UploadFile(stream FileRequest) returns (UploadResponse);
  rpc DownloadFile(FileDownloadRequest) returns (stream FileResponse);
  rpc GetMetadata(FileMetadataRequest) returns (MetadataResponse);
}

message FileRequest {
  bytes fileData = 1;
  string fileName = 2;
}

message UploadResponse {
  string message = 1;
}

message FileDownloadRequest {
  string fileName = 1;
}

message FileResponse {
  bytes fileData = 1;
}

message FileMetadataRequest {
  string fileName = 1;
}

message MetadataResponse {
  string fileName = 1;
  string uploadTime = 2;
  string version = 3;
}

• UploadFile: Streams file data from the client to the server.

• DownloadFile: Streams file data from the server to the client.

• GetMetadata: Retrieves metadata like file name, upload time, and version.

Step 4: Implementing the gRPC Server

Create a file called server.js:

require('dotenv').config();
const grpc = require('@grpc/grpc-js');
const protoLoader = require('@grpc/proto-loader');
const Minio = require('minio');
const fs = require('fs');
const path = require('path');

const packageDefinition = protoLoader.loadSync('protos/storage.proto');
const storageProto = grpc.loadPackageDefinition(packageDefinition).FileStorage;

// Set up MinIO clients for each node
const minioClients = [
  new Minio.Client({
    endPoint: process.env.MINIO_ENDPOINT_1.split(':')[0],
    port: parseInt(process.env.MINIO_ENDPOINT_1.split(':')[1]),
    accessKey: process.env.MINIO_ACCESS_KEY,
    secretKey: process.env.MINIO_SECRET_KEY,
    useSSL: false,
  })
];

// Upload file to MinIO
async function uploadFile(call, callback) {
  const chunks = [];
  call.on('data', (chunk) => chunks.push(chunk.fileData));
  call.on('end', async () => {
    const buffer = Buffer.concat(chunks);
    const fileName = call.metadata.get('fileName')[0];

    // Store file in MinIO
    const client = minioClients[0];
    await client.putObject('files', fileName, buffer);
    callback(null, { message: `File ${fileName} uploaded successfully` });
  });
}

// Download file from MinIO
function downloadFile(call) {
  const { fileName } = call.request;
  const client = minioClients[0];

  client.getObject('files', fileName, (err, stream) => {
    if (err) return call.emit('error', err);
    stream.on('data', (chunk) => call.write({ fileData: chunk }));
    stream.on('end', () => call.end());
  });
}

function main() {
  const server = new grpc.Server();
  server.addService(storageProto.FileStorage.service, { uploadFile, downloadFile });
  server.bindAsync('0.0.0.0:5000', grpc.ServerCredentials.createInsecure(), () => {
    console.log('gRPC server running on port 5000');
    server.start();
  });
}

main();

Here’s what’s going on in this code:

1. uploadFile: Handles file uploads by streaming data to the server and storing it in MinIO.

2. downloadFile: Streams the requested file back to the client from MinIO.

3. MinIO Clients: We set up multiple MinIO clients to handle distributed storage.

Step 5: Creating the Client

Create a file named client.js:

const grpc = require('@grpc/grpc-js');
const protoLoader = require('@grpc/proto-loader');
const fs = require('fs');

const packageDefinition = protoLoader.loadSync('protos/storage.proto');
const storageProto = grpc.loadPackageDefinition(packageDefinition).FileStorage;
const client = new storageProto('localhost:5000', grpc.credentials.createInsecure());

function uploadFile(filePath) {
  const call = client.uploadFile();
  const fileName = filePath.split('/').pop();
  const stream = fs.createReadStream(filePath);

  stream.on('data', (chunk) => call.write({ fileData: chunk }));
  stream.on('end', () => call.end());
  call.on('data', (response) => console.log(response.message));
}

function downloadFile(fileName) {
  const call = client.downloadFile({ fileName });
  const writeStream = fs.createWriteStream(`downloaded_${fileName}`);

  call.on('data', (chunk) => writeStream.write(chunk.fileData));
  call.on('end', () => console.log(`Downloaded ${fileName}`));
}

uploadFile('test.txt');  // Example usage

Step 6: Running the System

1. Start the gRPC Server:

    node server.js
   

2. Run the Client:

    node client.js
   

Conclusion: What You’ve Learned

Congratulations! You've built a distributed file storage system using MinIO and gRPC. In this tutorial, you learned how to:

1. Set up a distributed object storage system using MinIO.

2. Use gRPC to handle file uploads, downloads, and metadata management.

3. Implement file replication and eventual consistency across multiple nodes.

4. Utilize Docker to simulate a scalable distributed environment.

Next Steps:

1. Add File Versioning: Store multiple versions of files for rollback.

2. Implement Authentication: Secure your gRPC endpoints with JWT.

3. Deploy with Kubernetes: Scale your system across multiple nodes for high availability.

Happy coding!

In this article, you will learn what local storage is and how to use it in modern web applications. You will also learn the advantages of using local storage, as well as some of its limitations.

Table of Contents

• What is Local Storage?

• Differences Between Local Storage and Session Storage

• How to Use Local Storage

• A Practical Example

• How to View Local Storage in DevTools

• Benefits of Using Local Storage

• Limitations of Local Storage

• Conclusion

What is Local Storage?

Local storage is a feature in web browsers that allows developers to save data in the user’s browser. It’s part of the web storage API, together with session storage.

Local storage works by accepting data in key-value pairs. It retains the data even when the user refreshes the page or closes the tab or browser.

Differences Between Local Storage and Session Storage

As I mentioned earlier, the web storage API in modern browsers provides two main features for data storage. These are local storage and session storage.

The key differences between the two are the lifespan of the stored data and their scope.

Data in local storage remains available even when the tab/browser is closed. But closing the tab/browser clears any data stored in session storage.

Also, data in local storage is accessible across multiple browser tabs and windows. On the other hand, data in session storage is only accessible within specific browser tabs and is not shared.

How to Use Local Storage

The local storage object provides different methods you can use to interact with it. With these methods, you can add, read, and delete data from local storage.

How to Store Data in Local Storage

To store data in local storage, you use the setItem() method. This method takes in two arguments, a key and a value.

localStorage.setItem(key, value)

If the key does not exist in local storage, the setItem() method will create a new key and assign the given value to it. But if a key with the same name exists in local storage, it will update the value of the key with the provided value.

How to Read Data From Local Storage

To retrieve and use data from local storage, you use the getItem() method. This method takes in a key as an argument.

localStorage.getItem(key)

If the given key exists in local storage, the method returns the value of that key. If it doesn’t, the method returns null.

How to Store and Read Complex Data Values in Local Storage

Local storage can only store strings. This means if you need to store values like objects or arrays, you first need to get a string representation of the value. You do this using the JSON.stringify() method.

Example:

const userObj = {
  username = "Maria",
  email: "maria@mail.com"
}

localStorage.setItem('user', JSON.stringify(userObj))

The JSON.stringify() method converts the userObj object into a string representation before sending it to local storage.

Now, when you want to retrieve the data back from local storage, you also need to change it from its string representation back to the original form. And you do that using the JSON.parse() method.

Example:

const storedUserData = localStorage.getItem('user')

if (storedUserData) {
  const userData = JSON.parse(storedUserData)
  // You can use userData here...
} else {
  console.log('User data not found in local storage')
}

In the example above, we first check if there is data for ‘user’ in local storage before using the JSON.parse() method. This is important because if it does not exist in local storage, JSON.parse() will be applied to a null value (which will result in an error).

How to Delete Data from Local Storage

There are two methods available for deleting data from local storage. One is the removeItem() method and the other is the clear() method.

You use the removeItem() method when you want to delete a single item from local storage. The method takes in a key as an argument and deletes the corresponding key-value pair from local storage.

localStorage.removeItem(key)

But what if, instead of deleting a single key-value pair, you want to clear all data from the local storage? Well, local storage has a method for that - the clear() method.

localStorage.clear()

The clear() method deletes all key-value pairs in the local storage for the current domain.

How to Get the Name of a Key in Local Storage

If you want to get the name of a key at a particular index in local storage, you can use the key() method. It takes in a number as an argument and returns the name of the key at that specified index.

Example:

localStorage.key(0)

The example above will return the name of the key at index 0. If there is no key at the specified index, the method will return null.

A Practical Example

The following shows a practical demo of the difference between local storage and session storage.

In this example, we'll save the user's name in local storage and save the age in session storage.

<!-- HTML --> 
<body>

  <h1 class="userName"></h1>
  <h2 class="userAge"></h2>

  <input type="text" class="name" placeholder="Enter name here"/>
  <button class="saveNameBtn">Save Name</button>

  <br />

  <input type="text" class="age" placeholder="Enter age here"/>
  <button class="saveAgeBtn">Save Age</button>

</body>

The markup includes two header elements. One for userName and the other for userAge. It also includes two input elements for name and age. Each input has an associated button we'll use for saving the data.

Now, let's use the querySelector method to select the various elements.

const userNameText = document.querySelector(".userName")
const userAgeText = document.querySelector(".userAge")

const saveNameButton = document.querySelector(".saveNameBtn")
const saveAgeButton = document.querySelector(".saveAgeBtn")

Code Example for Local Storage

saveNameButton.addEventListener("click", () => {
  const userName = document.querySelector(".name").value
  userNameText.textContent = userName
  localStorage.setItem("name", userName)
})

First, we get the value of the name input, set it as the textContent of userNameText. And then use the setItem() of local storage to save the userName value in local storage.

Next, let's see how we can get the name value from local storage when we need it.

function displayUserName () {
  const nameFromLocalStorage = localStorage.getItem("name")

  if (nameFromLocalStorage) {
    userNameText.textContent = nameFromLocalStorage
  } else {
    userNameText.textContent = "No name data in local storage"
  }
}

displayUserName()

The displayUserName function gets nameFromLocalStorage using the getItem() method. If the value exists in local storage, we set it as the textContent of the userNameText element. If it's null or doesn't exist, then we set textContent to the string "No name data in local storage".

Code Example for Session Storage

Now, let's do the same thing for the age value. The only difference here will be using session storage instead of local storage.

saveAgeButton.addEventListener("click", () => {
  const userAge = document.querySelector(".age").value
  userAgeText.textContent = userAge
  sessionStorage.setItem("age", userAge)
})

function displayUserAge () {
  const ageFromSessionStorage = sessionStorage.getItem("age")

  if (ageFromSessionStorage) {
    userAgeText.textContent = ageFromSessionStorage
  } else {
    userAgeText.textContent = "No age data in session storage"
  }
}

displayUserAge()

The setItem and getItem methods also works for session storage.

Demo:

Local storage and session storage demo.

As you can see from the demo above, when you close and reopen the page, the name data from local storage persists. But the age data from session storage is cleared once the page closes.

Try your hands on the code sample on StackBlitz

How to View Local Storage in DevTools

You can follow the steps below to inspect the contents of local storage in your browser's developer tools.

First, open DevTools. You can do that by right clicking on the web page and selecting "Inspect".

Demo of how to open the DevTools.

Then, select the "Application" tab on the DevTools panel. Depending on your browser, this panel may have a different name. For example, it's called "Storage" in Safari and Firefox.

Demo of how to open the "Application" panel in DevTools.

Locate the "Storage" section on the sidebar showing a list of the various web storage options.

Click on "Local Storage" to expand and view its contents.

Demo of how to open the local storage tab in the storage panel.

You can click on individual items to view the corresponding key-value pair.

Benefits of Using Local Storage

The following are some of the benefits local storage has over other storage mechanisms in modern web development.

1. Persistent data: When you use local storage, the stored data remains even when the user closes the tab or the browser. This is useful for saving user preferences, settings, and other relevant data. It can help create a seamless user experience.

2. Offline access: You can use local storage as a means to cache data which can be accessed even with limited or no internet. This makes it a useful feature for apps that rely on caching data for offline use like news readers, productivity apps, and so on.

3. More storage capacity: Compared to other storage means, local storage has a relatively high capacity. For example, cookies are limited to 4 kilobytes per domain. But local storage can store up to 5 megabytes of data per domain.

Limitations of Using Local Storage

1. Stores only strings: As you learned earlier, local storage can only store string values. You can use the JSON stringify and parse methods to work around it. But some web developers may not prefer it as it can lead to writing complex code that’s difficult to debug.

2. Security concerns: Data in the local storage can be prone to attacks like cross-site scripting (XSS). As such, you should be cautious when working with sensitive information. It’s advisable to assess security implications and consider other alternatives where necessary.

3. Not accessible to web workers: Local storage is part of the Window object. As such, it’s tied to the main execution thread of the web page. This means it's not accessible to web workers. So if you run any background processes, you cannot use local storage within the web worker scripts.

Conclusion

Local storage is a feature in modern web browsers that makes it easy for web developers to store and persist data between browser sessions.

Compared to traditional cookies, it provides larger storage capacities. Also, unlike cookies, it does not rely on server-side processes. This reduces the need for frequent server requests and helps improve performance.

In this article, you learn about how to use local storage. We covered saving, retrieving, and deleting data from local storage. You also learned about some of the benefits of using local storage in your project, and some of its limitations too.

Thanks for reading. And happy coding! For more in-depth tutorials, feel free to subscribe to my YouTube channel.

In this detailed guide, we'll explore the features of Master Boot Record (MBR) disk partitioning using the parted command on Linux.

Additionally, we'll cover the process of adding a new hard disk in VMware, as well as how to format and mount a newly created partition using mkfs and fstab, respectively.

Table Of Contents

Here's what we'll cover in this comprehensive guide:

• Overview of MBR

• Introduction to Parted Utility

• How to Identify Disks

• How to Launch Parted

• How to Create a New MBR Partition Table

• How to Create Primary Partitions

• How to Display Partition Information

• How to Format the Partition with mkfs

• How to Automate Mounting with fstab

• How to Mount a Partition

• How to Create Logical Partitions

• Wrapping Up

Overview of MBR

MBR is a legacy partitioning scheme used on BIOS-based systems, with a 512-byte sector at the beginning of the storage device which contains partition information and the bootloader.

The 512 bytes of the MBR are organized as follows:

• Master Boot Code (446 bytes): This section contains the executable code responsible for loading the operating system. It locates the active (bootable) partition and passes control to its boot sector.

• Partition Table (64 bytes): The next 64 bytes are reserved for the partition table, which can describe up to four primary partitions or three primary partitions and one extended partition.

• MBR Signature (2 bytes): The last two bytes contain a signature (0x55AA), indicating that this is a valid MBR.

Introduction to the Parted Utility

parted is a command-line utility designed for creating, resizing, and managing disk partitions on Linux systems.

It offers a comprehensive set of features for handling disk-related tasks and is widely used due to its flexibility and robust capabilities. The primary goals of parted include:

• Partition Creation and Resizing: parted enables users to create new partitions on a disk, resize existing partitions, and adjust the allocation of disk space.

• File System Support: It supports various file systems, including ext2, ext3, ext4, FAT, NTFS, and more, allowing users to choose the file system that best suits their needs.

• Partition Table Manipulation: parted facilitates the management of partition tables, including creating, modifying, and deleting partitions within these tables.

• Sector-level Operations: The utility operates at the sector level, allowing for precise control and manipulation of disk structures.

How to Install Parted

You can install parted on your system using this command:

yum install parted

Before creating partition we need to add new disk to our virtual machine. I am using VMware. Go on and start RHEL virtual machine.

First we need to go to Player option on top left corner and then to the Manage option. Using that option, you can select virtual machine settings.

Add new hard disk in VMware

Click on "add to new hard disk". Then select hard disk and keep pressing on next until it asks for disk capacity. For learning purposes, add only 5 GB disk.

Continue clicking next until you get to the finish option.

Adding virtual hard disk - 1

Adding virtual hard disk - 2

Adding virtual hard disk - 3

We are focusing on creating partition, so we'll not focus on parts which are not needed for partition.

Next, restart your virtual machine to reflect the newly added hard disk.

How to Identify Disks

Identifying disks is a crucial step in the process of disk partitioning as it ensures that you are working with the correct storage device on your system.

Use the lsblk command to display information about block devices, including hard drives and partitions.

lsblk

Look for the disk you want to partition. Disk names are typically in the format /dev/sdX, where 'X' is a lowercase letter representing the disk. For example, /dev/sda, /dev/sdb, etc.

How to Launch Parted

Open a terminal and launch parted for the target disk:

parted /dev/sdX

This initiates the parted utility with elevated privileges and directs it to focus on the specified storage device (/dev/sdX).

Once executed, the user enters the parted interactive mode for the specified device, enabling the configuration and management of partitions on that particular storage device.

How to Create a New MBR Partition Table

If the disk is unpartitioned, create a new MBR partition table using this command:

(parted) mklabel msdos

This instructs parted to create a new MBR partition table on the currently selected storage device. This action will remove any existing partition information on the device, effectively starting with a clean slate for partitioning.

It's important to note that this command should be used with caution, as it erases the existing partition table and all associated data on the device.

How to Create Primary Partitions

You can create a primary partition with specified filesystem type, start, and end:

(parted) mkpart primary [filesystem-type] [start] [end]

• mkpart: This is the parted command used to create a new partition. The term "mkpart" stands for "make partition."

• primary: This specifies the type of partition to be created. In this case, it's a primary partition. Primary partitions are the basic partitions on a disk, and they can be used to install an operating system or store data.

• [filesystem-type]: Replace this placeholder with the desired filesystem type for the partition. Common filesystem types include ext4, ntfs, fat32, and so on.

• [start] and [end]: These placeholders represent the starting and ending points of the partition, specified in megabytes (MB) or as a percentage of the total disk size. For example, you might define the partition to start at 1 GB and end at 10 GB.

How to Display Partition Information

Use this command to verify the partition layout:

(parted) print free

When you enter (parted) print free in the parted interactive mode, it will provide a summary of the free space available on the selected disk, including details such as the starting and ending points of the unallocated space, its size, and any constraints on creating new partitions within that space.

Here's the output:

(parted) print free
Model: ABC Storage Device
Disk /dev/sdX: 1000GB
Sector size (logical/physical): 512B/4096B
Partition Table: msdos
Disk Flags:

Number  Start   End     Size    Type  File system  Flags
        32.3kB  1049kB  1017kB  Free Space
 1      1049kB  256MB   255MB   primary  ext4         boot
 2      256MB   512MB   256MB   primary  linux-swap
 3      512MB   1000GB  1000GB  primary  ntfs

Press ctrl + d to get out of parted.

How to Format the Partition with mkfs

After partitioning, format the partition with a desired filesystem (for example: ext4)

mkfs.[filesystem-type] /dev/sdX1

This tells the system to create a file system of the specified type on the designated partition. After executing this command, the chosen file system will be created on the partition, and it will be ready to store files and data.

How to Automate Mounting with fstab

You can find the UUID of the partition using this command:

sudo blkid /dev/sdX1

Edit /etc/fstab to include an entry for automatic mounting, and make sure filesystem-type is as same as filesystem type of mkfs.[filesystem-type]

The /etc/fstab file, short for "file systems table," is a crucial configuration file on Unix and Unix-like operating systems, including Linux. It is used to define how storage devices (such as hard drives and partitions) should be mounted into the file system.

The primary purpose of the /etc/fstab file is to specify how various storage devices should be mounted during the system's startup process.

UUID=[your-uuid] /dir-to-mount-path [filesystem-type] defaults 0 0

Here's an example:

UUID=127854Vd344HHttRpq977739 /kedar ext4 defaults 0 0

UUID=127854Vd344HHttRpq977739: This part specifies the Universally Unique Identifier (UUID) of the block device or partition to be mounted. The UUID is a unique identifier assigned to the filesystem, ensuring a consistent reference even if the device names change. In this case, the UUID is "127854Vd344HHttRpq977739."

/kedar: This part indicates the mount point, which is the directory in the file system where the device specified by the UUID will be attached. In this example, the mount point is "/kedar."

ext4: This specifies the type of the file system on the device. In this case, it's "ext4," which is a commonly used file system on Linux.

defaults: The "defaults" option is a shorthand for a set of commonly used mount options. It includes options like read/write access and allows executing binaries from the file system. If more specific options are needed, they can be listed explicitly.

0: This field represents the dump option. It is used by the dump command to determine whether the file system should be backed up. A value of 0 indicates that no automatic backups are required.

0: The last field represents the pass option. It is used by the fsck (file system check) utility to determine the order in which file systems are checked at boot time. A value of 0 indicates that the file system should not be checked.

This /etc/fstab entry directs the system to mount a device, identified by the specified UUID and formatted as ext4, onto the "/kedar" directory during the boot process. The mount is performed with default options, and there is no requirement for automatic backups or file system checks.

Here are some reasons to use UUID instead of partition name

• Device names (e.g., /dev/sda1) can change, especially in systems with multiple storage devices or when hardware configurations are modified. UUIDs remain constant.

• During the boot process, multiple storage devices may be detected simultaneously. UUIDs eliminate the possibility of race conditions, where the operating system may assign different names to the same physical device in different boot instances.

• If device names are used in /etc/fstab and names change due to hardware changes, it may lead to mounting errors or data corruption. UUIDs ensure the correct device is always mounted.

• UUIDs are more human-readable and less prone to typos than device names, making the /etc/fstab file easier to read and maintain.

• UUIDs remain the same when moving a storage device from one system to another, ensuring correct mounting without modifying /etc/fstab for the new system.

Here's an example /etc/fstab entry using UUID:

UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx /mnt/data ext4 defaults 0 2

Replace xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx with the actual UUID of the partition.

How to Mount a Partition

Once everything is done, we will mount partition on one of your directory.

mount -av

If this returns a "successfully mounted" message on kedar directory, then your partition has mounted on kedar directory.

Now every time you turn on your system, the specified partition will automatically be mounted on the "/kedar" directory, ensuring seamless access to its contents throughout your computing sessions.

This persistent mount configuration is facilitated by the entry in the /etc/fstab file, providing a consistent and reliable attachment of the partition during the system's startup process.

How to Create Logical Partitions

Logical partitions are a key concept in disk partitioning, especially in systems using the Master Boot Record (MBR) partitioning scheme.

MBR allows for a maximum of four primary partitions, and to overcome this limitation, one of the primary partitions can be designated as an extended partition.

Within the extended partition, multiple logical partitions can then be created.

Identify the Disk

Begin by identifying the disk where you want to create logical partitions. You can use the lsblk or fdisk -l commands to list the available disks and their partitions

lsblk

Launch Parted for the Disk

Use the parted command to launch the interactive interface for the chosen disk (replace 'X' with the appropriate disk identifier):

parted /dev/sdX

Create an Extended Partition

If not already created, you need to establish an extended partition within which the logical partitions will reside.

This step is essential as logical partitions can only exist within an extended partition. Assume the free space is from 50% to 100% of the disk:

(parted) mkpart extended 50% 100%

Create Logical Partitions

Now, within the extended partition, you can create logical partitions. The following example creates a logical partition using ext4 file system from 0% to 25% of the extended partition.

You can also give size using MB format. For that you need to know the free space. You can do that using print free command inside parted.

(parted) mkpart logical ext4 0% 25%

You can repeat this step to create additional logical partitions with different sizes or file systems.

Partitioning Scheme Numbering Convention

In the Master Boot Record (MBR) partitioning scheme, the numbering convention for logical partitions within an extended partition typically starts from 5. This is because the four primary partitions (if they exist) are assigned numbers 1 to 4, and logical partitions are numbered starting from 5.

Here's a common numbering scheme:

• Primary Partition 1: /dev/sdX1

• Primary Partition 2: /dev/sdX2

• Primary Partition 3: /dev/sdX3

• Primary Partition 4: /dev/sdX4

• Extended Partition: /dev/sdX4 (the extended partition is counted as one of the four)

• Logical Partition 5: /dev/sdX5

• Logical Partition 6: /dev/sdX6

• Logical Partition 7: /dev/sdX7, and so on.

Wrapping Up

Thank you for exploring how MBR disk partitioning works with me today. You can dive deeper into the realm of Linux expertise and stay tuned for more insightful content in my future tutorials.

You can follow me on:

• Twitter

• LinkedIn

The data stored in the Web Storage use the key/value pair format, and both data will be stored as strings.

There are two types of storage introduced in the Web Storage API: Local Storage and Session Storage.

In this article, I’m going to show you how to use the Web Storage API and why it’s useful for web developers.

How the Web Storage API Works

The Web Storage API exposes a set of objects and methods that you can use to store data in the browser. The data you store in Web Storage is private, which means no other website can access it.

In Google Chrome, you can view Web Storage by opening the developer tools window and going to the Application tab as shown below:

The web storage location in Google Chrome

In the picture above, you can see that the Storage menu also has other storage types like Indexed DB, Web SQL, and cookies. The Web SQL standard has been deprecated, and IndexedDB is rarely used because it’s too complex. Any data you store in IndexedDB might better be stored on the server.

As for cookies, it’s a more traditional mechanism of storing data that only allows you to store a maximum of 4 KB of data. By contrast, the Local Storage capacity is 10 MB and the session storage has 5 MB capacity.

This is why we’re going to focus only on Local Storage and Session Storage in this article.

Local Storage and Session Storage Explained

Local Storage and Session Storage are the two standard mechanisms supported by the Web Storage API.

Web storage is domain specific, meaning data stored under one domain (netflix.com) can’t be accessed by another domain (www.netflix.com or members.netflix.com)

Web storage is also protocol specific. This means the data you store in a http:// site won’t be available under the https:// site.

The main difference between Local and Session Storage is that Local Storage will store your data forever. If you want to remove the data, you need to use the available method or clear it manually from the Applications tab.

By contrast, the data stored in session storage is only available during the page session. When you close the browser or the tab, the session storage for that specific tab is removed.

Both Local and Session Storage can be accessed through the window object under the variables localStorage and sesionStorage, respectively. Let’s see the methods and properties of these storage types next.

Methods and Properties of Local and Session Storage

Both Local and Session Storage have the same methods and properties. To set a new key/value pair in the Local Storage, you can use the setItem() method of the localStorage object:

localStorage.setItem('firstName', 'Nathan');

If you look into the Local Storage menu in the browser, you should see the data above saved into the storage as follows:

Storing a key/value pair in Local Storage

The key you used in localStorage must be unique. If you set another data with a key that already exists, then the setItem() method will replace the previous value with the new one.

To get the value out of local storage, you need to call the getItem() method and pass the key you used when saving the data. If the key doesn’t exist, then getItem() will return null back:

const firstName = localStorage.getItem('firstName');
console.log(firstName); // Nathan

const lastName = localStorage.getItem('lastName');
console.log(lastName); // null

To remove the data you have in local storage, call the removeItem() method and pass the key pointing to the data you want to remove:

localStorage.removeItem('firstName');

The removeItem() method will always return undefined. When the data you want to remove doesn’t exist, the method simply does nothing.

If you want to clear the storage, you can use the clear() method:

localStorage.clear();

The clear() method removes all key/value pairs from the storage object you are accessing.

Properties of Local and Session Storage

Both storage types have only one property, which is the length property that shows the amount of data stored in them.

sessionStorage.setItem('firstName', 'Nathan');
sessionStorage.setItem('lastName', 'Sebhastian');

console.log(sessionStorage.length); // 2

sessionStorage.clear();
console.log(sessionStorage.length); // 0

And that’s all the methods and properties you can access in localStorage and sessionStorage.

How to Store JSON Strings in Web Storage Storage

Since Web Storage always stores data as strings, you can store complex data as a JSON string, and then convert that string back into an object when you access it.

For example, suppose I want to store the following information about a user:

const user = {
  firstName: 'Nathan',
  lastName: 'Sebhastian',
  url: 'https://codewithnathan.com',
};

At first, I might store the data as a series of key/value pairs like this:

localStorage.setItem('firstName', user.firstName);
localStorage.setItem('lastName', user.lastName);
localStorage.setItem('url', user.url);

But a better way is to convert the JavaScript object into a JSON string, and then store the data under one key as follows:

const user = {
  firstName: 'Nathan',
  lastName: 'Sebhastian',
  url: 'https://codewithnathan.com',
};

const userData = JSON.stringify(user);

localStorage.setItem('user', userData);

Now the local storage will have only one key/value pair with the JSON string as its value. You can open the Applications tab to see this:

Storing a JSON string in Local Storage

When you need the data, call the getItem() and JSON.parse() methods as follows:

const getUser = JSON.parse(localStorage.getItem('user'));

console.log(getUser);
// {firstName: 'Nathan', lastName: 'Sebhastian', url: 'https://codewithnathan.com'}

Here, you can see that the data is returned as a regular JavaScript object.

Local Storage vs Session Storage – Which One to Use?

Based on my experience, localStorage is the preferred Web Storage mechanism because the data will persist as long as you need it to. When you don’t need the data, you can remove it using the removeItem() method.

sessionStorage is only used when you need to store temporary data, like tracking whether a popup box has been shown to the user or not.

But this is also open to discussion because you might not want to show a popup every time the user logs into your web application, but only once. In that case, you should use localStorage instead.

My rule of thumb is to use localStorage first, and sessionStorage when the situation needs it.

Benefits of Using the Web Storage API

Now that you know how the Web Storage API works, you can see that there are some benefits of using it:

1. Storing data on the browser reduces the need to make a server request for a piece of information. This can improve the performance of your web applications.
2. The simple key/value pair format allows you to store user preferences and local settings that should persist between sessions.
3. The Web Storage API is simple to use, providing only a few methods and one property. It’s simple to set and retrieve data using JavaScript
4. It has offline support. By storing necessary data locally, the Web Storage enables your web application to work offline.
5. The Web Storage is also a standardized API, meaning the code you write will work in many different browsers.

But of course, not all data should be stored in the Web Storage API. You still need a server database to keep records that are important for your application.

Conclusion

Web Storage is a useful API that allows you to quickly store and retrieve data from the browser. Using Web Storage, you can store the user’s preferences when accessing your application.

localStorage allows you to store data forever until it’s removed manually, while sessionStorage will persist as long as the browser or tab is open.

Some benefits of using the Web Storage API include reducing server requests, offline support, and a simple API that’s easy to use. It’s also standardized, so it will work on different browsers.

If you enjoyed this article and want to take your JavaScript skills to the next level, I recommend you check out my new book Beginning Modern JavaScript here.

The book is designed to be easy for beginners and accessible to anyone looking to learn JavaScript. It provides a step-by-step gentle guide that will help you understand how to use JavaScript to create a dynamic web application.

Here's my promise: You will actually feel like you understand what you're doing with JavaScript.

See you later!

Session and local storage are the two main types of web storage. They are similar to regular properties objects, but they persist (do not disappear) when the webpage reloads.

This article aims to show you exactly how the two storage objects work in JavaScript. We will also use a To-Do list exercise to practice using web storage in a web app project.

Table of Contents

1. What is the Session Storage Object?
2. What is the Local Storage Object?
3. How to Access the Session and Local Storage Objects
4. What are Web Storage’s Built-In Interfaces?
   • What is web storage’s setItem() method?
   • What is web storage’s key() method?
   • What is web storage’s getItem() method?
   • What is web storage’s length property?
   • What is web storage’s removeItem() method?
   • What is web storage’s clear() method?
5. Time to Practice with Web Storage 🤸‍♂️🏋️‍♀️
   • The Problem
   • Your Exercise
   • Bonus Exercise
6. How Did You Go About Solving the Web Storage Exercise?
   • How to prevent the Session Storage pane’s To-Do items from disappearing on page reload
   • How to prevent the Local Storage pane’s To-Do items from disappearing on page reload or reopen
   • How to auto-display the Session section’s previously added tasks on page reload
   • How to auto-display the Local section’s previously added tasks on page reload or reopen
   • How to check the total items in the browser’s session storage
   • How to display the local storage’s zeroth index item’s name
   • How to empty the browser’s session storage
7. How to Continue Practicing with Web Storage 🧗‍♀️🚀
8. Web Storage vs. Cookies: What is the Difference?
9. Wrapping up

Without further ado, let’s discuss session storage.

What is the Session Storage Object?

The session storage object (window.sessionStorage) stores data that persists for only one session of an opened tab.

In other words, whatever gets stored in the window.sessionStorage object will not disappear on a reload of the web page. Instead, the computer will delete the stored data only when users close the browser tab or window.

Note the following:

• The data stored inside the session storage is per-origin and per-instance. In other words, http://freecodecamp.com’s sessionStorage object is different from https://freecodecamp.com’s sessionStorage object because the two origins use different schemes (http and https).
• Per-instance means per-window or per-tab. In other words, the sessionStorage object’s lifespan expires once users close the instance (window or tab).
• Browsers create a unique page session for each new tab or window. Therefore, users can run multiple instances of an app without interfering with each instance’s session storage. (Note: Cookies do not have good support for running multiple instances of the same app. Such an attempt can cause errors such as double entry of bookings.)
• Session storage is a property of the global Window object. So sessionStorage.setItem() is equivalent to window.sessionStorage.setItem().

What is the Local Storage Object?

The local storage object (window.localStorage) stores data that persists even when users close their browser tab (or window).

In other words, whatever gets stored in the window.localStorage object will not disappear during a reload or reopening of the web page or when users close their browsers. Those data have no expiration time. Browsers never clear them automatically.

The computer will delete the window.localStorage object’s content in the following instances only:

1. When the content gets cleared through JavaScript
2. When the browser’s cache gets cleared

Note the following:

• The window.localStorage object’s storage limit is larger than the window.sessionStorage.
• The data stored inside the local storage is per-origin. In other words, http://freecodecamp.com’s localStorage object is different from https://freecodecamp.com’s localStorage object because the two origins use different schemes (http and https).
• There are inconsistencies with how browsers handle the local storage of documents not served from a web server (for instance, pages with a file: URL scheme). Therefore, the localStorage object may behave differently among browsers when used with non-HTTP URLs, such as file:///document/on/users/local/system.html.
• Local storage is a property of the global Window object. Therefore, localStorage.setItem() is equivalent to window.localStorage.setItem().

How to Access the Session and Local Storage Objects

You can access the two web storages by:

1. Using the same technique as you'd use for accessing regular JavaScript objects
2. Using web storage’s built-in interfaces

For instance, consider the snippet below:

sessionStorage.bestColor = "Green";
sessionStorage["bestColor"] = "Green";
sessionStorage.setItem("bestColor", "Green");

The three statements above do the same thing—they set bestColor’s value. But the third line is recommended because it uses web storage’s setItem() method.

Tip: you should prefer using the web storage’s built-in interfaces to avoid the pitfalls of using objects as key/value stores.

Let’s discuss more on the web storage’s built-in interfaces below.

What are Web Storage’s Built-In Interfaces?

The web storage built-in interfaces are the recommended tools for reading and manipulating a browser’s sessionStorage and localStorage objects.

The six (6) built-in interfaces are:

• setItem()
• key()
• getItem()
• length
• removeItem()
• clear()

Let’s discuss each one now.

What is web storage’s setItem() method?

The setItem() method stores its key and value arguments inside the specified web storage object.

Syntax of the setItem() method

setItem() accepts two required arguments. Here is the syntax:

webStorageObject.setItem(key, value);

• webStorageObject represents the storage object (localStorage or sessionStorage) you wish to manipulate.
• key is the first argument accepted by setItem(). It is a required string argument representing the name of the web storage property you want to create or update.
• value is the second argument accepted by setItem(). It is a required string argument specifying the value of the key you are creating or updating.

Note:

• The key and value arguments are always strings.
• Suppose you provide an integer as a key or value. In that case, browsers will convert them to strings automatically.
• setItem() may display an error message if the storage object is full.

Example 1: How to store data in the session storage object

1. Invoke sessionStorage’s setItem() method.
2. Provide the name and value of the data you wish to store.

// Store color: "Pink" inside the browser's session storage object:
sessionStorage.setItem("color", "Pink");

// Log the session storage object to the console:
console.log(sessionStorage);

// The invocation above will return:
{color: "Pink"}

Try Editing It

Note: Your browser’s session storage may contain additional data if it already uses the storage object to store information.

Example 2: How to store data in the local storage object

1. Invoke localStorage’s setItem() method.
2. Provide the name and value of the data you wish to store.

// Store color: "Pink" inside the browser's local storage object:
localStorage.setItem("color", "Pink");

// Log the local storage object to the console:
console.log(localStorage);

// The invocation above will return:
{color: "Pink"}

Try Editing It

Note:

• Your browser’s local storage may contain additional data if it already uses the storage object to store information.
• It is best to serialize objects before storing them in local or session storage. Otherwise, the computer will store the object as "[object Object]".

Example 3: Browsers use "[object Object]" for non-serialized objects in the web storage

// Store myBio object inside the browser's session storage object:
sessionStorage.setItem("myBio", { name: "Oluwatobi" });

// Log the session storage object to the console:
console.log(sessionStorage);

// The invocation above will return:
{myBio: "[object Object]", length: 1}

Try Editing It

You can see that the computer stored the object as "[object Object]" because we did not serialize it.

Example 4: How to store serialized objects in the web storage

// Store myBio object inside the browser's session storage object:
sessionStorage.setItem("myBio", JSON.stringify({ name: "Oluwatobi" }));

// Log the session storage object to the console:
console.log(sessionStorage);

// The invocation above will return:
{myBio: '{"name":"Oluwatobi"}', length: 1}

Try Editing It

We used JSON.stringify() to convert the object to JSON before storing it in the web storage.

Tip: Learn how to convert JSON to JavaScript objects.

What is web storage’s key() method?

The key() method retrieves a specified web storage item’s name (key).

Syntax of the key() method

key() accepts one required argument. Here is the syntax:

webStorageObject.key(index);

• webStorageObject represents the storage object (localStorage or sessionStorage) whose key you wish to get.
• index is a required argument. It is an integer specifying the index of the item whose key you want to get.

Example 1: How to get the name of an item in the session storage object

1. Invoke sessionStorage’s key() method.
2. Provide the index of the item whose name you wish to get.

// Store carColor: "Pink" inside the browser's session storage object:
sessionStorage.setItem("carColor", "Pink");

// Store pcColor: "Yellow" inside the session storage object:
sessionStorage.setItem("pcColor", "Yellow");

// Store laptopColor: "White" inside the session storage object:
sessionStorage.setItem("laptopColor", "White");

// Get the name of the item at index 1:
sessionStorage.key(1);

Try Editing It

Important: The user-agent defines the order of items in the session storage. In other words, key()’s output may vary based on how the user-agent orders the web storage’s items. So you shouldn't rely on key() to return a constant value.

Example 2: How to get the name of an item in the local storage object

1. Invoke localStorage’s key() method.
2. Provide the index of the item whose name you wish to get.

// Store carColor: "Pink" inside the browser's local storage object:
localStorage.setItem("carColor", "Pink");

// Store pcColor: "Yellow" inside the local storage object:
localStorage.setItem("pcColor", "Yellow");

// Store laptopColor: "White" inside the local storage object:
localStorage.setItem("laptopColor", "White");

// Get the name of the item at index 1:
localStorage.key(1);

Try Editing It

Important: The user-agent defines the order of items in the local storage. In other words, key()’s output may vary based on how the user-agent orders the web storage’s items. So you shouldn't rely on key() to return a constant value.

What is web storage’s getItem() method?

The getItem() method retrieves the value of a specified web storage item.

Syntax of the getItem() method

getItem() accepts one required argument. Here is the syntax:

webStorageObject.getItem(key);

• webStorageObject represents the storage object (localStorage or sessionStorage) whose item you wish to get.
• key is a required argument. It is a string specifying the name of the web storage property whose value you want to get.

Example 1: How to get data from the session storage object

1. Invoke sessionStorage’s getItem() method.
2. Provide the name of the data you wish to get.

// Store color: "Pink" inside the browser's session storage object:
sessionStorage.setItem("color", "Pink");

// Get color's value from the session storage:
sessionStorage.getItem("color");

// The invocation above will return:
"Pink"

Try Editing It

Example 2: How to get data from the local storage object

1. Invoke localStorage’s getItem() method.
2. Provide the name of the data you wish to get.

// Store color: "Pink" inside the browser's local storage object:
localStorage.setItem("color", "Pink");

// Get color's value from the local storage:
localStorage.getItem("color");

// The invocation above will return:
"Pink"

Try Editing It

Note: The getItem() method will return null if its argument does not exist in the specified web storage.

What is web storage’s length property?

The length property returns the number of properties in the specified web storage.

Syntax of the length property

Here is length’s syntax:

webStorageObject.length;

webStorageObject represents the storage object (localStorage or sessionStorage) whose length you wish to verify.

Example 1: How to verify the number of items in the session storage object

Invoke sessionStorage’s length property.

// Store carColor: "Pink" inside the browser's session storage object:
sessionStorage.setItem("carColor", "Pink");

// Store pcColor: "Yellow" inside the session storage object:
sessionStorage.setItem("pcColor", "Yellow");

// Store laptopColor: "White" inside the session storage object:
sessionStorage.setItem("laptopColor", "White");

// Verify the number of items in the session storage:
sessionStorage.length;

// The invocation above may return:
3

Try Editing It

Note: Your sessionStorage.length invocation may return a value greater than 3 if your browser’s session storage already contains some stored information.

Example 2: How to verify the number of items in the local storage object

Invoke localStorage’s length property.

// Store carColor: "Pink" inside the browser's local storage object:
localStorage.setItem("carColor", "Pink");

// Store pcColor: "Yellow" inside the local storage object:
localStorage.setItem("pcColor", "Yellow");

// Store laptopColor: "White" inside the local storage object:
localStorage.setItem("laptopColor", "White");

// Verify the number of items in the local storage:
localStorage.length;

// The invocation above may return:
3

Try Editing It

Note: Your localStorage.length invocation may return a value greater than 3 if your browser's local storage already contains some stored information.

What is web storage’s removeItem() method?

The removeItem() method removes a property from the specified web storage.

Syntax of the removeItem() Method

removeItem() accepts one required argument. Here is the syntax:

webStorageObject.removeItem(key);

• webStorageObject represents the storage object (localStorage or sessionStorage) whose item you wish to remove.
• key is a required argument. It is a string specifying the name of the web storage property you want to remove.

Example 1: How to remove data from the session storage object

1. Invoke sessionStorage’s removeItem() method.
2. Provide the name of the data you wish to remove.

// Store carColor: "Pink" inside the browser's session storage object:
sessionStorage.setItem("carColor", "Pink");

// Store pcColor: "Yellow" inside the session storage object:
sessionStorage.setItem("pcColor", "Yellow");

// Store laptopColor: "White" inside the session storage object:
sessionStorage.setItem("laptopColor", "White");

// Remove the pcColor item from the session storage:
sessionStorage.removeItem("pcColor");

// Confirm whether the pcColor item still exists in the session storage:
sessionStorage.getItem("pcColor");

// The invocation above will return:
null

Try Editing It

Note: The removeItem() method will do nothing if its argument does not exist in the session storage.

Example 2: How to remove data from the local storage object

1. Invoke localStorage’s removeItem() method.
2. Provide the name of the data you wish to remove.

// Store carColor: "Pink" inside the browser's local storage object:
localStorage.setItem("carColor", "Pink");

// Store pcColor: "Yellow" inside the local storage object:
localStorage.setItem("pcColor", "Yellow");

// Store laptopColor: "White" inside the local storage object:
localStorage.setItem("laptopColor", "White");

// Remove the pcColor item from the local storage:
localStorage.removeItem("pcColor");

// Confirm whether the pcColor item still exists in the local storage:
localStorage.getItem("pcColor");

// The invocation above will return:
null

Try Editing It

Note: The removeItem() method will do nothing if its argument does not exist in the local storage.

What is web storage’s clear() method?

The clear() method clears (deletes) all the items in the specified web storage.

Syntax of the clear() Method

clear() accepts no argument. Here is the syntax:

webStorageObject.clear();

webStorageObject represents the storage object (localStorage or sessionStorage) whose items you wish to clear.

Example 1: How to clear all items from the session storage object

Invoke sessionStorage’s clear() method.

// Store carColor: "Pink" inside the browser's session storage object:
sessionStorage.setItem("carColor", "Pink");

// Store pcColor: "Yellow" inside the session storage object:
sessionStorage.setItem("pcColor", "Yellow");

// Store laptopColor: "White" inside the session storage object:
sessionStorage.setItem("laptopColor", "White");

// Clear all items from the session storage:
sessionStorage.clear();

// Confirm whether the session storage still contains any item:
console.log(sessionStorage);

// The invocation above will return:
{length: 0}

Try Editing It

Example 2: How to clear all items from the local storage object

Invoke localStorage’s clear() method.

// Store carColor: "Pink" inside the browser's local storage object:
localStorage.setItem("carColor", "Pink");

// Store pcColor: "Yellow" inside the local storage object:
localStorage.setItem("pcColor", "Yellow");

// Store laptopColor: "White" inside the local storage object:
localStorage.setItem("laptopColor", "White");

// Clear all items from the local storage:
localStorage.clear();

// Confirm whether the local storage still contains any item:
console.log(localStorage);

// The invocation above will return:
{length: 0}

Try Editing It

Now that we know what web storage is and how to access it, we can practice using it in a JavaScript project.

Time to Practice with Web Storage 🤸‍♂️🏋️‍♀️

Consider the following To-Do List app:

The Problem

The issue with the To-Do List app is this:

• Tasks disappear whenever users refresh the webpage.

Your Exercise

Use the appropriate Web Storage APIs to accomplish the following tasks:

1. Prevent the Session pane’s To-Do items from disappearing whenever users reload the browser.
2. Prevent the Local section’s To-Do items from disappearing whenever users reload or close their browser tab (or window).
3. Auto-display the Session section's previously added tasks on page reload.
4. Auto-display the Local section's previously added tasks on page reload (or browser reopen).

Bonus Exercise

Use your browser’s console to:

1. Check the number of items in your browser’s session storage object.
2. Display the name of your local storage’s zeroth index item.
3. Delete all the items in your browser’s session storage.

Try the Web Storage Exercise

Note: You will benefit much more from this tutorial if you attempt the exercise yourself.

If you get stuck, don’t be discouraged. Instead, review the lesson and give it another try.

Once you’ve given it your best shot (you’ll only cheat yourself if you don’t!), we can discuss how I approached the exercise below.

How Did You Go About Solving the Web Storage Exercise?

Below are feasible ways to get the exercise done.

How to prevent the Session Storage pane’s To-Do items from disappearing on page reload

Whenever users click the “Add task” button,

1. Get existing session storage’s content, if any. Otherwise, return an empty array.
2. Merge the existing to-do items with the user’s new input.
3. Add the new to-do list to the browser’s session storage object.

Here’s the code:

sessionAddTaskBtn.addEventListener('click', () => {
  // Get existing session storage's content, if any. Otherwise, return an empty array:
  const currentTodoArray =
    JSON.parse(sessionStorage.getItem('codesweetlyStore')) || [];

  // Merge currentTodoArray with the user's new input:
  const newTodoArray = [
    ...currentTodoArray,
    { checked: false, text: sessionInputEle.value },
  ];

  // Add newTodoArray to the session storage object:
  sessionStorage.setItem('codesweetlyStore', JSON.stringify(newTodoArray));

  const todoLiElements = createTodoLiElements(newTodoArray);
  sessionTodosContainer.replaceChildren(...todoLiElements);
  sessionInputEle.value = '';
});

Try Editing It

Note: The three dots (...) preceding the currentTodoArray variable represent the spread operator. We used it in the newTodoArray object to copy currentTodoArray’s items into newTodoArray.

How to prevent the Local Storage pane’s To-Do items from disappearing on page reload or reopen

1. Get existing local storage’s content, if any. Otherwise, return an empty array.
2. Merge the existing to-do items with the user’s new input.
3. Add the new to-do list to the browser’s local storage object.

Here’s the code:

localAddTaskBtn.addEventListener('click', () => {
  // Get existing local storage's content, if any. Otherwise, return an empty array:
  const currentTodoArray =
    JSON.parse(localStorage.getItem('codesweetlyStore')) || [];

  // Merge currentTodoArray with the user's new input:
  const newTodoArray = [
    ...currentTodoArray,
    { checked: false, text: localInputEle.value },
  ];

  // Add newTodoArray to the local storage object:
  sessionStorage.setItem('codesweetlyStore', JSON.stringify(newTodoArray));

  const todoLiElements = createTodoLiElements(newTodoArray);
  localTodosContainer.replaceChildren(...todoLiElements);
  localInputEle.value = '';
});

Try Editing It

Note: The localTodosContainer.replaceChildren(...todoLiElements) statement tells the browser to replace localTodosContainer’s current children elements with the list of <li>s in the todoLiElements array.

How to auto-display the Session section’s previously added tasks on page reload

Whenever users reload the page,

1. Get existing session storage’s content, if any. Otherwise, return an empty array.
2. Use the retrieved content to create <li> elements.
3. Populate the tasks display space with the <li> elements.

Here’s the code:

window.addEventListener('load', () => {
  // Get existing session storage's content, if any. Otherwise, return an empty array:
  const sessionTodoArray =
    JSON.parse(sessionStorage.getItem('codesweetlyStore')) || [];

  // Use the retrieved sessionTodoArray to create <li> elements:
  const todoLiElements = createTodoLiElements(sessionTodoArray);

  // Populate the tasks display space with the todoLiElements:
  sessionTodosContainer.replaceChildren(...todoLiElements);
});

Try Editing It

How to auto-display the Local section’s previously added tasks on page reload or reopen

Whenever users reload or reopen the page,

1. Get existing local storage’s content, if any. Otherwise, return an empty array.
2. Use the retrieved content to create <li> elements.
3. Populate the tasks display space with the <li> elements.

Here’s the code:

window.addEventListener('load', () => {
  // Get existing local storage's content, if any. Otherwise, return an empty array:
  const localTodoArray =
    JSON.parse(localStorage.getItem('codesweetlyStore')) || [];

  // Use the retrieved localTodoArray to create <li> elements:
  const todoLiElements = createTodoLiElements(localTodoArray);

  // Populate the tasks display space with the todoLiElements:
  localTodosContainer.replaceChildren(...todoLiElements);
});

Try Editing It

How to check the total items in the browser’s session storage

Use session storage’s length property like so:

console.log(sessionStorage.length);

Try Editing It

How to display the local storage’s zeroth index item’s name

Use the local storage’s key() method as follows:

console.log(localStorage.key(0));

Try Editing It

How to empty the browser’s session storage

Use the session storage’s clear() method as follows:

sessionStorage.clear();

How to Continue Practicing with Web Storage 🧗‍♀️🚀

The to-do app still has a lot of potential. For instance, you can:

• Convert it to a React TypeScript application.
• Make it keyboard accessible.
• Allow users to delete or edit individual tasks.
• Allow users to star (mark as important) specific tasks.
• Let users specify due dates.

So, feel free to continue developing what we’ve built in this tutorial so you can better understand the web storage objects.

For instance, here’s my attempt at making the two panes functional:

Before we wrap up our discussion, you should know some differences between web storage and cookies. So, let’s talk about that below.

Web Storage vs. Cookies: What is the Difference?

Web storage and cookies are two main ways to store data locally within a user’s browser. But they work differently. Below are the main distinctions between them.

Storage limit

Cookies: Have 4 kilobytes maximum storage limit.

Web storage: Can store a lot more than 4 kilobytes of data. For instance, Safari 8 can store up to 5 MB, while Firefox 34 permits 10 MB.

Data transfer to the server

Cookies: Transfer data to the server whenever browsers send HTTP requests to the web server.

Web storage: Never transfers data to the server.

Note: It is a waste of users’ bandwidth to send data to the server if such information is needed only by the client (browser), not the server.

Weak integrity and confidentiality

Cookies: Suffer from weak integrity and weak confidentiality issues.

Web storage: Do not suffer from weak integrity and confidentiality issues because it stores data per origin.

Property

Cookies: Cookies are a property of the Document object.

Web storage: Web storage is a property of the Window object.

Expiration

Cookie: You can specify a cookie’s expiration date.

Web storage: Browsers determine web storage’s expiration date.

Retrieving individual data

Cookies: There’s no way to retrieve individual data. You always have to recall all the data to read any single one.

Web storage: You can choose the specific data you wish to retrieve.

The syntax for storing data

Cookies:

document.cookie = "key=value";

Web storage:

webStorageObject.setItem(key, value);

The syntax for reading data

Cookies:

document.cookie;

Web storage:

webStorageObject.getItem(key);

The syntax for removing data

Cookies:

document.cookie = "key=; expires=Thu, 01 May 1930 00:00:00 UTC";

The snippet above deletes the cookie by assigning an empty value to the key property and setting a past expiration date.

Web storage:

webStorageObject.removeItem(key);

Wrapping up

In this article, we discussed how to use web storage and its built-in interfaces. We also used a to-do list project to practice using the local and session storage objects to store data locally and securely within users’ browsers.

Thanks for reading!

And here’s a useful React TypeScript resource:

I wrote a book about Creating NPM Packages!

It is a beginner-friendly book that takes you from zero to creating, testing, and publishing NPM packages like a pro.

This tutorial will explain these types of storage, their real-world use cases, and some trade-offs.

What is Block Storage?

Imagine you have a large bookshelf with many shelves, and each shelf can hold a specific number of pages from a book.

Now, let's say you have a collection of books, but they are all different sizes. To efficiently store these books, you decide to divide them into smaller uniform pieces, called blocks, that fit nicely on the shelves.

Each shelf can only store 100 pages from a book. The Great Gatsby has about 200 pages, so each shelf can only store half the book. Therefore, this single book will be stored in two separate shelves as shown below, with half of the book in one location and the other in another location.

We have defined a block as the maximum number of pages that can be stored in a shelf. In this example, the block size is 100 pages.

With block storage, data is divided into fixed-size blocks, just like the pages of a book in our analogy. These blocks are usually several thousand bytes.

Each block is assigned a unique address, similar to the location of a specific book (or pages of a book) on a shelf. These addresses allow you to quickly find and access individual blocks of data without having to go through the entire storage.

When you want to store or retrieve data using block storage, you interact with the blocks directly. You can write new data to an empty block or overwrite existing data in a block. If you need to retrieve specific information, you can request the block by its unique address, and it will be returned to you.

Hard disk drives (HDD) and solid state drives (SSD) that are attached to a computer physically or via a network are examples of block storage devices.

The main cloud providers all have block storage options:

1. AWS – Elastic Block Storage (EBS)

2. GCP – Persistent Disks

3. Azure – Managed Disks

Block storage devices are usually only attached to a single instance. This is one way in which block storage differs from file storage, which I will explain in the next section.

Physically attached block storage is not persistent, meaning it only lasts as long as the instance is not terminated. Network attached block storage persists beyond the life of the instance.

Bringing back the analogy of storing a fixed number of pages from a book in a bookshelf, block storage will allow you to modify or retrieve specific pages without having to handle the entire book.

However, why would you ever need to do this? Isn’t it better to simply have the shelves store a book in its entirety?

For humans, this is certainly preferred. For computers, storing information in blocks has some advantages.

Since block storage presents raw blocks to the compute instance, the instance has flexibility over how the blocks are managed. This is ideal for applications that require high performance and low latency storage like databases, high performance computing applications and ETL (Extract Transform Load), among other applications.

Block storage devices are also used to store the operating system. They are also bootable. “Bootable” simply refers to the ability of a device to start or initiate the process of loading an operating system or software program when a computer is powered on or restarted.

A bootable device contains the necessary files and data that enable a computer to begin its startup sequence and load the operating system into the computer's memory.

What is File Storage?

Block storage is the lowest level abstraction of storage. It provides a low-level interface where you can read from or write to individual blocks of data. But it does not inherently understand the concept of files, directories, or the hierarchical structure typically associated with file systems.

File storage is an abstraction built on top of block storage. It introduces the concept of files, directories, and a hierarchical structure for organising and managing data.

With file storage, you can group related blocks of data together to form files and organise files within directories. This allows for a more intuitive way to access and manage data, as you can work with files and directories rather than dealing with individual blocks.

Using our bookshelf analogy, with file storage, all you interact with are files and their hierarchies, just like how a book store can organise their books in a structured way to make it easier for customers to find and browse through the selections. Books can be arranged alphabetically by author name, by genre, or a combination of both as shown below.

Books can also be further arranged by sub genres (science fiction, fantasy, mystery etc), bestsellers, new releases, books on sale, staff recommendations and children’s books.

Similarly with file storage, you have flexibility over the hierarchical structure which makes it easier for the users (humans or other applications) to access data.

This hierarchical file structure is just an abstraction. Behind the scenes, the operating system abstracts away the underlying block storage and instead gives the appearance of a file cabinet with a folder-like structure. This simplifies access to applications trying to read or write files on the disk.

Applications don’t need to know the underlying block address to retrieve the files, which makes it easier for the application to interact with the files. This ease of interaction comes with a performance cost, which is acceptable for some use cases.

The main cloud providers all have file storage options:

1. AWS – Elastic File Storage (EFS)

2. GCP – Cloud Filestore

3. Azure – Azure Files

Unlike block storage, multiple compute instances can be mounted on the same file storage device.

What is Object Storage?

This is the newest form of storage on the cloud. Object storage stores all data as objects in a flat structure. There is no hierarchy, unlike in file storage. But an artificial folder-like hierarchy is imparted to block storage to give it the appearance of having a structure.

Object storage is highly scalable. It can store multiple billions of objects. As of 2009, it stored 82 billion objects. It would not be a surprise if this has surpassed trillions of objects as of this writing in 2023.

Objects can be any file. It could be video, audio, image, text file, Excel file, word document, HTML, CSS, XML, JSON, and so on.

Object storage is highly durable – that is, there is a very low probability that any object stored there will be lost.

Object storage offered by cloud providers usually provides 99.999999999% durability over a year. This is colloquially referred to as 11 nines of durability.

Durability is defined as the probability of not losing an object. A storage system that has a durability of 99.999999999% has a 0.000000001% chance of losing a single object in a year. This means that even if you have a million objects stored in object storage, you are likely to only lose a single object in 100,000 years.

This is a remarkable level of durability that is not matched by the other storage systems.

Naturally, all these probabilities are meaningless if Earth is destroyed, since all the servers storing these objects currently reside on a single planet.

The main cloud providers all have object storage options:

1. AWS – S3

2. GCP – Cloud Storage

3. Azure – Azure Blob storage

Patterns & Anti-patterns – When to Use Each Storage Type

The table below summarises the trade-offs between the different storage types plus some use cases.

Note: "Mountable" refers to the capability of connecting or attaching a storage device or file system to a specific location in a computer's file system hierarchy.

When a storage device or file system is "mountable," it means that it can be integrated and made accessible to the operating system and applications running on the computer.

As you can see in the image above, some common use cases for each storage type are as follows:

• Block storage: databases, ETL, high performance computing, OS storage, and boot volume

• File storage: sharing files across many compute instances

• Object storage: large, scalable, and durable storage of different objects (like images, audio, and video), disaster recovery, and archiving data.

Bringing it Together

Let’s imagine a simple scenario where you would need to use block, file, and object storage together.

Imagine you have been tasked with designing the cloud architecture for a law firm. You need to store a large number of evidence files in different formats (audio, video, image, text, Excel, JSON, and so on).

You need to processes these files, extract the useful information, and make it available for further processing by different people. You also need to make it available for direct use to a team of lawyers in different parts of the world.

From a high level, how could you design such a solution?

You can see how to do this in the image below:

First, the raw files could be stored using either object storage or file storage. Object storage is preferable because it is lower cost. It is also ideal for low cost archival storage if you need to keep files for several years.

To process the files, you can run multiple applications on an EC2 instance that uses block storage. The other alternative is to use file storage.

Block storage is preferable because these processing tasks may include things like transcribing audio files, extracting text from images, improving and stabilising video files, a database that extracts data in JSON format and stores it in a relational database, and so on. These are all tasks that require higher performance, which is an ideal use case for block storage.

The processed files are then stored in S3 again before they are loaded into a file system with several instances mounted on it.

The up-to-date processed files must be available for further processing or for direct use by a team of lawyers. Block storage is not ideal here because block storage cannot be shared across multiple instances. Object storage is also not ideal because its not mountable (can’t be attached to a compute instance).

In this case, file storage is ideal because it has non of these constraints – it can be shared across multiple instances and it is mountable.

Summary

In summary, block storage is ideal for high performance applications. File storage is ideal for sharing files across multiple instances.

Like block storage, file storage is mountable – that is, it can be integrated and made accessible to the operating system and applications running on the instance.

Object storage is ideal for low cost durable and scalable storage where high performance is not important.

Thank you for reading!

Governments build their policies around economic and population data. Scientists build their theories around environmental, physical, and biological data. Businesses build their plans around production, sales, and consumer behavior data.

This article was taken from the book, Keeping Up: Backgrounders to All the Big Technology Trends You Can't Afford to Ignore. If you'd prefer to watch this chapter as a video, feel free to follow along here:

Data is being generated at rates previously undreamed of. I've read that the sensors on a pair of General Electric GEnx engines on a Boeing 787 Dreamliner generate a terabyte of data each day.

A single network-connected car (like a Tesla) might upload around 100MB of location, performance, and maintenance-related data on any average day.

Multiply that by the millions of such cars that will soon be in use around the world, and multiply that number by the thousands of other devices that are out there, and the scale of the data "problem" should be clear.

Got plans to add your own data to the flood and you feel the need to save and store it, too? You'll need to be able to explain why you need it so you'll know how it should be done.

I can't help you with that "why," but I think I can give you some useful thoughts about the "how."

The way you store data will depend on what it looks like as it's produced and how you may need to access it later. Where you store your data will depend on how much of it there is, how deeply you'd be impacted by its loss, and how often you'll need to take it out and play with it. Let's take a look at both of those variables.

Data Storage Formats

Since not all data is created equal, it'll make sense to look for the tools and environments that'll most closely match the work you're planning to do. Here are some options:

Spreadsheets

They may be flashy, colorful, consumer-facing applications, but spreadsheets are no lightweights when it comes to serious data processing.

As we'll see in more detail a bit later, spreadsheets have their limitations. But when it comes to presenting data in visually accessible ways, applying mathematical, statistical, and financial operations to that data, and even integrating remote data sources (like stock market quotes), no other tool comes close.

Spreadsheets can import simple, plain text data from files of just about any size as long as the text can be delimited. That is, breaks between data divisions should be marked by some consistent character.

When you import the data, you can specify the appropriate delimiter. Tabs, hard returns, and commas are common delimiting characters. In fact, the popular acronym CSV stands for comma-separated values.

Here's what a few lines of CSV text might look like. Note that the first row contains column headings. Spreadsheets can easily understand how those should be treated differently.

Year,Volume,Rate,Growth
2015,56,10,15
2020,90,11,(2)
2022,109,8,12

Spreadsheets display their data in cells, which are arranged into horizontal rows and vertical columns. You an apply functions to the contents of individual cells or to some or all of the cells in a column or row, and can incorporate values in cells in relative locations.

Data sets within a spreadsheet can be rendered as graphs. Spreadsheets can also be used as web forms where users can input data that's saved for future use.

The most popular spreadsheet is probably Microsoft's Excel, which is part of their Microsoft 365 Office Suite. But feature for feature, the open source Calc that comes with the LibreOffice suite is a viable alternative. Google Sheets is a cloud-based spreadsheet solution that may lack some of the feature depth of the others, but is a strong collaboration tool.

Databases

As a rule, databases are not built for visualizing data in attractive and intuitive formats. And, on their own, they're not known for complex mathematical calculations either. But boy, can they handle extra-large data sets and multi-table relationships.

When I say that databases don't really help you visualize your data, that's generally because they're meant to be used "behind" front end applications in multi-tier deployments.

For instance, an e-commerce website will display web pages where users can browse what you've got for sale, add items to a virtual shopping cart, and check out using their payment information.

The web page itself just draws a graphical interface and shows you where to click your mouse, but the information about the items you're selling - including their price and associated images - are probably retrieved from a backend database whenever the page loads.

Similarly, your selections and, eventually, payment information will be written to a different database. The software process that handles your shipping might later consult the payment database for the shipping address. Databases are there at every stage, but no one will ever actually see them.

Administrating large databases so they're efficient, secure, and reliable takes serious engineering and, in some cases, an enormous amount of money.

Before you build your database deployment, you'll need to know whether your operation requires strong Atomicity, Consistency, Isolation, and Durability (ACID) and support for complex and flexible queries. If it does, you may be looking for a relational database engine like SQL Server, MariaDB, or Amazon's Aurora.

Or perhaps you need speedy performance and would prefer a more flexible schema-less environment (suggesting you'd be better off with a NoSQL solution, like MongoDB or Redis).

SQL, by the way, stands for structured query language - which is an established syntax for using language-like code for interacting with your data.

Counterintuitively, depending on who you ask, NoSQL might not stand for Not SQL. Some prefer to think of it as Not Only SQL instead.

Jupyter Notebook

Don't think you have to consume your data using the same tool that's storing it. It's possible, for example, to import existing data that's stored either locally or at a remote site into an interactive compute environment like a Jupyter Notebook.

The advantage of this kind of setup is that the data can now be addressed within the context of, say, a robust Python programming environment without actually touching - or potentially corrupting - the original source.

The open source JupyterLab is a popular resource for working with large data sets using Python. You can download and build your own JupyterLab or run it remotely through a cloud provider like Amazon's Elastic Map Reduce service, or Microsoft's Azure Notebooks.

For particularly large data sets - especially those that already live in the cloud - an existing cloud platform can make sense.

Data Storage Devices

Although it's not quite this simple, let's say that there are four broad categories of data storage media drives:

• Magnetic tape on open reels, cartridges, or cassettes
• Optical including Compact Disk (CD) and Digital Video Disc (DVD)
• Magnetic media in 2.5 and 3.5 inch drive casing - including spinning hard drives
• Solid state including SSD drives in 2.5 and 3.5 inch drive casing, SD cards, and USB flash drives

A few magnetic tape systems may still exist here and there, but the days of laboriously and slowly copying large data sets to banks of multiple backup tapes - and hoping the backup would actually work - are pretty much over. Trust me: no one is complaining.

CDs and DVDs are headed the same direction. Their maximum capacities are no match for the sheer volume of today's enterprise data needs, and consumers don't make local copies of nearly as many large media files as they once did.

Which leaves spinning magnetic and solid state drives.

Gigabyte for gigabyte, spinning hard drives are probably still a bit less expensive than their solid state equivalents (although the price difference is narrowing), but the performance gains delivered by SSDs are very noticeable.

Some time ago, I realized that I could actually save money by buying smaller capacity SSDs for my personal workstations and laptops instead of larger hard disk drives (HDDs).

Let me explain. The way we use data on our personal computers has changed in recent years. Rather than storing media and software archives locally, we're much more likely to assume they'll be available to stream or download whenever we need them.

For most of us, faster download speeds have made "living in the cloud" easy. So we normally just don't need as much storage space any more.

The 500GB SSD drive plugged into my busy workstation is barely half full - even taking into account the dozen or so virtual machines and the many ISO images I've got there. And the drive cost me less than I would have paid for a one or two terabyte HDD.

One of the primary roles of storage is data backup. Rather than physically transferring backups between media, local data archiving - using either SSD or HDD media - generally works by moving archives across networks.

The trick is designing a backup system that automatically provides you with sufficient duplicates of your archives, rotates them through appropriate life cycles (where, eventually, they're retired and destroyed), and all without generating unnecessary network traffic overhead.

Besides backups, you'll also often want to share data among users working throughout your campus.

Two tools for managing both backups and file sharing are network-attached storage (NAS) and storage area networks (SAN). Their similar names suggest they're in the same business. Trust me: they're not.

Network-attached storage (NAS)

NAS is a relatively simple and inexpensive way to share files across a local network. It works through a standalone server device that contains multiple storage drives. The drives will normally be configured as a Redundant array of inexpensive disks (RAID) array to provide redundancy and performance benefits.

The NAS device connects to the network over ethernet cabling and uses regular TCP/IP networking. Client machines in the LAN will see the NAS resources through standard file sharing protocols like Server Message Block (SMB) and Network File System (NFS).

NAS solutions can be great for smaller home or office environments, but the fun will quickly fade as you grow. NAS devices themselves are generally not powerful enough to handle too much of a client workload, and working with large files over an ethernet network may slow things down.

Storage area network (SAN)

If NAS setups are "relatively simple and inexpensive," SANs are complex and expensive. Not by accident were they designed for large enterprise deployments. As a result of the high end hardware you throw into a NAS system, performance will be great and you'll scale much further.

Rather than ethernet, SANs run through much faster Fibre Channel switches (or, sometimes, the slower iSCSI). They provide block-based storage rather than file systems and are mounted on client machines as local drives.

Data Storage Services

As internet connection speeds have improved, it's become more practical to move at least some data archives to the cloud.

Instead of local backups - which could be lost in a catastrophic event like a fire - data could regularly be saved to online platforms. Once there, you'd have a viable, off-site backup. But, if you wanted, you'd also have access to that data from anywhere on earth. If you work remotely with a distributed team, that can be helpful.

You probably already own and have even collaborated on documents that live on Dropbox, Microsoft 365, or Google Drive. For most people, the primary point of interaction for those services is a web browser.

But serious data management - or even relatively complex and regular file backups - aren't practical within a browser. So cloud computing providers offer storage and archiving services whose administration can be scripted and automated.

Cloud storage services, like Amazon's Simple Storage Service (S3), provide full archive life cycle management. Data that must remain highly available could, for example, be saved to the S3 Standard storage class.

After a few months - when you're less likely to need the data, but must still retain a copy for regulatory reasons - you could move your archive to the cheaper S3 Glacier class. Data in Glacier is secure and durable, but would take much longer to access.

After a full year you might be able to delete it altogether. Better yet, there are simple ways to automate the way your data moves through its life cycle.

All major cloud providers will have their own comparable data storage services. Naturally, prices and exact service features will differ from one another. And, of course, feature and pricing details will often change.

It may not always be practical to transfer data to the cloud over the internet. Extremely large data sets can take a very long time to upload even using fast internet connections.

Sure, if you're lucky enough to have a fibre optics connection giving you one gigabyte/second, then a one terabyte upload would take only two and a half hours or so (assuming no one else was using the connection).

But what about 100 TBs of data (that'll take you more than ten days)? And what if you only get 100MB/second (more than three months)? In cases like these, if you're uploading jumbo-sized archives weekly or have other uses for your internet connection, then uploading isn't an option.

For such cases, you can still get your data into the cloud, but it'll have to find another ride. AWS, as it turns out, offers their Snow Family services.

Snowball is a large, secure storage device. It can be safely shipped to AWS customers, loaded up with dozens of terabytes of data, and then shipped back. Once back home at Amazon, the data will be directly uploaded to a bucket in the customer's account. Alternatively, Snowballs can be kept on-location and used as edge compute devices.

Snowball's big brother is AWS Snowmobile, a 45-foot long secure shipping container capable of handling Exabyte-scale digital migration.

Snowball's little cousin, AWS Snowcone, is a rugged container the size of a tissue box that can handle eight TB of usable storage, along with the possibility of virtual cloud instances and network connectivity to the AWS cloud. Besides transferring your data, Snowcones can be used as highly mobile edge compute devices in their own right.

And that's it for today. Thank you for reading. Now, hopefully you better understand how we store data and what your data storage options are.

YouTube videos of all ten chapters from this book are available here. Lots more tech goodness - in the form of books, courses, and articles - can be had here. And consider taking my AWS, security, and container technology courses here.

In this article, you'll learn what is a web storage, the different types of web storage, and when to use each one of them.

What is Web Storage?

Web storage is an HTML5 feature that allows you to store data in key value pairs in the browser. This enables applications to store data in the client side so you can access it or manipulate it later. All data stored in web storage stays in the browser and is not transferred anywhere else.

Types of web storage

The two main types of web storage are‌‌ local storage‌‌ and session storage‌‌. Each one has its own unique characteristics.

But one thing they have in common is that they store the data in the particular browser the user uses to access the webpage. You won't be able to access the same data through another browser.

Now let's discuss both of them in detail.

Local Storage

Local storage can store 5MB of data per app for the lifetime of the app. Closing the browser will not affect the data in any way – it stays there unless you delete it.

You can only access the local storage object through localStorage. The methods you can use to perform operations on the localStorage object are:

localStorage // to access the localStorage object 
localStorage.setItem('name', 'John') // sets name equal to john localStorage.getItem('name') // "John" 
localStorage.removeItem('name') // removes name from the localStorage localStorage.clear() // clears the localStorage

localStorage.setItem() takes a key and value as parameters and sets a new item in the local storage object equal to the given key value pair.

localStorage.getItem() takes a key as a parameter and returns the value stored to that key in the storage.

localStorage.clear() clears the whole localStorage object.

localStorage.removeItem() takes in a key as a parameter and removes the corresponding key-value pair. ‌‌

Any item that you store in localStorage will be stored as a string. This means that you need to convert other data types such as arrays or objects to strings – otherwise you lose their structure.

See the example below:

const scores = [10, 8, 6, 3, 9] 
const scoresJSON = JSON.stringify(scores) 
localStorage.setItem('scores', scoresJSON) 
localStorage // output >> {scores: '[10, 8, 6, 3, 9]', length: 1}

In the example above, we first created an array score, then converted it into a string using JSON.stringify(), and finally saved the stringified scores array in localStorage.

Take your time to have a look at the output I get when I run the same code snippet in my browser console.

Example code in browser console

Note that the key scores has a value equal to our stringified scores array. But if we don't convert the scores array into a string, our array will loose it's structure and the items will be saved as a string like below:

const scores = [10, 8, 6, 3, 9] 
localStorage.setItem('scores', scores) 
localStorage // output >> {scores: '10, 8, 6, 3, 9', length: 1}

Let's also run the code in the browser console to see what is logs to the console:

Saving an array without converting it into a string

See how our scores array is converted into a string when we do not convert it into a string using JSON.stringify()?

Session Storage

Session storage allows you to store data in the browser depending on the system memory and the data stored in the browser until the browser is closed. In other words, closing the browser will clear all the data stored in session storage.

Like localStorage, you can access session storage by typing sessionStorage in the browser console.

sessionStorage // access the session storage 
sessionStorage.setItem('name', 'John') // add name to session storage with value john 
sessionStorage.getItem('name') // get the name item from session storage sessionStorage.removeItem('name') // remove name item from the session storage sessionStorage.clear() // clear the session storage

sessionStorage.setItem() takes a key and value as parameters and sets a new item in the local storage object equal to the given key value pair.

sessionStorage.getItem() takes key as a parameter and returns the value stored to that key in the storage.

sessionStorage.removeItem() takes in a key as parameter and removes the corresponding key-value pair.

sessionStorage.clear() clears the whole localStorage object. ‌‌

Like localStorage, any item stored in sessionStorage will be stored as a string. This means we need to convert them into strings before storing in the sessionStorage.

Web Storage Example Use Cases

You can use local storage when you want your data to be made available when the user revisits the web page, such as for a shopping cart or game/quiz score. Just keep in mind that the information saved in local storage should not be sensitive.‌‌

You can use session storage when the data that needs to be saved is sensitive. User authentication is an example of data that you would like to clear as soon as the user closes the tab.

Conclusion

In this article, you have learned the modern ways of storing temporary data in the browser. I hope this has helped you understand how and when to use both types of web storage in your projects.

Interested in connecting on LinkedIn? Hit me up at Tooba Jamal.

The non-volatile nature of hard drives means they don’t lose data, even if power is lost. Due to this, they help computers store files and other data for a long time – as long as they don’t get damaged or corrupted.

Since the first release of hard drives by IBM in 1956, hard drives have evolved from being the size of a refrigerator and having a storage capacity of just 5MB to ones that are pocket-sized and have up to 4 TB of storage capacity.

In this article, I will cover the different types of hard drives so you can choose the best for your computer.

But firstly, let's look at interfaces – the connecting part between the computer and the hard drive.

Types of Interfaces

• Parallel Advanced Technology Attachment (PATA)
• Serial Advanced Technology Attachment (SATA)
• Small Computer System Interface (SCSI)
• NVMe (Non-volatile Memory Express)

These names come from the way they connect to the computer. So, there are PATA hard drives, SATA hard drives, SCSI hard drives, and NVMe drives.

In this article, I’m now going to elaborate on each of these interfaces and types of hard drives as concisely as possible.

Parallel Advanced Technology Attachment (PATA)

The PATA interfaces were first introduced to the market by Compaq and Western Digital in 1986. They can have up to 80GB capacity and transfer data as fast as 133 MB/S.

They were named Parallel Advanced Technology Attachment because they use a parallel ATA interface to connect to the computer. Apart from PATA, they are also called Integrated Drive Electronics (IDE) and Enhanced Integrated Drive Electronics (EIDE).

PATA interfaces are made of mechanical moving parts and are based on parallel signaling technology – meaning they transmit multiple bits of data simultaneously.

Serial Advanced Technology Attachment (SATA)

In recent times, a lot of desktop and laptop computers have gotten SATA interfaces because they have superseded PATA interfaces in size, power consumption, and even better pricing.

The mode of connection to a computer remains the same as PATA, but instead of parallel signaling technology for data transmission, they use serial signaling technology. This means that they transfer data one bit at a time.

A notable advantage SATA interfaces have over PATA interfaces is the transmission of data at a rate of 150 – 300 MB/S. In addition, they have thinner cables and a cable limit of 1 meter.

Small Computer System Interface (SCSI)

SCSI interface is an upgrade over SATA and PATA interfaces for many reasons such as round-the-clock operations, speed, storage, and several others.

For connection, SCSI hard drives use a small computer system interface – which is a standard for connecting peripheral devices such as printers, scanners, and others.

Best of all, they allow the connection of peripheral devices such as printers, scanners, and other hard drives. In addition, they transmit data at 320 MB/S and you can connect them internally or externally.

Connections through SCSI on personal computers have now been replaced by the Universal Serial BUS (USB). This means that SCSI is no longer used as consumer hardware.

NVMe (Non-volatile Memory Express)

NVMe interface is a revolution. It is built for high demanding and computing-intensive tasks So, it is an upgrade over the interfaces already discussed.

Because of its high performance and scalability, you'll find it mostly in data centers and newer computers. There are now SSDs using the NVMe interface.

Its many advantages over other interfaces are:

• lower power consumption
• multiple command queues
• reduced latency and input/output overhead – leading to better performance
• utilization of CPU to full potential

One of the disadvantages of NVMe interfaces is that they're more expensive than other interfaces.

Types of Hard Drives

Currently, there are two common types of hard drive:

• Hard Disk Drive (HDD)
• Solid State Drive (SSD)

A hard drive utilizes any of the interfaces such as PATA, SATA, SCSI. Today, there are SSDs that utilize the NVMe interface.

Let's look at the two types of hard drives in detail.

Hard Disk Drive (HDD)

Hard disk drives are not that new and you will find them installed in older computers. They consist of a moving part that rotates during reading and writing operations, so they usually make noise.

These days, you may not see many newer desktop and laptop computers with hard disk drives, but they remain popular because of their storage space. 16 GB, 32 GB, and 64 GB are the lowest you can find out there. There are 500 GB, 1 TB, 2TB, and more than 4 TB hard drives.

This enormous storage space is the reason hard disk drives may go nowhere any time soon. There are even external hard disk drives you can use to backup your files.

Solid State Drive (SSD)

SSD hard drives are one of the latest hard drive technologies at the time of writing this article. Most newer computers have SSDs instead of HDDs.

Unlike the hard drive technologies before SSD drives, they don’t consist of moving parts and they don’t use magnetism for storing data. So, they don't make noise like hard disk drives (HDDs).

Instead, they use integrated circuits (ICs) just like third-generation computers. This makes them more durable, faster, and less prone to damage and corruption.

SSD hard drives have a notable advantage of transferring data at speed of 550 MB/S and allow faster boot times than the types of hard drives before them.

Solid state drives also have enormous sizes, but they are very expensive when compared to hard disk drives.

Conclusion

This article explained what different types of hard drives are and how they work, so you can learn more about hard drives and choose the best one for your needs.

From PATA to SATA, SCSI and NVMe, hard drive interfaces continue to evolve and research to make better ones is ongoing.

There is a new variation of SSD hard drives called NVMe (Non-volatile Memory Express) SSDs that has the capacity to transfer data as fast as 3.5 GB/S. This makes them the best choice for video editing and high-resolution gaming, though they require more power than the actual SSDs.

Thank you for reading.

If you find this article helpful, don’t hesitate to share it with your friends and family.

It's a bit tricky to explain what exactly a file system is in just one sentence.

That's why I decided to write an article about it. This post is meant to be a high-level overview of file systems. But I'll sneak into the lower-level concepts as well, as long as it doesn't get boring. :)

What is a file system?

Let's start with a simple definition:

A file system defines how files are named, stored, and retrieved from a storage device.

Every time you open a file on your computer or smart device, your operating system uses its file system internally to load it from the storage device.

Or when you copy, edit, or delete a file, the file system handles it under the hood.

Whenever you download a file or access a web page over the Internet, a file system is involved too.

For instance, if you access a page on freeCodeCamp, your browser sends an HTTP request to freeCodeCamp's server to fetch the page. If the requested resource is a file, it's fetched from a file system.

When people talk about file systems, they might refer to different aspects of a file system depending on the context - that's where things start to seem knotty.

And you might end up asking yourself, WHAT IS A FILE SYSTEM ANYWAY? 🤯

This guide helps you understand file systems in many contexts. I'll cover partitioning and booting too!

To keep this guide manageable, I'll concentrate on Unix-like environments when explaining the lower-level concepts or console commands.

However, these concepts remain relevant to other environments and file systems.

Why do we need a file system in the first place, you may ask?

Well, without a file system, the storage device would contain a big chunk of data stored back to back, and the operating system wouldn't be able to tell them apart.

The term file system takes its name from the old paper-based data management systems, where we kept documents as files and put them into directories.

Imagine a room with piles of papers scattered all over the place.

A storage device without a file system would be in the same situation - and it would be a useless electronic device.

However, a file system changes everything:

A file system isn't just a bookkeeping feature, though.

Space management, metadata, data encryption, file access control, and data integrity are the responsibilities of the file system too.

Everything begins with partitioning

Storage devices must be partitioned and formatted before the first use.

But what is partitioning?

Partitioning is splitting a storage device into several logical regions, so they can be managed separately as if they are separate storage devices.

We usually do partitioning by a disk management tool provided by operating systems, or as a command-line tool provided by the system's firmware (I'll explain what firmware is).

A storage device should have at least one partition or more if needed.

Why should we split the storage devices into multiple partitions anyways?

The reason is that we don't want to manage the whole storage space as a single unit and for a single purpose.

It's just like how we partition our workspace, to separate (and isolate) meeting rooms, conference rooms, and various teams.

For example, a basic Linux installation has three partitions: one partition dedicated to the operating system, one for the users' files, and an optional swap partition.

A swap partition works as the RAM extension when RAM runs out of space.

For instance, the OS might move a chunk of data (temporarily) from RAM to the swap partition to free up some space on the RAM.

Operating systems continuously use various memory management techniques to ensure every process has enough memory space to run.

File systems on Windows and Mac have a similar layout, but they don't use a dedicated swap partition; Instead, they manage to swap within the partition on which you've installed your operating system.

On a computer with multiple partitions, you can install several operating systems, and every time choose a different operating system to boot up your system with.

The recovery and diagnostic utilities reside in dedicated partitions too.

For instance, to boot up a MacBook in recovery mode, you need to hold Command + R as soon as you restart (or turn on) your MacBook. By doing so, you instruct the system's firmware to boot up with a partition that contains the recovery program.

Partitioning isn't just a way of installing multiple operating systems and tools, though; It also helps us keep critical system files apart from ordinary ones.

So no matter how many games you install on your computer, it won't have any effect on the operating system's performance - since they reside in different partitions.

Back to the office example, having a call center and a tech team in a common area would harm both teams' productivity because each team has its own requirements to be efficient.

For instance, the tech team would appreciate a quieter area.

Some operating systems, like Windows, assign a drive letter (A, B, C, or D) to the partitions. For instance, the primary partition on Windows (on which Windows is installed) is known as C:, or drive C.

In Unix-like operating systems, however, partitions appear as ordinary directories under the root directory - we'll cover this later.

In the next section, we'll dive deeper into partitioning and get to know two concepts that will change your perspective on file systems: system firmware and booting.

Are you ready?

Away we go! 🏊‍♂️

Partitioning schemes, system firmware, and booting

When partitioning a storage device, we have two partitioning methods (or schemes 🙄) to choose from:

• Master boot record (MBR) Scheme
• GUID Partition Table (GPT) Scheme

Regardless of what partitioning scheme you choose, the first few blocks on the storage device will always contain critical data about your partitions.

The system's firmware uses these data structures to boot up the operating system on a partition.

Wait, what is the system firmware? You may ask.

Here's an explanation:

A firmware is a low-level software embedded into electronic devices to operate the device, or bootstrap another program to do it.

Firmware exists in computers, peripherals (keyboards, mice, and printers), or even electronic home appliances.

In computers, the firmware provides a standard interface for complex software like an operating system to boot up and work with hardware components.

However, on simpler systems like a printer, the firmware is the operating system. The menu you use on your printer is the interface of its firmware.

Hardware manufacturers make firmware based on two specifications:

• Basic Input/Output (BIOS)
• Unified Extensible Firmware Interface (UEFI)

Firmwares - BIOS-based or UEFI-based - reside on a non-volatile memory, like a flash ROM attached to the motherboard.

[CC BY 2.0](https://www.flickr.com/photos/computerhotline/5794340306">BIOS By Thomas Bresson, Licensed under <a href="https://creativecommons.org/licenses/by/2.0/)

When you press the power button on your computer, the firmware is the first program to run.

The mission of the firmware (among other things) is to boot up the computer, run the operating system, and pass it the control of the whole system.

A firmware also runs pre-OS environments (with network support), like recovery or diagnostic tools, or even a shell to run text-based commands.

The first few screens you see before your Windows logo appears are the output of your computer's firmware, verifying the health of hardware components and the memory.

The initial check is confirmed with a beep (usually on PCs), indicating everything is good to go.

MBR partitioning and BIOS-based firmware

MBR partitioning scheme is a part of the BIOS specifications and is used by BIOS-based firmware.

On MBR-partitioned disks, the first sector on the storage device contains essential data to boot up the system.

This sector is called MBR.

MBR contains the following information:

• The boot loader, which is a simple program (in machine code) to initiate the first stage of the booting process
• A partition table, which contains information about your partitions.

BIOS-based firmware boots the system differently than UEFI-based firmware.

Here's how it works:

Once the system is powered on, the BIOS firmware starts and loads the boot loader program (contained in MBR) onto memory. Once the program is on the memory, the CPU begins executing it.

Having the boot loader and the partition table in a predefined location like MBR enables BIOS to boot up the system without having to deal with any file.

If you are curious about how the CPU executes the instructions residing in the memory, you can read this beginner-friendly and fun guide on how the CPU works.

The boot loader code in the MBR takes between 434 bytes to 446 bytes of the MBR space (out of 512b). Additionally, 64 bytes are allocated to the partition table, which can contain information about a maximum of four partitions.

446 bytes isn't big enough to accommodate too much code, though. That said, sophisticated boot loaders like GRUB 2 on Linux split their functionality into pieces or stages.

The smallest piece of code known as the first-stage boot loader is stored in the MBR. It's usually a simple program, which doesn't require much space.

The responsibility of the first-stage boot loader is to initiate the next (and more complicated) stages of the booting process.

Immediately after the MBR, and before the first partition starts, there's a small space, around 1MB, called the MBR gap.

MBR gap can be used to place another piece of the boot loader program if needed.

A boot loader, such as GRUB 2, uses the MBR gap to store another stage of its functionality. GRUB calls this the stage 1.5 boot loader, which contains a file system driver.

Stage 1.5 enables the next stages of GRUB to understand the concept of files, rather than loading raw instructions from the storage device (like the first-stage boot loader).

The second stage boot loader, which is now capable of working with files, can load the operating system's boot loader file to boot up the respective operating system.

This is when the operating system's logo fades in...

Here's the layout of an MBR-partition storage device:

And if we magnify the MBR, its content would look like this:

Although MBR is simple and widely supported, it has some limitations 😑.

MBR's data structure limits the number of partitions to only four primary partitions.

A common workaround is to make an extended partition beside the primary partitions, as long as the total number of partitions won't exceed four.

An extended partition can be split into multiple logical partitions. Making extended partitions is different across operating systems. Over this quick guide Microsoft explains how it should be done on Windows.

When making a partition, you can choose between primary and extended.

After this is solved, we'll encounter the second limitation.

Each partition can be a maximum of 2TiB 🙄.

And wait, there's more!

The content of the MBR sector has no backup 😱, meaning if MBR gets corrupted due to an unexpected reason, we'll have to find a way to recycle that useless piece of hardware.

This is where GPT partitioning stands out 😎.

GPT partitioning and UEFI-based firmware

The GPT partitioning scheme is more sophisticated than MBR and doesn't have the limitations of MBR.

For instance, you can have as many partitions as your operating system allows.

And every partition can be the size of the biggest storage device available in the market - actually a lot more.

GPT is gradually replacing MBR, although MBR is still widely supported across old PCs and new ones.

As mentioned earlier, GPT is a part of the UEFI specification, which is replacing the good old BIOS.

That means that UEFI-based firmware uses a GPT-partitioned storage device to handle the booting process.

Many hardware and operating systems now support UEFI and use the GPT scheme to partition storage devices.

In the GPT partitioning scheme, the first sector of the storage device is reserved for compatibility reasons with BIOS-based systems. The reason is some systems might still use a BIOS-based firmware but have a GPT-partitioned storage device.

This sector is called Protective MBR. (This is where the first-stage boot loader would reside in an MBR-partitioned disk)

After this first sector, the GPT data structures are stored, including the GPT header and the partition entries.

The GPT entries and the GPT header are backed up at the end of the storage device, so they can be recovered if the primary copy gets corrupted.

This backup is called Secondary GPT.

The layout of a GPT-partitioned storage device looks like this:

_[CC BY-SA 2.5](https://commons.wikimedia.org/wiki/File:GUID_Partition_Table_Scheme.svg">GUID Partition Table Scheme By Kbolino, Licensed under <a href="https://creativecommons.org/licenses/by-sa/2.5/)_

In GPT, all the booting services (boot loaders, boot managers, pre-os environments, and shells) live in a dedicated partition called EFI System Partition (ESP), which UEFI firmware can use.

ESP even has its own file system, which is a specific version of FAT. On Linux, ESP resides under the /sys/firmware/efi path.

If this path cannot be found on your system, then your firmware is probably BIOS-based firmware.

To check it out, you can try to change the directory to the ESP mount point, like so:

cd /sys/firmware/efi

UEFI-based firmware assumes that the storage device is partitioned with GPT and looks up the ESP in the GPT partition table.

Once the EFI partition is found, it looks for the configured boot loader - usually, a file ending with .efi.

UEFI-based firmware gets the booting configuration from NVRAM (a non-volatile RAM).

NVRAM contains the booting settings and paths to the operating system boot loader files.

UEFI firmware can do a BIOS-style boot too (to boot the system from an MBR disk) if configured accordingly.

You can use the parted command on Linux to see what partitioning scheme is used for a storage device.

sudo parted -l

And the output would be something like this:

Model: Virtio Block Device (virtblk)
Disk /dev/vda: 172GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system  Name  Flags
14      1049kB  5243kB  4194kB                     bios_grub
15      5243kB  116MB   111MB   fat32              msftdata
 1      116MB   172GB   172GB   ext4

Based on the above output, the storage device's ID is /dev/vda with a capacity of 172GB. The storage device is partitioned based on GPT and has three partitions; The second and third partitions are formatted based on the FAT32 and EXT4 file systems respectively.

Having a BIOS GRUB partition implies the firmware is still BIOS-based firmware.

Let's confirm that with the dmidecode command like so:

sudo dmidecode -t 0

And the output would be:

# dmidecode 3.2
Getting SMBIOS data from sysfs.
SMBIOS 2.4 present.

...

✅ Confirmed!

Formatting partitions

When partitioning is done, the partitions should be formatted.

Most operating systems allow you to format a partition based on a set of file systems.

For instance, if you are formatting a partition on Windows, you can choose between FAT32, NTFS, and exFAT file systems.

Formatting involves the creation of various data structures and metadata used to manage files within a partition.

These data structures are one aspect of a file system.

Let's take the NTFS file system as an example.

When you format a partition to NTFS, the formatting process places the key NTFS data structures and the Master file table (MFT) on the partition.

Alright, let's get back file systems with our new background about partitioning, formatting, and booting.

How it started, how it's going

A file system is a set of data structures, interfaces, abstractions, and APIs that work together to manage any type of file on any type of storage device, in a consistent manner.

Each operating system uses a particular file system to manage the files.

In the early days, Microsoft used FAT (FAT12, FAT16, and FAT32) in the MS-DOS and Windows 9x family.

Starting from Windows NT 3.1, Microsoft developed New Technology File System (NTFS), which had many advantages over FAT32, such as supporting bigger files, allowing longer filenames, data encryption, access management, journaling, and a lot more.

NTFS has been the default file system of the Window NT family (2000, XP, Vista, 7, 10, etc.) ever since.

NTFS isn’t suitable for non-Windows environments, though 🤷🏻.

For instance, you can only read the content of an NTFS-formatted storage device (like flash memory) on a Mac OS, but you won’t be able to write anything to it - unless you install an NTFS driver with write support.

Or you can just use the exFat file system.

Extended File Allocation Table (exFAT) is a lighter version of NTFS created by Microsoft in 2006.

exFAT was designed for high-capacity removable devices, such as external hard disks, USB drives, and memory cards.

exFAT is the default file system used by SDXC cards.

Unlike NTFS, exFAT has read and write support on Non-Windows environments as well, including Mac OS — making it the best cross-platform file system for high-capacity removable storage devices.

So basically, if you have a removable disk you want to use on Windows, Mac, and Linux, you need to format it to exFAT.

Apple has also developed and used various file systems over the years, including
Hierarchical File System (HFS), HFS+, and recently Apple File System (APFS).

Just like NTFS, APFS is a journaling file system and has been in use since the launch of OS X High Sierra in 2017.

But how about file systems in Linux distributions?

The Extended File System (ext) family of file systems was created for the Linux kernel - the core of the Linux operating system.

The first version of ext was released in 1991, but soon after, it was replaced by the second extended file system (ext2) in 1993.

In the 2000s, the third extended filesystem (ext3) and fourth extended filesystem (ext4) were developed for Linux with journaling capability.

ext4 is now the default file system in many distributions of Linux, including Debian and Ubuntu.

You can use the findmnt command on Linux to list your ext4-formatted partitions:

findmnt -lo source,target,fstype,used -t ext4

The output would be something like:

SOURCE    TARGET FSTYPE  USED
/dev/vda1 /      ext4    3.6G

Architecture of file systems

A file system installed on an operating system consists of three layers:

• Physical file system
• Virtual file system
• Logical file system

These layers can be implemented as independent or tightly coupled abstractions.

When people talk about file systems, they refer to one of these layers or all three as one unit.

Although these layers are different across operating systems, the concept is the same.

The physical layer is the concrete implementation of a file system; It's responsible for data storage and retrieval and space management on the storage device (or precisely: partitions).

The physical file system interacts with the storage hardware via device drivers.

The next layer is the virtual file system or VFS.

The virtual file system provides a consistent view of various file systems mounted on the same operating system.

So does this mean an operating system can use multiple file systems at the same time?

The answer is yes!

It's common for a removable storage medium to have a different file system than that of a computer.

For instance, on Windows (which uses NTFS as the primary file system), a flash memory might have been formatted to exFAT or FAT32.

That said, the operating system should provide a unified interface between computer programs (file explorers and other apps that work with files) and the different mounted file systems (such as NTFS, APFS, ext4, FAT32, exFAT, and UDF).

For instance, when you open up your file explorer program, you can copy an image from an ext4 file system and paste it over to your exFAT-formatted flash memory - without having to know that files are managed differently under the hood.

This convenient layer between the user (you) and the underlying file systems is provided by the VFS.

A VFS defines a contract that all physical file systems must implement to be supported by that operating system.

However, this compliance isn't built into the file system core, meaning the source code of a file system doesn't include support for every operating system's VFS.

Instead, it uses a file system driver to adhere to the VFS rules of every file system. A driver is a program that enables software to communicate with another software or hardware.

Although VFS is responsible for providing a standard interface between programs and various file systems, computer programs don't interact with VFS directly.

Instead, they use a unified API between programs and the VFS.

Can you guess what it is?

Yes, we're talking about the logical file system.

The logical file system is the user-facing part of a file system, which provides an API to enable user programs to perform various file operations, such as OPEN, READ, and WRITE, without having to deal with any storage hardware.

On the other hand, VFS provides a bridge between the logical layer (which programs interact with) and a set of the physical layer of various file systems.

A high-level architecture of the file system layers

What does it mean to mount a file system?

On Unix-like systems, the VFS assigns a device ID (for instance, dev/disk1s1) to each partition or removable storage device.

Then, it creates a virtual directory tree and puts the content of each device under that directory tree as separate directories.

The act of assigning a directory to a storage device (under the root directory tree) is called mounting, and the assigned directory is called a mount point.

That said, on a Unix-like operating system, all partitions and removable storage devices appear as if they are directories under the root directory.

For instance, on Linux, the mounting points for a removable device (such as a memory card), are usually under the /media directory.

That said, once a flash memory is attached to the system, and consequently, auto mounted at the default mounting point (/media in this case), its content would be available under the /media directory.

However, there are times you need to mount a file system manually.

On Linux, it’s done like so:

mount /dev/disk1s1 /media/usb

In the above command, the first parameter is the device ID (/dev/disk1s1), and the second parameter (/media/usb) is the mount point.

Please note that the mount point should already exist as a directory.

If it doesn’t, it has to be created first:

mkdir -p /media/usb
mount /dev/disk1s1 /media/usb

If the mount-point directory already contains files, those files will be hidden for as long as the device is mounted.

Files metadata

File metadata is a data structure that contains data about a file, such as:

• File size
• Timestamps, like creation date, last accessed date, and modification date
• The file's owner
• The file's mode (who can do what with the file)
• What blocks on the partition are allocated to the file
• and a lot more

Metadata isn’t stored with the file content, though. Instead, it’s stored in a different place on the disk - but associated with the file.

In Unix-like systems, the metadata is in the form of data structures, called inode.

Inodes are identified by a unique number called the inode number.

Inodes are associated with files in a table called inode tables.

Each file on the storage device has an inode, which contains information about it such as the time it was created, modified, etc.

The inode also includes the address of the blocks allocated to the file; On the other hand, where exactly it's located on the storage device

In an ext4 inode, the address of the allocated blocks is stored as a set of data structures called extents (within the inode).

Each extent contains the address of the first data block allocated to the file and the number of the continuous blocks that the file has occupied.

Whenever you open a file on Linux, its name is first resolved to an inode number.

Having the inode number, the file system fetches the respective inode from the inode table.

Once the inode is fetched, the file system starts to compose the file from the data blocks registered in the inode.

You can use the df command with the -i parameter on Linux to see the inodes (total, used, and free) in your partitions:

df -i

The output would look like this:

udev           4116100    378 4115722    1% /dev
tmpfs          4118422    528 4117894    1% /run
/dev/vda1      6451200 175101 6276099    3% /

As you can see, the partition /dev/vda1 has a total number of 6,451,200 inodes, of which 3% have been used (175,101 inodes).

To see the inodes associated with files in a directory, you can use the ls command with -il parameters.

ls -li

And the output would be:

1303834 -rw-r--r--  1 root www-data  2502 Jul  8  2019 wp-links-opml.php
1303835 -rw-r--r--  1 root www-data  3306 Jul  8  2019 wp-load.php
1303836 -rw-r--r--  1 root www-data 39551 Jul  8  2019 wp-login.php
1303837 -rw-r--r--  1 root www-data  8403 Jul  8  2019 wp-mail.php
1303838 -rw-r--r--  1 root www-data 18962 Jul  8  2019 wp-settings.php

The first column is the inode number associated with each file.

The number of inodes on a partition is decided when you format a partition. That said, as long as you have free space and unused inodes, you can store files on your storage device.

It's unlikely that a personal Linux OS would run out of inodes. However, enterprise services that deal with a large number of files (like mail servers) have to manage their inode quota smartly.

On NTFS, the metadata is stored differently, though.

NTFS keeps file information in a data structure called the Master File Table (MFT).

Every file has at least one entry in MFT, which contains everything about it, including its location on the storage device - similar to the inodes table.

On most operating systems, you can grab metadata via the graphical user interface.

For instance, when you right-click on a file on Mac OS, and select Get Info (Properties in Windows), a window appears with information about the file. This information is fetched from the respective file’s metadata.

Space Management

Storage devices are divided into fixed-sized blocks called sectors.

A sector is the minimum storage unit on a storage device and is between 512 bytes and 4096 bytes (Advanced Format).

However, file systems use a high-level concept as the storage unit, called blocks.

Blocks are an abstraction over physical sectors; Each block usually consists of multiple sectors.

Depending on the file size, the file system allocates one or more blocks to each file.

Speaking of space management, the file system is aware of every used and unused block on the partitions, so it’ll be able to allocate space to new files or fetch the existing ones when requested.

The most basic storage unit in ext4-formatted partitions is the block. However, the contiguous blocks are grouped into block groups for easier management.

The layout of a block group within an ext4 partition

Each block group has its own data structures and data blocks.

Here are the data structures a block group can contain:

• Super Block: a metadata repository, which contains metadata about the entire file system, such as the total number of blocks in the file system, total blocks in block groups, inodes, and more. Not all block groups contain the superblock, though. A certain number of block groups store a copy of the super as a backup.
• Group Descriptors: Group descriptors also contain bookkeeping information for each block group
• Inode Bitmap: Each block group has its own inode quota for storing files. A block bitmap is a data structure used to identify used and unused inodes within the block group. 1 denotes used and 0 denotes unused inode objects.
• Block Bitmap: a data structure used to identify used & unused data blocks within the block group. 1 denotes used and 0 denotes unused data blocks
• Inode Table: a data structure that defines the relation of files and their inodes. The number of inodes stored in this area is related to the block size used by the file system.
• Data Blocks: This is the zone within the block group where file contents are stored.

Ext4 file system even takes one step further (comparing to ext3), and organizes block groups into a bigger group called flex block groups.

The data structures of each block group, including the block bitmap, inode bitmap, and inode table, are concatenated and stored in the first block group within each flex block group.

Having all the data structures concatenated in one block group (the first one) frees up more contiguous data blocks on other block groups within each flex block group.

These concepts might be confusing, but you don't have to master every bit of them. It's just to depict the depth of file systems.

The layout of the first block group looks like this:

The layout of the first block in an ext4 flex block group

When a file is being written to a disk, it is written to one or more blocks within a block group.

Managing files at the block group level improves the performance of the file system significantly, as opposed to organizing files as one unit.

Size vs size on disk

Have you ever noticed that your file explorer displays two different sizes for each file: size, and size on disk.

Size and Size on disk

Why are size and size on disk slightly different? You may ask.

Here’s an explanation:

We already know depending on the file size, one or more blocks are allocated to a file.

One block is the minimum space that can be allocated to a file. This means the remaining space of a partially-filled block cannot be used by another file. This is the rule!

Since the size of the file isn't an integer multiple of blocks, the last block might be partially used, and the remaining space would remain unused - or would be filled with zeros.

So "size" is basically the actual file size, while "size on disk" is the space it has occupied, even though it’s not using it all.

You can use the du command on Linux to see it yourself.

du -b "some-file.txt"

The output would be something like this:

623 icon-link.svg

And to check the size on disk:

du -B 1 "icon-link.svg"

Which will result in:

4096    icon-link.svg

Based on the output, the allocated block is about 4kb, while the actual file size is 623 bytes. This means each block size on this operating system is 4kb.

What is disk fragmentation?

Over time, new files are written to the disk, existing files get bigger, shrunk, or deleted.

These frequent changes in the storage medium leave many small gaps (empty spaces) between files. These gaps are due to the same reason file size and file size on disk are different. Some files won't fill up the full block, and lots of space will be wasted. And over time there' won't be enough consequent blocks to store new files.

That's when new files need to be stored as fragments.

File Fragmentation occurs when a file is stored as fragments on the storage device because the file system cannot find enough contiguous blocks to store the whole file in a row.

An example of a fragmented and non-fragmented file

Let's make it more clear with an example.

Imagine you have a Word document named myfile.docx.

myfile.docx is initially stored in a few contiguous blocks on the disk; Let's say this is how the blocks are named: LBA250, LBA251, and LBA252.

Now, if you add more content to myfile.docx and save it, it will need to occupy more blocks on the storage medium.

Since myfile.docx is currently stored on LBA250, LBA251, and LBA252, the new content should preferably sit within LBA253 and so forth - depending on how many more blocks are needed to accommodate the new changes.

Now, imagine LBA253 is already taken by another file (maybe it’s the first block of another file). In that case, the new content of myfile.docx has to be stored on different blocks somewhere else on the disks, for instance, LBA312 and LBA313.

myfile.docx got fragmented 💔.

File fragmentation puts a burden on the file system because every time a fragmented file is requested by a user program, the file system needs to collect every piece of the file from various locations on a disk.

This overhead applies to saving the file back to the disk as well.

The fragmentation might also occur when a file is written to the disk for the first time, probably because the file is huge and not many continuous blocks are left on the partition.

Fragmentation is one of the reasons some operating systems get slow as the file system ages.

Should We Care About Fragmentation these days?

The short answer is: not anymore!

Modern file systems use smart algorithms to avoid (or early-detect) fragmentation as much as possible.

Ext4 also does some sort of preallocation, which involves reserving blocks for a file before they are actually needed - making sure the file won't get fragmented if it gets bigger over time.

The number of the preallocated blocks is defined in the length field of the file's extent of its inode object.

Additionally, ext4 uses an allocation technique called delayed allocation.

The idea is instead of writing to data blocks one at a time during a write, the allocation requests are accumulated in a buffer and are written to the disk at once.

Not having to call the file system's block allocator on every write request helps the file system make better choices with distributing the available space. For instance, by placing large files apart from smaller files.

Imagine that a small file is located between two large files. Now, if the small file is deleted, it leaves a small space between the two files.

Spreading the files out in this manner leaves enough gaps between data blocks, which helps the filesystem manage (and avoid) fragmentation more easily.

Delayed allocation actively reduces fragmentation and increases performance.

Directories

A Directory (Folder in Windows) is a special file used as a logical container to group files and directories within a file system.

On NTFS and Ext4, directories and files are treated the same way. That said, directories are just files that have their own inode (on Ext4) or MFT entry (on NTFS).

The inode or MFT entry of a directory contains information about that directory, as well as a collection of entries pointing to the files "under" that directory.

The files aren't literally contained within the directory, but they are associated with the directory in a way that they appear as directory's children at a higher level, such as in a file explorer program.

These entries are called directory entries. Directory entries contain file names mapped to their inode/MFT entry.

In addition to the directory entries, there are two more entries. The . entry, which points to the directory itself, and .., which points to the parent directory of this directory.

On Linux, you can use the ls in a directory to see the directory entries with their associated inode numbers:

ls -lai

And the output would be something like this:

63756 drwxr-xr-x 14 root root   4096 Dec  1 17:24 .
     2 drwxr-xr-x 19 root root   4096 Dec  1 17:06 ..
 81132 drwxr-xr-x  2 root root   4096 Feb 18 06:25 backups
 81020 drwxr-xr-x 14 root root   4096 Dec  2 07:01 cache
 81146 drwxrwxrwt  2 root root   4096 Oct 16 21:43 crash
 80913 drwxr-xr-x 46 root root   4096 Dec  1 22:14 lib

 ...

Rules for naming files

Some file systems enforce limitations on filenames.

The limitation can be in the length of the filename or filename case sensitivity.

For instance, in NTFS (Windows) and APFS (Mac) file systems, MyFile and myfile refer to the same file, while on ext4 (Linux), they point to different files.

Why does this matter? You may ask.

Imagine that you’re creating a web page on your Windows machine. The web page contains your company logo, which is a PNG file, like this:

<!DOCTYPE html>
<html>
    <head>
        <title>Products - Your Website</title>
    </head>
    <body>
        <!--SOME CONTENT-->
        <img src="img/logo.png">
        <!--SOME MORE CONTENT-->
    </body>
</html>

If the actual file name is Logo.png (note the capital L), you can still see the image when you open your web page on your web browser (on your Windows machine).

However, once you deploy it to a Linux server and view it live, you'll see a broken image.

Why?

Because in Linux (ext4 file system) logo.png and Logo.png point to two different files.

So keep that in mind when developing on Windows and deploying to a Linux server.

Rules for file size

One important aspect of file systems is the maximum file size they support.

An old file system like FAT32 (used by MS-DOS +7.1, Windows 9x family, and flash memories) can’t store files more than 4 GB, while its successor, NTFS allows file sizes to be up to 16 EB (1000 TB).

Like NTFS, exFAT allows a file size of 16 EB too. This makes exFAT an ideal option for storing massive data objects, such as video files.

Practically, there’s no limitation on the file size in the exFAT and NTFS file systems.

Linux’s ext4 and Apple’s APFS support files up to 16 TiB and 8 EiB respectively.

File manager programs

As you know, the logical layer of the file system provides an API to enable user applications to perform file operations, such as read, write, delete, and execute operations.

The file system’s API is a low-level mechanism, though, designed for computer programs, runtime environments, and shells - not designed for daily use.

That said, operating systems provide convenient file management utilities out of the box for your day-to-day file management.

For instance, File Explorer on Windows, Finder on Mac OS, and Nautilus on Ubuntu are examples of file manager programs.

These utilities use the logical file system’s API under the hood.

Apart from these GUI tools, operating systems expose the file system’s APIs via the command-line interfaces too, like Command Prompt on Windows, and Terminal on Mac and Linux.

These text-based interfaces help users do all sorts of file operations as text commands - Like how we did in the previous examples.

File access management

Not everyone should be able to remove or modify a file they don’t own or are not authorized to do so.

Modern file systems provide mechanisms to control users’ access and capabilities concerning files.

The data regarding user permissions and file ownership is stored in a data structure called Access-Control List (ACL) on Windows or Access-Control Entries (ACE) on Unix-like operating systems (Linux and Mac OS).

This feature is also available in the CLI (Command prompt or Terminal), where a user can change file ownerships or limit permissions of each file right from the command line interface.

For instance, a file owner (on Linux or Mac) can configure a file to be available to the public, like so:

chmod 777 myfile.txt

777 means everyone can do every operation (read, write, execute) on myfile.txt. Please note this is just an example, and you should not set a file's permission to 777.

Maintaining data integrity

Let’s suppose you've been working on your thesis for a month now. One day, you open the file, make some changes and save it.

Once you save the file, your word processor program sends a “write” request to the file system’s API (the logical file system).

The request is eventually passed down to the physical layer to store the file on several blocks.

But what if the system crashes while the older version of the file is being replaced with the new version?

In older file systems (like FAT32 or ext2) the data would be corrupted because it was partially written to the disk.

This is less likely to happen with modern file systems as they use a technique called journaling.

Journaling file systems record every operation that’s about to happen in the physical layer but hasn’t happened yet.

The main purpose is to keep track of the changes that haven't yet been committed to the file system physically.

The journal is a special allocation on the disk where each writing attempt is first stored as a transaction.

Once the data is physically placed on the storage device, the change is committed to the filesystem.

In case of a system failure, the file system will detect the incomplete transaction and roll it back as if it never happened.

That said, the new content (that was being written) may still be lost, but the existing data would remain intact.

Modern file systems such as NTFS, APFS, and ext4 (even ext3) use journaling to avoid data corruption in case of system failure.

Database File Systems

Typical file systems organize files as directory trees.

To access a file, you traverse to the respective directory, and you'll have it.

cd /music/country/highwayman

However, in a database file system, there’s no concept of paths and directories.

The database file system is a faceted system which groups files based on various attributes and dimensions.

For instance, MP3 files can be listed by artist, genre, release year, and album - at the same time!

A database file system is more like a high-level application to help you organize and access your files more easily and more efficiently. However, you won’t be able to access the raw files outside of this application.

A database file system cannot replace a typical file system, though. It’s just a high-level abstraction for easier file management on some systems.

The iTunes app on Mac OS is a good example of a database file system.

Wrapping Up

Wow! You made it to the end, which means you know a lot more about file systems now. But I'm sure this won't be the end of your file system studies.

So again - can we describe what a file system is and how it works in one sentence?

We can't! 😁

But let's finish this post with the brief description I used at the beginning:

A file system defines how files are named, stored, and retrieved from the storage device.

Alright, I think it does it for this write-up. If you notice something is missing or that I've gotten wrong, please let me in the comments below. That would help me and others too!

By the way, if you like more comprehensive guides like this one, visit my website decodingweb. dev and follow me on Twitter because, besides freeCodeCamp, those are the channels I use to share my everyday findings.

Thanks for reading, and enjoy learning! 😃

Then there are more complex analyses that you'll need to do. At times relationships between two seemingly unrelated data sets can provide surprising insights and unveil important opportunities for the organization.

Data scientists and engineers are continuing to improve how they break down and work on data. Experimentation entails discovering the right correlations among data points.

This means they also need to do some sort of parallelization of such data and resulting models. Parallelization simply means that the same data set is being operated upon in many different ways without damaging the integrity of the original data.

In this article we are going to talk about how you can make sure you're doing such experimentation and parallel processing efficiently and that it provides the maximum insights. We will be tackling different concepts related to data storage and data versioning.

Block Storage vs Object Storage

For the uninitiated, we first must understand the difference between block and object storage and why the latter is the better option when dealing with data experimentation.

_Image source_

What is Block Storage?

It is called “block storage” (also known as SAN) because each dataset (in the form of files) is grouped into blocks stored in disks.

A classic example of block storage is the file system on your personal computer. For enterprise-level use-cases, it is scaled through a network of hard drives connected through fiber optic cables.

There are a few disadvantages to using block storage. First, if a sector (or a block) becomes corrupted, it can damage the files. Another problem is the lack of scalability (expanding the network of fiber optic cables is costly).

What is Object Storage?

In object storage, data is stored as objects. Each object contains the actual data, called the blob, a unique identifier (UUID), and metadata, which contains information about the object (such as timestamp, version, and author).

Object storage makes it cost-effective to scale your data store—you don’t need complex hardware for this. It also makes data retrieval faster as each object can be retrieved through its UUID.

This is in contrast to block storage, where each data location needs to be identified before the actual information can be retrieved.

One disadvantage of using object storage is that data can only be written once and cannot be updated. But this isn’t really a disadvantage as we will see further on in this article.

What Problems Does Object Storage Solve?

As we have already seen, data retrieval can be incredibly fast with object storage (no matter the size of the data store). But when it comes to data experimentation and data parallelization, object storage shines the brightest.

As mentioned before, you can't overwrite any data already stored as an object. This ensures object storage is protected from unwanted (or unauthorized) data destruction or updating. That’s great to know if you do a lot of data processing where accidental corruption of information could happen.

One other problem that object storage can solve is that it doesn’t require data to be structured. As companies produce and consume tremendous amounts of information every moment, often non-structured data (such as PDFs, videos, images) are not so easily processed into useful forms (such as for analytics or dashboards).

With object storage, this is now possible. You can now use non-structured data to develop machine learning models.

With data storage, it’s possible to have different versions of the same blob (with different metadata). As there is Git for code version control, we can have similar ways of managing different versions of the same data.

This brings us to the concept of data lakes.

What are Data Lakes?

Data lakes are central repositories of data that don’t care which format such data is in.

Companies produce and consume tremendous amounts of data. Such data traditionally sits in silos because they belong to different departments or are in different forms (for example, videos aren’t stored in the same directory as the data in the MySQL database).

With data lakes, any department in the enterprise can store information without the need to pre-process it. Likewise, any data can be retrieved and analyzed by anybody from any department.

Data lakes are important because they make data analytics extremely fast and convenient.

How Data Experimentation and Parallelization Work with Object Storage

As with developing software, working with data requires us to utilize tools that can aid us in our workflow. A powerful open source tool for experimenting with data and performing parallelization (that is working on the same data to create different sets of machine learning models) is LakeFS.

LakeFS is an open source platform that provides Git-like capabilities when working with data. This means you can create branches (allowing you to experiment with data) and commit versions of data (and data models).

Why is this Git-like feature important?

First, you need to make sure that your data lake is ACID compliant. This means that your data changes can happen in isolation (in branches). Thus, the integrity of the data is maintained in the master branch (until such changes are ready to be merged).

Another important feature of LakeFS is continuous integration of data (again, much like in software development). Enterprises need to incorporate new data quickly and without being disrupted. Therefore, this ability to have a CI/CD workflow is invaluable.

So, let’s see how we can get started with using LakeFS with our object storage experimentation and parallelization.

How to Install LakeFS

Locally you can install LakeFS by running the following command in your terminal:*

Code source

_*This is assuming you have Docker and Docker-Compose installed in your system. If you don’t have Docker and Docker-Compose, you may try other installation methods here._

Now visit http://127.0.0.1:8000/setup in your browser to verify you have installed it correctly.

How to Create a Repository in LakeFS

Once you’ve verified that LakeFS is installed correctly, go ahead and create an admin user.

Image source

_Image source_

Click on the login link and log in as an administrator.

On the page to which you get redirected, click on Create Repository. A popup will appear:

_Image source_

Congratulations! You now have your first repository. This is the main “bucket” in which you are going to store your data.

Next, we’ll start adding some data.

How to Add Data to your LakeFS Repository

Visit here to install AWS CLI.

With the credentials created during the admin-user creation phase, configure a new connection profile:

Code source

To test if the connection is working, run the following:

Code source

Now, to copy files into the main branch:

Code source

Just note that we need to prefix the path with the name of the branch we want to use.

Now, we will see the file we’ve added in the UI:

_Image source_

Next, we will need to know how to commit and create branches. To do that, we will need to install the LakeFS CLI.

How to Install the LakeFS CLI

You need to first download the binary file here.

Again, we need to use the earlier created admin credentials:

Code source

Here are some of the commands we can run to try it out:

Code source

You can find all the other commands, such as branch creation, and so on, online.

There you have it! Now, you can work with your data any way you like. Experiment without guilt and create multiple versions of your data models.

In Closing

In this article, we covered a bit of ground. We learned the different kinds of data storage mechanisms and why object storage has a lot of edge when dealing with data experimentations and parallelism.

Next, we looked into data lakes and LakeFS, which is a powerful tool for working with data.

At first, it might seem a daunting task. But, as we’ve shown here, with the right set of tools and knowledge, there’s a lot you can accomplish.

As you can imagine, there are many different storage options.

In this article, we’ll briefly discuss two of the most used cloud solutions: block storage (also known as SAN or storage area network) and object storage. After this, we will go through my top 5 suggested object storage solutions.

There is a third type of storage that’s commonly used: file system storage. However, this mechanism can also concur with SANs and object storage, so we won't go too deep into it.

What is Block Storage?

Block storage is a network of hard drives connected via fiber-optic network. This gives it an edge over copper cables due to the increased speed.

The reason it's called block storage is that each file in this system is divided into “blocks” of data stored in a disk. Sectors in the disk hold onto individual blocks of data, and these blocks, when combined, form the whole file.

So while there are advantages of using SAN, like high scalability, it is costly and can get incredibly complex as the network grows.

What is Object Storage?

The defining feature of object storage is that, instead of storing files as blocks, data is stored as objects.

Typically these objects will have more data attached to them than the blocks used for block storage. The objects often include:

• A blob which contains all the payload (i.e., image, video, text content)
• Metadata, which tells us more about the file (timestamps, permissions, author, revision, and so on)
• A universally unique ID (UUID)

One major advantage of this type of storage is that objects are easily obtained and found because of their UUID. With block storage, there’s a specific hierarchy of files that a user goes through before getting the data they need, which can considerably slow down data retrieval.

Now that we have that out of the way, here’s a list of my top 5 object storage tools for developers:

Amazon AWS S3

S3 is one of the pioneers of object storage. It manages gigantic loads of data from all over the world across hundreds of industries.

Features:

• High reliability and durability as it stores S3 objects in copies across multiple systems.
• Allows you to manage costs through its S3 Storage Classes, which provides different rates depending on access patterns.
• Provides the highest security and protection for your data.

Google Cloud Storage (GCS)

Google offers four different storage types for business levels of all sizes. When you’re moving data across each of those storage types, it will provide you with the data lifecycle. With this, you can manage how long data should be stored until it has to be deleted.

Features:

• You don’t have a minimum object size.
• You have access to storage locations all around the world.
• Very high durability and low latency.
• Data has redundancy across several geographic locations.

LakeFS

LakeFS is an open-source tool that works with object storage data lakes. Data lakes usually store files or blobs in raw format centrally through a repository.

Data lakes, on their own, are limited by the lack of frequent communication between entities. LakeFS solves this by using data versioning.

Features:

• Through S3 or GCS, it allows scaling up to Petabytes in size by using a system that mimics Git.
• You can experiment as it provides you with a development environment with your data.
• Since it uses a Git-like scheme, you can safely use new data in another branch without affecting the main branch. You can then, later on, merge it safely once each aspect of new data checks out (schema, etc.).

MiniIO

MiniIO is another open-source solution. It utilizes the Amazon S3 API, which makes it perfect for high scale projects that require super strict security.

Features:

• It calls itself the world’s fastest object storage as it has a read/write speed of up to 183 GB.
• It applies web scaling principles – a cluster can join forces with other clusters until it forms multiple data centers.
• It’s Kubernetes friendly.
• Because it's open source, users can improve and freely redistribute it.

Stackpath

StackPath offers both a Content-Delivery Network service, Edge Computing, and an S3 compatible Object Storage. It touts itself as a cheaper option to Amazon S3 and other cloud providers.

Features:

• It is six times faster than competing services, especially when combined with the CDN or the Edge Computing platform.
• It is serverless, which means it needs no warmup.
• It has 45 edge locations, which means your application is available worldwide with the same performance anywhere.

In Closing

There you have it – a short list of the top object storage tools that you can use for your next web or mobile project. Object storage has indeed proven a great way to store data when scalability is the most significant consideration.

Thanks for reading this article! I hope you learned a thing or two about storage models, especially about object storage. Please feel free to connect with me on LinkedIn and Twitter.

In this article we'll go over what a file system is, the FAT32 standard, and several ways to format a storage device to FAT32 on Windows 10.

What's a file system?

A file system is a standardized way of organizing data on a computer storage device like a flash drive or HDD.

A file system divides a storage device into virtual compartments, almost like a wall of post office boxes, and keeps track of all the information that gets stored in each box.

Some of the most common file system formats for portable storage devices are FAT32, NTFS, and ExFAT.

FAT32 compared to other formats

Of those three common formats, FAT32 is the oldest and most widely supported. Every major operating system will allow you to read and write from a USB flash drive that's formatted to FAT32.

Meanwhile, macOS can only read NTFS drives, and you would need to install third-party software to write back to the drive.

However, though FAT32 is well supported, its maximum drive and file size is severely limited when compared to newer formats like NTFS and ExFAT:

* 1 petabyte is about 1 thousand terabytes ** 1 exabyte is about 1 million terabytes

Note that the maximum drive and file size of NTFS and ExFAT is so large that there's basically no limit. (But it would be nice to have a 128 PB USB drive, wouldn't it?)

On the other hand, FAT32's max file size of 4 GB is almost nothing now that phones can record 4K videos. Also, it's a little more difficult to format a drive larger than 32 GB to FAT32 on Windows 10.

These days, the only reason why you'd choose to format a drive to FAT32 is for compatibility. For example, if you need to boot up an old computer, maybe with a different operating system, and backup some of its files. But you'd need to be sure that none of those files are greater than 4 GB.

If you're sure you want to go with FAT32, here's how to format a storage drive on Windows 10.

Important note: Before you format a drive, make sure that you backup all of your important files. In fact, make two backups, and keep one on a remote service like Google Drive or Dropbox.

Formatting a drive will delete all of the data that's currently on it.

How to use Windows File Explorer to format a USB drive to FAT32

A quick note about this method: it only works on USB flash drives that are less that 32 GB. If your USB drive is larger than 32 GB, check out one of the later methods.

With that out of the way, plug your USB drive into your computer and open Windows File Explorer.

Next, right-click on the drive on the left hand side of the File Explorer window and click "Format":

In the window that pops up, ensure that "FAT32" is selected. Also, feel free to rename the USB drive whatever you'd like:

You can leave the rest of the options alone. Just click start to format your drive.

Once it's done, your USB drive should be formatted to use the FAT32 file system.

To double check this, open File Explorer, right click on your USB drive, and click "Properties".

A window will pop up and you should see that the file system is now FAT32:

How to use Rufus to format a USB drive to FAT32

If your USB drive is larger than 32 GB, you'll need to use a third-party program like Rufus to format it.

There are lots of other programs that can format USB drives, but Rufus is really small and portable. This means you can stick Rufus right on a USB drive, plug it into any Windows computer, and format other drives on the go.

After you download Rufus, double click on the .exe file to start the application.

Make sure your USB drive is selected. Then, click the "Boot selection" dropdown and select "Non bootable":

Next, click the "File system" dropdown and select "FAT32".

Also, feel free to change the name of your USB drive under "Volume label":

Then, click the "Start" button to format your drive. After a few seconds it'll be formatted to FAT32.

How to use PowerShell to format a USB drive to FAT32

While this method works with drives larger than 32 GB, it's really slow – even formatting a 32 GB drive can take up to an hour depending on your computer.

But, if you aren't able to use the previous two methods for some reason, this will work in a pinch.

First, click on the Windows Search Bar and type in "powershell". Then, click "Run as administrator" to launch PowerShell with elevated privileges:

In the PowerShell terminal, enter the following command:

format /FS:FAT32 DRIVE_LETTER:

Use the File Explorer to double check your drive letter. My drive letter was D, so I entered format /FS:FAT32 D:.

Press Enter, make sure your USB drive is plugged in, and press the Enter key again to start the process:

Then go run some errands or something – it will take awhile.

Once the format command is finished, your drive should be formatted to FAT32.

In closing

Now you should be able to format a USB drive of any size to FAT32 on Windows 10. And with just a little modification, any of these methods can be used to format your drive to another file system like NTFS or ExFAT.

Now get out there and format all your USB drives. (But only after you backup everything important!)

Was this helpful? Is there a better method that you know of? Tweet at me and let me know how you format things on Windows 10.

If you have ever worked as a developer, you have likely come across file storage use cases. From simple images to large videos, uploading, storing, and accessing those files when you need them is always tricky.

The usual answer to file storage is to keep them on the same server where you host your web applications. But with the advent of serverless architectures and single-page applications, storing files on the same server is not a good idea.

You could argue that you can store files in databases. But trust me, it won’t be a pleasant experience.

So what's another option?

What is S3?

Let's look at AWS S3. S3 is an easy-to-use, scalable, and cheap storage service from Amazon. You can use S3 to store any amount of data for a wide range of use cases.

Static website hosting, data archival, and software delivery are a few general scenarios where S3 would be a perfect tool.

You can easily push and pull data with S3 using the AWS SDK. S3 also supports a number of popular programming languages, so you can use your existing stack and integrate S3 pretty easily.

AWS Console

S3 also offers a great user interface via the AWS console. You can use it to view the data pushed to S3 along with additional options such as security and version control.

Buckets

In S3, files are stored in buckets. Buckets are similar to folders on your computer.

Every bucket has its own unique name which can be used only once. For example, if there is a bucket called “freecodecamp”, neither you nor anyone else can re-use the same bucket name.

This is useful to uniquely identify resources and for static website hosting with domain names.

There are no limits on the number of files you can store in a bucket. Buckets also provide additional features such as version control and policies.

You can also use different buckets for a single application. For example, an app that stores medical records can use two buckets: one for private customer data and another public bucket that contains whitepapers.

S3 is also an object-based storage service which means S3 considers each file an object. Every object can have its own metadata that includes the name, size, date, and other information.

S3 Storage Types

S3 has three storage classes based on general use cases.

S3 Standard

S3 Standard is the default storage plan you will be put into when you start using S3. The standard storage class has excellent performance, durability, and availability.

S3 Standard is best if you have data that you have to access frequently.

S3 Infrequent Access (S3-IA)

S3 Infrequent Access offers a lower price for data compared to the standard plan. You can use S3-IA for data that you need less often.

S3-IA is great for use cases such as backups and disaster recovery.

Glacier

Glacier is the least expensive storage option in S3 but is designed for archival storage. You cannot fetch data from Glacier as fast as Standard or S3-IA, but it is a great option for long term data archival.

In addition to choosing one of these three storage classes, you can also set lifecycle policies in S3. This means that you can schedule files to be moved automatically to S3-IA or Glacier after a certain period of time.

Why Use S3?

Companies like Netflix, Dropbox, and Reddit are avid users of S3. The popular file storage system Dropbox built its entire storage capacity on top of Amazon S3.

Let’s look at some of the core features of S3 and understand why it's so popular among enterprises and startups alike.

It's Affordable

S3 is cheap. I mean super cheap compared to other storage solutions. And with S3, you only pay for what you use. There are no upfront costs, no setup. It's just plug and play.

In addition to affordable pricing, S3 offers a Free tier. This free tier comes with 5GB of storage space, 20,000 GET Requests, 2,000 PUT, COPY, POST, or LIST Requests and 15GB of Data Transfer. The free tier is available every month for the first year.

With S3 you can avoid paying for space or bandwidth you might not even need.

It's Scalable

S3 scales with your application. Since you pay only for that you use, there is no limit to the data you can store in S3.

This is helpful during multiple scenarios, especially during an unexpected surge in user growth. You don’t have to buy extra space. S3 has you covered.

It's Secure

One of the many reasons companies prefer S3 is its inclination towards security. While you have to secure custom server setups, S3 is secure by default.

This does not mean you cannot store publicly accessible information in S3. S3 locks up all your data with high security unless you explicitly configure not to.

S3 also maintains compliance programs, such as PCI-DSS, HIPAA/HITECH, FedRAMP, EU Data Protection Directive, and FISMA, to help you meet your industry’s regulatory requirements.

It Has Versioning

Versioning means keeping multiple copies of a file and tracking its changes over time. This is useful, especially when you handle sensitive data.

You can also retrieve accidentally deleted files when you enable versioning with S3.

However, if you enable versioning, you are storing multiple copies of the same document. This can have an effect on pricing as well as read/write requests you make.

So just take that into account while integrating versioning for your application.

Versioning is disabled by default for S3 but you can enable versioning using the AWS Console.

It's Durable

Data durability is an underrated feature of S3. Given how common data loss is among companies, data durability is a core factor to consider when building enterprise software.

S3 provides a highly durable storage infrastructure. S3 redundantly stores data in multiple facilities, making you data safe in the event of a system failure. S3 also performs regular data integrity checks to make sure your data is intact.

S3 offers 99.999999999% durability (called the 9s durability) and 99.99% availability of objects over a given year.

S3 Use Cases

Static Website Hosting

You can use S3 as a static website hosting platform. The difference between static and dynamic websites is that dynamic websites receive and process user input. Static websites are used only for displaying information.

With the advent of Single Page Applications, you can host a complete web app on S3, often free of charge.

Frameworks like React and Angular have made user input processing happen within the browser. You can build a SPA that listens to third party APIs and host it within S3.

S3 also has great support for routing, so you can use your own custom domain as well.

I recently wrote an article on hosting a React web app using S3 and you can find the article here.

Analytics

You can run queries on your S3 data without moving your data to an analytics platform. This makes S3 a great use case for building powerful analytics applications.

S3 offers multiple options including S3 Select, Amazon Athena, and Amazon Redshift Spectrum. You can also combine these with AWS Lambda to perform data processing on the fly.

File Sharing

Amazon S3 can be also used as a cheap file sharing solution. Like I mentioned earlier in the article, the famous file sharing service Dropbox was first built on top of S3.

With flexible security policies, you can configure your S3 buckets with custom permissions for different customers. S3 also offers transfer acceleration to speed up large file transfers across longer distances.

Summary

Amazon S3 is a great tool to work with for your web or mobile application storage requirements. With on-demand pricing and scalability at its core, S3 has been the favored cloud storage solution for small and large businesses alike.

Companies from Netflix to Pinterest trust S3 with their data, thanks to the 99.999999999% data durability promise from Amazon.

You can also use Amazon S3 as a personal storage solution or host your next project via static site hosting. In a nutshell, S3 is a great multi-purpose storage solution catering to a wide range of use cases.

I regularly write about Machine Learning, Cyber Security, and AWS. You can signup for my weekly newsletter here.

There are two main possibilities for browser storage: localStorage and sessionStorage.

localStorage

Any content/data saved to the localStorage object will be available after the browser has been restarted (closed and opened again). In order to save an item to localStorage, you can use the method setItem(). This method must be handed a key and a value.

Example: localStorage.setItem("mykey","myvalue");

To retrieve the item from the localStorage, the method getItem must be used. The getItem method must be handed the key of the data you would like to retrieve:

  Example: localStorage.getItem("mykey");

You can remove an item from localStorage by using the removeItem() method. This method must be handed the key of the item to be removed:

  Example: localStorage.removeItem("mykey");

To clear the entire localStorage, you should use the clear() method on the localStorage object:

  Example: localStorage.clear();

sessionStorage

Items saved in the sessionStorage object will remain until the browser is closed by the user. Then, the storage will be cleared.

You can save an item to sessionStorage, please use the method setItem() on the sessionStorage object:

Example: sessionStorage.setItem("mykey","myvalue");

To retrieve the item from the sessionStorage, the method getItem must be used. The getItem method must be handed the key of the data you would like to retrieve:

  Example: sessionStorage.getItem("mykey");

You can remove an item from sessionStorage by using the removeItem() method. This method must be handed the key of the item to be removed:

  Example: sessionStorage.removeItem("mykey");

To clear the entire sessionStorage, you should use the clear() method on the sessionStorage object:

  Example: sessionStorage.clear();

Saving arrays to localStorage and sessionStorage

You cannot just save single values to the localStorage and sessionStorage, but you can also save the content of an array.

In this example, we have an array with numbers:

var ourArray =[1,2,3,4,5];

We can now save it to localStorage or sessionStorage using the setItem() method:

localStorage.setItem("ourarraykey",JSON.stringify(ourArray));

or, for sessionStorage:

sessionStorage.setItem("ourarraykey",JSON.stringify(ourArray));

In order to be saved, the array must first be converted to a string. In the example shown above, we are using the JSON.stringify method to accomplish this.

When retrieving our data from the localStorage or sessionStorage, convert it back to an array:

var storedArray = localStorage.getItem("ourarraykey");
ourArray = JSON.parse(storedArray);

or, for sessionStorage:

var storedArray = sessionStorage.getItem("ourarraykey");
ourArray = JSON.parse(storedArray);