Choosing a storage solution is one of the most significant decisions a developer (or development team) needs to make when building a web or mobile application.
As you can imagine, there are many different storage options.
In this article, we’ll briefly discuss two of the most used cloud solutions: block storage (also known as SAN or storage area network) and object storage. After this, we will go through my top 5 suggested object storage solutions.
There is a third type of storage that’s commonly used: file system storage. However, this mechanism can also concur with SANs and object storage, so we won't go too deep into it.
What is Block Storage?
Block storage is a network of hard drives connected via fiber-optic network. This gives it an edge over copper cables due to the increased speed.
The reason it's called block storage is that each file in this system is divided into “blocks” of data stored in a disk. Sectors in the disk hold onto individual blocks of data, and these blocks, when combined, form the whole file.
So while there are advantages of using SAN, like high scalability, it is costly and can get incredibly complex as the network grows.
What is Object Storage?
The defining feature of object storage is that, instead of storing files as blocks, data is stored as objects.
Typically these objects will have more data attached to them than the blocks used for block storage. The objects often include:
- A blob which contains all the payload (i.e., image, video, text content)
- Metadata, which tells us more about the file (timestamps, permissions, author, revision, and so on)
- A universally unique ID (UUID)
One major advantage of this type of storage is that objects are easily obtained and found because of their UUID. With block storage, there’s a specific hierarchy of files that a user goes through before getting the data they need, which can considerably slow down data retrieval.
Now that we have that out of the way, here’s a list of my top 5 object storage tools for developers:
Amazon AWS S3
S3 is one of the pioneers of object storage. It manages gigantic loads of data from all over the world across hundreds of industries.
Features:
- High reliability and durability as it stores S3 objects in copies across multiple systems.
- Allows you to manage costs through its S3 Storage Classes, which provides different rates depending on access patterns.
- Provides the highest security and protection for your data.
Google Cloud Storage (GCS)
Google offers four different storage types for business levels of all sizes. When you’re moving data across each of those storage types, it will provide you with the data lifecycle. With this, you can manage how long data should be stored until it has to be deleted.
Features:
- You don’t have a minimum object size.
- You have access to storage locations all around the world.
- Very high durability and low latency.
- Data has redundancy across several geographic locations.
LakeFS
LakeFS is an open-source tool that works with object storage data lakes. Data lakes usually store files or blobs in raw format centrally through a repository.
Data lakes, on their own, are limited by the lack of frequent communication between entities. LakeFS solves this by using data versioning.
Features:
- Through S3 or GCS, it allows scaling up to Petabytes in size by using a system that mimics Git.
- You can experiment as it provides you with a development environment with your data.
- Since it uses a Git-like scheme, you can safely use new data in another branch without affecting the main branch. You can then, later on, merge it safely once each aspect of new data checks out (schema, etc.).
MiniIO
MiniIO is another open-source solution. It utilizes the Amazon S3 API, which makes it perfect for high scale projects that require super strict security.
Features:
- It calls itself the world’s fastest object storage as it has a read/write speed of up to 183 GB.
- It applies web scaling principles – a cluster can join forces with other clusters until it forms multiple data centers.
- It’s Kubernetes friendly.
- Because it's open source, users can improve and freely redistribute it.
Stackpath
StackPath offers both a Content-Delivery Network service, Edge Computing, and an S3 compatible Object Storage. It touts itself as a cheaper option to Amazon S3 and other cloud providers.
Features:
- It is six times faster than competing services, especially when combined with the CDN or the Edge Computing platform.
- It is serverless, which means it needs no warmup.
- It has 45 edge locations, which means your application is available worldwide with the same performance anywhere.
In Closing
There you have it – a short list of the top object storage tools that you can use for your next web or mobile project. Object storage has indeed proven a great way to store data when scalability is the most significant consideration.
Thanks for reading this article! I hope you learned a thing or two about storage models, especially about object storage. Please feel free to connect with me on LinkedIn and Twitter.