Your production will keep using a managed S3 / Cloudflare R2 / Hetzner Object Storage, while every staging upload, download, and pre-signed URL goes to your own server for free.

Table of Contents

• 1. Why Self‑Host Object Storage on Staging?

• 2. The Architecture: Production vs. Staging

• 3. Prerequisites

• 4. Step 1 — DNS: Point Your Domains to the Staging Server

• 5. Step 2 — Run MinIO with Docker Compose

• 6. Step 3 — Expose MinIO over HTTPS with Traefik

• 7. Step 4 — Create the Bucket and Access Keys

• 8. Step 5 — Configure Your App to Use MinIO on Staging Only

• 9. Step 6 — Upload Files (3 Ways)

• 10. Step 7 — Generate Presigned URLs (PUT and GET)

• 11. Step 8 — Get Public URLs for Documents

• 12. Step 9 — Lock Down CORS, Lifecycle, and Security

• 13. Step 10 — Backups and Monitoring

• 14. Troubleshooting Cheat Sheet

• 15. Wrapping Up

1. Why Self‑Host Object Storage on Staging?

If your app handles documents — PDFs, profile pictures, application transcripts, recordings — every test upload your QA team makes costs real money on AWS S3, Cloudflare R2, or Hetzner Object Storage. The price isn't huge per file, but staging is where you:

• run automated end‑to‑end tests that upload thousands of dummy files,

• reset databases nightly (which leaves orphan objects behind),

• let developers experiment with broken code that re‑uploads the same files,

• and hold months of test data nobody ever deletes.

In production those costs are justified. Managed storage gives you replication, availability, and someone else's pager. In staging, those costs are pure waste.

MinIO is a free, open‑source, S3‑compatible object server. Same API, same SDKs, same presigned URLs, same mc/aws s3 CLIs — but running on your own VPS, billed at $0 per gigabyte. Point your staging app at MinIO, point your production app at S3/R2, and the only thing that changes is an environment variable.

The result: identical code paths in both environments, zero storage bill on staging, and a nice fallback if your cloud provider ever has an outage.

2. The Architecture: Production vs. Staging

In real-world applications, you usually don’t want your development or staging environment writing directly to production storage.

A common and cost-effective setup is:

• Production: managed cloud object storage

• Staging / Development: self-hosted S3-compatible storage

The good part is that your application code doesn't need to change.

As long as both services are S3-compatible, the same SDK and upload logic work everywhere. Only the environment variables differ.

High-Level Architecture

The above diagram illustrates how the same application can communicate with different storage providers depending on the deployment environment.

In the production environment, uploads are stored in a managed object storage service such as AWS S3, Cloudflare R2, or Hetzner Object Storage. These services handle durability, scalability, backups, and infrastructure management.

In the staging environment, uploads are directed to a self-hosted MinIO instance running inside Docker on a VPS. MinIO implements the S3 API, making it behave similarly to production storage while keeping costs low.

Because both storage systems are S3-compatible, the application uses the same upload logic in every environment. The only difference is the configuration provided through environment variables.

Why This Architecture Is Useful

This setup gives you:

• A cheap staging environment

• Production-like testing

• Zero storage vendor lock-in

• The ability to switch providers without rewriting application code

Because both environments speak the S3 protocol, your upload logic remains identical.

Example Environment Variables

Your application only reads environment variables like these:

S3_ENDPOINT=
S3_REGION=
S3_ACCESS_KEY=
S3_SECRET_KEY=
S3_BUCKET=

Switch the values, and the exact same application now uploads files to a different backend.

Production Storage Example

In production, you typically use managed object storage providers such as:

• AWS S3

• Cloudflare R2

• Hetzner Object Storage

Example:

S3_ENDPOINT=https://<region>.r2.cloudflarestorage.com

The benefits are that it's highly scalable, globally available, durable, has managed backups, and doesn't have infrastructure maintenance.

Staging Environment Example

For staging, a lightweight self-hosted MinIO container is often enough.

Next.js App
     ↓
MinIO Container (inside Docker on VPS)

Example domains:

This allows you to:

• Test uploads safely

• Avoid production storage costs

• Reproduce production-like behavior locally

3. Prerequisites

You'll need:

• A Linux VPS (Hetzner, DigitalOcean, Contabo, OVH — anything with a public IP).

• Two A records pointing at that IP (we'll register them next).

• Docker + Docker Compose v2.

• Traefik v2 in front, with Let's Encrypt configured (any reverse proxy works – the labels below are Traefik's flavor).

• Open ports 80 and 443 on the firewall for Let's Encrypt + HTTPS.

• ~10 GB free disk for the MinIO data volume to start.

If Docker isn't installed:

curl -fsSL https://get.docker.com | sh
sudo apt-get install -y docker-compose-plugin
docker --version && docker compose version

4. Step 1 — DNS: Point Your Domains to the Staging Server

In your DNS provider (Cloudflare, Route 53, Namecheap, and so on), create two A records pointing at your staging server's public IP:

minio-staging.domain.com           A    203.0.113.45
minio-console-staging.domain.com   A    203.0.113.45

If you use Cloudflare, set the proxy status to DNS only (gray cloud) for minio-staging.*. Cloudflare's free plan caps uploads at 100 MB, and you don't want it stripping S3 signing headers. The console subdomain can stay proxied if you want a WAF in front of it.

Wait a minute and verify:

dig +short minio-staging.domain.com
# 203.0.113.45

5. Step 2 — Run MinIO with Docker Compose

Add this service to your staging compose file (docker-compose.staging.yml). MinIO is just one container — the disk is mounted as a Docker volume so data survives upgrades.

# docker-compose.staging.yml
networks:
  proxy:
    external: true
    name: proxy
  internal:
    name: internal

volumes:
  minio-data:

services:
  minio:
    image: minio/minio:latest
    container_name: minio-staging
    restart: unless-stopped
    environment:
      - MINIO_ROOT_USER=${MINIO_ROOT_USER:-admin}
      - MINIO_ROOT_PASSWORD=${MINIO_ROOT_PASSWORD:-change-me-please}
      # Tell MinIO which public domain to sign URLs with
      - MINIO_SERVER_URL=https://minio-staging.domain.com
      - MINIO_BROWSER_REDIRECT_URL=https://minio-console-staging.domain.com
    command: server /data --console-address ":9001"
    volumes:
      - minio-data:/data
    networks:
      - proxy
      - internal
    ports:
      - "9000:9000"  # S3 API
      - "9001:9001"  # Web console
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 10s
      timeout: 5s
      retries: 3
      start_period: 30s

Two things deserve attention:

• MINIO_SERVER_URL is the secret sauce. Without it, MinIO signs presigned URLs using its internal hostname (http://minio:9000), which then fails verification when the browser hits the public domain. Set it to the exact HTTPS URL clients will use.

• MINIO_BROWSER_REDIRECT_URL does the same for the web console (login redirects, OIDC callbacks, and so on).

Bring it up:

docker compose -f docker-compose.staging.yml up -d minio
docker compose -f docker-compose.staging.yml logs -f minio

You should see API: http://... and Console: http://... lines.

6. Step 3 — Expose MinIO over HTTPS with Traefik

We don't expose ports 9000/9001 to the world directly — Traefik does that for us, terminating TLS with a free Let's Encrypt certificate.

Add these labels to the minio service:

    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=proxy"

      # ---- S3 API (port 9000) ----
      - "traefik.http.routers.minio-staging.rule=Host(`minio-staging.domain.com`)"
      - "traefik.http.routers.minio-staging.entrypoints=websecure"
      - "traefik.http.routers.minio-staging.tls.certresolver=letsencrypt"
      - "traefik.http.routers.minio-staging.service=minio-staging"
      - "traefik.http.services.minio-staging.loadbalancer.server.port=9000"

      # ---- Web Console (port 9001) ----
      - "traefik.http.routers.minio-console-staging.rule=Host(`minio-console-staging.domain.com`)"
      - "traefik.http.routers.minio-console-staging.entrypoints=websecure"
      - "traefik.http.routers.minio-console-staging.tls.certresolver=letsencrypt"
      - "traefik.http.routers.minio-console-staging.service=minio-console-staging"
      - "traefik.http.services.minio-console-staging.loadbalancer.server.port=9001"

You also need an entrypoint for :443 and a certificatesresolver named letsencrypt. Here's the minimum Traefik config (traefik.staging.yml):

api:
  dashboard: true

entryPoints:
  web:
    address: ":80"
  websecure:
    address: ":443"

certificatesResolvers:
  letsencrypt:
    acme:
      httpChallenge:
        entryPoint: web
      email: admin@domain.com
      storage: /etc/traefik/acme.json

providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false
    network: proxy

Restart and watch the cert get issued:

docker compose -f docker-compose.staging.yml up -d
docker compose -f docker-compose.staging.yml logs -f traefik | grep -i acme

Sanity check from your laptop:

curl -I https://minio-staging.domain.com/minio/health/live
# HTTP/2 200

You can now log in to the web console at https://minio-console-staging.domain.com with admin / change-me-please.

Important upload size tweak: if you're behind Cloudflare or NGINX in front of Traefik, raise the request body limit. Traefik itself has no default limit, but Cloudflare's free plan refuses anything over 100 MB. For self‑hosted edge proxies, set client_max_body_size 0; (NGINX) or the equivalent.

7. Step 4 — Create the Bucket and Access Keys

Anything that speaks S3 can talk to MinIO. The easiest tool is mc (the official MinIO client), shipped inside the same image.

7.1 Connect mc to your server

docker exec -it minio-staging \
  mc alias set local http://localhost:9000 admin change-me-please

7.2 Create a bucket

docker exec -it minio-staging mc mb local/domain-files-staging

7.3 Choose a bucket policy

You have three choices, so just pick based on what you store:

Set one:

# Private (recommended for documents)
docker exec -it minio-staging \
  mc anonymous set none local/domain-files-staging

# OR public read for static assets only:
docker exec -it minio-staging \
  mc anonymous set download local/domain-files-staging

7.4 Create a dedicated app user (don't use root keys!)

The admin account can wipe everything. Make a least‑privilege user for your app:

docker exec -it minio-staging mc admin user add local \
  domain-app a-long-random-secret-key

# Attach the built-in read/write policy, scoped to one bucket via JSON:
cat > /tmp/policy.json <<'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:*"],
      "Resource": [
        "arn:aws:s3:::domain-files-staging",
        "arn:aws:s3:::domain-files-staging/*"
      ]
    }
  ]
}
EOF

docker cp /tmp/policy.json minio-staging:/tmp/policy.json
docker exec -it minio-staging \
  mc admin policy create local domain-rw /tmp/policy.json
docker exec -it minio-staging \
  mc admin policy attach local domain-rw --user domain-app

Save those two values — they are your S3_ACCESS_KEY and S3_SECRET_KEY.

8. Step 5 — Configure Your App to Use MinIO on Staging Only

The trick to "MinIO in staging, real S3 in prod" is to use the same S3 client in your code and only swap the env vars.

Your staging.env (loaded by your staging compose stack):

# ---- Staging: self-hosted MinIO ----
STORAGE_ENABLED=true
S3_ENDPOINT=https://minio-staging.domain.com
S3_PUBLIC_ENDPOINT=https://minio-staging.domain.com
S3_BUCKET=domain-files-staging
S3_ACCESS_KEY=domain-app
S3_SECRET_KEY=a-long-random-secret-key
S3_REGION=us-east-1
S3_FORCE_PATH_STYLE=true

Your production.env:

# ---- Production: Cloudflare R2 ----
STORAGE_ENABLED=true
S3_ENDPOINT=https://<account-id>.r2.cloudflarestorage.com
S3_PUBLIC_ENDPOINT=https://files.domain.com
S3_BUCKET=domain-files
S3_ACCESS_KEY=<r2-access-key>
S3_SECRET_KEY=<r2-secret-key>
S3_REGION=auto
S3_FORCE_PATH_STYLE=true

S3_FORCE_PATH_STYLE=true is critical for both MinIO and R2/Hetzner. Without it, the SDK tries https://bucket.minio-staging.domain.com (virtual‑host style), which won't resolve.

Now in your application code (Node.js example using AWS SDK v3):

// src/lib/s3.js
import { S3Client } from "@aws-sdk/client-s3";

export const s3 = new S3Client({
  endpoint: process.env.S3_ENDPOINT,
  region: process.env.S3_REGION,
  credentials: {
    accessKeyId: process.env.S3_ACCESS_KEY,
    secretAccessKey: process.env.S3_SECRET_KEY,
  },
  forcePathStyle: process.env.S3_FORCE_PATH_STYLE === "true",
});

export const BUCKET = process.env.S3_BUCKET;
export const PUBLIC_ENDPOINT = process.env.S3_PUBLIC_ENDPOINT;

The same s3 instance now talks to MinIO on staging and to R2 in production with no code change.

9. Step 6 — Upload Files (3 Ways)

9.1 From a server (best for trusted backends)

import { PutObjectCommand } from "@aws-sdk/client-s3";
import { s3, BUCKET } from "./lib/s3.js";
import { readFile } from "node:fs/promises";

export async function uploadDocument(localPath, key, contentType) {
  const Body = await readFile(localPath);
  await s3.send(new PutObjectCommand({
    Bucket: BUCKET,
    Key: key,
    Body,
    ContentType: contentType,
    // Optional: per-object metadata, useful for audits
    Metadata: { uploadedBy: "system", env: process.env.NODE_ENV },
  }));
  return key;
}

9.2 With the mc CLI (good for one‑off uploads / migrations)

mc alias set staging https://minio-staging.domain.com domain-app a-long-random-secret-key
mc cp ./report.pdf staging/domain-files-staging/reports/2026/report.pdf
mc ls staging/domain-files-staging --recursive

9.3 Directly from the browser via a presigned PUT URL

The recommended pattern for user uploads is: the file goes from the browser to MinIO with zero bytes touching your API server.

We'll cover this in detail next.

10. Step 7 — Generate Presigned URLs (PUT and GET)

A presigned URL is a regular HTTPS URL with a time‑limited signature in the query string. Anyone with the URL can do exactly the action it was signed for (PUT this object, or GET that object) for the next N minutes — and nothing else.

This is what makes "users upload directly to storage" safe.

10.1 Presigned PUT (for uploads)

// src/lib/presign.js
import { PutObjectCommand, GetObjectCommand } from "@aws-sdk/client-s3";
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";
import { s3, BUCKET } from "./s3.js";
import { randomUUID } from "node:crypto";

export async function presignUpload({ filename, contentType, userId }) {
  const key = `users/\({userId}/\){randomUUID()}-${filename}`;
  const cmd = new PutObjectCommand({
    Bucket: BUCKET,
    Key: key,
    ContentType: contentType,
  });
  const uploadUrl = await getSignedUrl(s3, cmd, { expiresIn: 60 * 5 }); // 5 min
  return { uploadUrl, key };
}

Wire it to your API:

// POST /api/uploads/presign
app.post("/api/uploads/presign", requireAuth, async (req, res) => {
  const { filename, contentType } = req.body;
  const result = await presignUpload({
    filename,
    contentType,
    userId: req.user.id,
  });
  res.json(result); // { uploadUrl, key }
});

The browser uploads straight to MinIO:

// In your frontend
async function uploadFile(file) {
  const { uploadUrl, key } = await fetch("/api/uploads/presign", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ filename: file.name, contentType: file.type }),
  }).then(r => r.json());

  await fetch(uploadUrl, {
    method: "PUT",
    headers: { "Content-Type": file.type },
    body: file,
  });

  // Persist `key` in your DB so you can retrieve it later
  await fetch("/api/documents", {
    method: "POST",
    body: JSON.stringify({ key, originalName: file.name }),
  });
}

The Content-Type you send during PUT must match the one you signed with, or MinIO will reject the request with SignatureDoesNotMatch. This catches everyone the first time.

10.2 Presigned GET (for downloads)

Same idea, but with GetObjectCommand:

export async function presignDownload(key, expiresIn = 60 * 10) {
  const cmd = new GetObjectCommand({ Bucket: BUCKET, Key: key });
  return getSignedUrl(s3, cmd, { expiresIn });
}

A typical "view document" endpoint:

app.get("/api/documents/:id/url", requireAuth, async (req, res) => {
  const doc = await db.documents.findById(req.params.id);
  if (!doc || !canUserSee(req.user, doc)) return res.sendStatus(403);
  const url = await presignDownload(doc.key, 600);
  res.json({ url });
});

The frontend just opens that URL — the file streams from MinIO directly to the user.

10.3 Why presigned URLs beat "proxy through the API"

11. Step 8 — Get Public URLs for Documents

Sometimes you want a permanent, unauthenticated URL — for example public profile pictures.

If the bucket policy allows anonymous reads (mc anonymous set download …), the public URL pattern is:

https://minio-staging.domain.com/<bucket>/<key>

So users/42/avatar.png becomes:

https://minio-staging.domain.com/domain-files-staging/users/42/avatar.png

In code:

export function publicUrl(key) {
  return `\({process.env.S3_PUBLIC_ENDPOINT}/\){BUCKET}/${key}`;
}

For private buckets (most documents), don't use public URLs at all — always go through presignDownload(key) so you can re‑check authorization on every request and expire links.

12. Step 9 — Lock Down CORS, Lifecycle, and Security

12.1 Allow your frontend origins (CORS)

Browser uploads need CORS rules on the bucket. Drop this JSON via mc:

cat > /tmp/cors.json <<'EOF'
{
  "CORSRules": [
    {
      "AllowedOrigins": [
        "https://crm-staging.domain.com",
        "http://localhost:3000"
      ],
      "AllowedMethods": ["GET", "PUT", "POST", "HEAD"],
      "AllowedHeaders": ["*"],
      "ExposeHeaders": ["ETag"],
      "MaxAgeSeconds": 3000
    }
  ]
}
EOF

docker cp /tmp/cors.json minio-staging:/tmp/cors.json
docker exec -it minio-staging \
  mc cors set local/domain-files-staging /tmp/cors.json

12.2 Auto‑delete old test files (lifecycle)

Staging accumulates junk. Tell MinIO to expire anything older than 30 days:

docker exec -it minio-staging \
  mc ilm rule add --expire-days 30 local/domain-files-staging

12.3 Encrypt at rest

docker exec -it minio-staging \
  mc encrypt set sse-s3 local/domain-files-staging

12.4 Hard rules

• Never ship MINIO_ROOT_USER=admin / MINIO_ROOT_PASSWORD=admin123 to a server reachable from the internet. Generate strong values and store them in your secret manager.

• The root account should be used only by mc admin, never by your app. The app uses a scoped IAM user (Step 7.4).

• Keep the console subdomain behind an IP allow‑list or basic auth via Traefik middleware if it's truly public.

• Rotate the app access keys at least every 90 days.

13. Step 10 — Backups and Monitoring

13.1 Backups: mirror to a cheap cold bucket weekly

Set up a tiny cron job that uses mc mirror to push to Backblaze B2, R2, or another cheap S3 endpoint:

mc alias set b2 https://s3.us-east-005.backblazeb2.com \(B2_KEY \)B2_SECRET
mc mirror --overwrite --remove \
  staging/domain-files-staging \
  b2/domain-staging-backup

Even at $6/TB/month this is essentially free for staging volumes.

13.2 Monitoring with Prometheus

MinIO exposes Prometheus metrics out of the box at /minio/v2/metrics/cluster. Scrape with:

scrape_configs:
  - job_name: minio
    metrics_path: /minio/v2/metrics/cluster
    scheme: https
    static_configs:
      - targets: ["minio-staging.domain.com"]

If you have Grafana, import dashboard ID 13502 for an instant overview (capacity, request rates, latency, error counts).

14. Troubleshooting Cheat Sheet

Useful diagnostics:

# Check MinIO health
curl -I https://minio-staging.domain.com/minio/health/live

# List all objects in a bucket
docker exec -it minio-staging mc ls --recursive local/domain-files-staging

# Tail MinIO logs
docker compose -f docker-compose.staging.yml logs -f minio

# Decode a presigned URL to see what it was signed for
echo "<paste url>" | tr '&' '\n'

15. Wrapping Up

Here's what you have now:

• A free, S3‑compatible object store running on your own staging server.

• Real HTTPS on a real domain (https://minio-staging.domain.com), thanks to Traefik + Let's Encrypt.

• A scoped, least‑privilege application user — root keys stay locked away.

• The same exact code paths in staging and production. Switching between MinIO / R2 / Hetzner / AWS S3 is a four‑variable change in the env file.

• Presigned PUT URLs so users upload straight to storage, bypassing your API.

• Presigned GET URLs so private documents are short‑lived and authorization‑gated.

• Lifecycle rules that nuke old test files automatically.

• Optional weekly mirror to a cold backup bucket.

Production keeps running on managed storage where the SLA matters. Staging now costs you exactly $0 per month per gigabyte uploaded — and you can finally stop telling QA to "delete the test files when you're done."

Further Reading

• MinIO Documentation

• AWS SDK v3 — getSignedUrl

• Traefik v2 Docker provider

• S3 bucket policy reference

If this guide saved your team a few dollars, share it with another team that's still uploading test PDFs to a $90/month S3 bucket. Happy shipping.

But it has always carried a cost that rarely gets discussed openly. That cost isn't just money, though a DynamoDB table with on-demand billing adds up across multiple teams and environments.

The real cost is complexity. Every new AWS environment needs both resources provisioned before Terraform can manage anything else. Every engineer who sets up their first Terraform backend has to understand why two completely different AWS services are responsible for what is logically one thing: storing and protecting state. And every incident involving a stuck lock has required someone to manually delete a record from DynamoDB to unblock the team.

In November 2024, AWS announced that S3 now supports native object locking for Terraform state files, meaning DynamoDB is no longer required for state locking. Terraform 1.10 added support for this feature, and it's now generally available.

In this tutorial, you'll learn:

• What S3 native locking is and how it works

• How to set it up from scratch if you're starting a new project

• How to migrate an existing S3 + DynamoDB setup to S3 native locking safely

• How to verify locking is working and handle edge cases

By the end, you'll have a simpler, cleaner Terraform backend with one fewer AWS resource to manage.

Table of Contents

• What Is Terraform State Locking?

• What Is S3 Native State Locking?

• How S3 Native Locking Compares to the S3 + DynamoDB Approach

• Prerequisites

• Part 1: Fresh Setup – How to Configure S3 Native Locking from Scratch

  • Step 1: Create the S3 Bucket with Versioning and Encryption

  • Step 2: Configure the Terraform Backend with Native Locking

  • Step 3: Initialize and Verify

• Part 2: Migration – How to Move from S3 + DynamoDB to S3 Native Locking

  • Step 1: Verify Your Current Setup

  • Step 2: Enable Object Lock on the Existing S3 Bucket

  • Step 3: Update the Terraform Backend Configuration

  • Step 4: Reinitialize Terraform

  • Step 5: Verify the Migration

  • Step 6: Clean Up the DynamoDB Table

• How to Verify That Locking Is Working

• How to Handle a Stuck Lock

• Rollback Plan: If Something Goes Wrong

• Security Best Practices for Your State Bucket

• Conclusion

• References

What is Terraform State Locking?

Before looking at the new approach, it helps to understand what state locking is solving.

Terraform stores everything it knows about your infrastructure in a state file – a JSON document that maps your configuration to real AWS resources. When you run terraform apply, Terraform reads this file, calculates the difference between the current state and your configuration, and makes the necessary changes.

The problem arises when two engineers or two CI/CD pipelines run and try to apply changes at the same time. If both read the state file simultaneously, calculate changes independently, and both try to write back, you get a race condition. The second write overwrites changes from the first, and your state is now out of sync with reality. This is a serious problem that can cause resources to be untracked, doubled, or destroyed unexpectedly.

State locking solves this by creating a lock when any operation starts that could modify state. If a lock already exists, Terraform refuses to proceed and reports who holds the lock and when it was acquired. Only one operation can hold the lock at a time. When the operation completes, the lock is released.

Terraform Run A                 State File / Lock                Terraform Run B
(User 1)                         (S3/DynamoDB)                   (User 2)

   |                                   |                            |
   |------- 1. Acquire Lock ---------->|                            |
   |                                   |                            |
   |<------ 2. Lock Granted -----------|                            |
   |                                   |                            |
   |                                   |------- 3. Acquire Lock --->|
   |            [PROCESSING]           |                            |
   |      (Modifying Infrastructure)   |<------ 4. Lock Denied -----|
   |                                   |        (Wait / Retry)      |
   |                                   |                            |
   |------- 5. Release Lock ---------->|                            |
   |                                   |                            |
   |           [COMPLETED]             |<------ 6. Lock Granted ----|
   |                                   |                            |
   |                                   |       [PROCESSING]         |
   |                                   | (Modifying Infrastructure) |              
   |                                   |                            |

What Is S3 Native State Locking?

Previously, Terraform's S3 backend used a DynamoDB table as the locking mechanism. When a lock was needed, Terraform wrote a record to DynamoDB with a LockID primary key. DynamoDB's conditional writes guaranteed that only one process could create that record, which is what made the locking atomic.

S3 native locking uses S3 Object Lock instead. S3 Object Lock is an S3 feature originally designed to enforce WORM (Write Once, Read Many) compliance for regulatory requirements. AWS extended this capability to support Terraform's state locking workflow.

When S3 native locking is enabled in your Terraform backend:

1. Terraform writes your state to an .tfstate object in S3 (as before)

2. To acquire a lock, Terraform uses S3's conditional write operations – specifically the if-none-match conditional header to create a lock file atomically

3. If the lock file already exists, S3 rejects the write, and Terraform reports that a lock is held

4. When the operation completes, Terraform deletes the lock file to release the lock.

The key difference from DynamoDB: the entire locking mechanism lives inside S3. No second service. No second set of IAM permissions. No second resource to provision.

Note: This feature requires Terraform version 1.10.0 or later and an S3 bucket with Object Lock enabled. Object Lock must be enabled at bucket creation time. You can't enable it on an existing bucket through the console or CLI. But there is a supported workaround for existing buckets, which we'll cover in Part 2.

How S3 Native Locking Compares to the S3 + DynamoDB Approach

The functional behavior from Terraform's perspective is identical. Locking works the same way. The lock information displayed when a lock is held has the same structure. The only difference is what happens under the hood.

Prerequisites

Before you start, make sure you have the following in place:

• Terraform 1.10.0 or later installed. Check your version:

terraform version

If you need to upgrade, follow the official upgrade guide.

• AWS CLI installed and configured with credentials that have permission to create and manage S3 buckets.

aws --version
aws sts get-caller-identity   # confirm you're authenticated

• IAM permissions to perform the following S3 actions:

  • s3:CreateBucket

  • s3:PutBucketVersioning

  • s3:PutBucketEncryption

  • s3:PutObjectLegalHold

  • s3:PutObjectRetention

  • s3:GetObject

  • s3:PutObject

  • s3:DeleteObject

  • s3:ListBucket

• For the migration path: access to your existing Terraform project and the S3 bucket and DynamoDB table currently in use.

Part 1: Fresh Setup – How to Configure S3 Native Locking from Scratch

Follow this section if you're starting a new Terraform project and want to use S3 native locking from the beginning.

Step 1: Create the S3 Bucket with Versioning and Encryption

Object Lock must be enabled at bucket creation time. You can't add it afterward through the standard console flow. Create the bucket using the AWS CLI with Object Lock enabled:

aws s3api create-bucket \
  --bucket your-project-terraform-state \
  --region us-east-1 \
  --object-lock-enabled-for-bucket

Note: For regions other than us-east-1, add the --create-bucket-configuration flag.

aws s3api create-bucket \
  --bucket your-project-terraform-state \
  --region eu-west-1 \
  --create-bucket-configuration LocationConstraint=eu-west-1 \
  --object-lock-enabled-for-bucket

Now enable versioning on the bucket. Versioning is required alongside Object Lock and allows Terraform to recover previous state versions if something goes wrong:

aws s3api put-bucket-versioning \
  --bucket your-project-terraform-state \
  --versioning-configuration Status=Enabled

Enable server-side encryption so your state files are encrypted at rest:

aws s3api put-bucket-encryption \
  --bucket your-project-terraform-state \
  --server-side-encryption-configuration '{
    "Rules": [
      {
        "ApplyServerSideEncryptionByDefault": {
          "SSEAlgorithm": "AES256"
        },
        "BucketKeyEnabled": true
      }
    ]
  }'

Block all public access to the bucket. A Terraform state file contains resource IDs, IP addresses, and potentially sensitive values. It should never be publicly accessible:

aws s3api put-public-access-block \
  --bucket your-project-terraform-state \
  --public-access-block-configuration \
    "BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true"

Verify the bucket configuration:

# Confirm Object Lock is enabled
aws s3api get-object-lock-configuration \
  --bucket your-project-terraform-state
 
# Confirm versioning is enabled
aws s3api get-bucket-versioning \
  --bucket your-project-terraform-state
 
# Confirm encryption is configured
aws s3api get-bucket-encryption \
  --bucket your-project-terraform-state

Expected output for the Object Lock check:

{
    "ObjectLockConfiguration": {
        "ObjectLockEnabled": "Enabled"
    }
}

Step 2: Configure the Terraform Backend with Native Locking

In your Terraform project, create or update your backend.tf file:

terraform {
  backend "s3" {
    bucket = "your-project-terraform-state"
    key    = "production/terraform.tfstate"
    region = "us-east-1"
 
    # Enable S3 native state locking
    # Requires Terraform 1.10.0+ and a bucket with Object Lock enabled
    use_lockfile = true
 
    # Encryption at rest
    encrypt = true
  }
}

The critical difference from the old configuration is the use_lockfile = true parameter. Notice what is absent: there's no dynamodb_table argument. No DynamoDB table. No second service.

Here's a direct comparison of the old and new configurations:

Old configuration (S3 + DynamoDB):

terraform {
  backend "s3" {
    bucket         = "your-project-terraform-state"
    key            = "production/terraform.tfstate"
    region         = "us-east-1"
    encrypt        = true
    dynamodb_table = "terraform-state-lock"   # this goes away
  }
}

New configuration (S3 native locking):

terraform {
  backend "s3" {
    bucket       = "your-project-terraform-state"
    key          = "production/terraform.tfstate"
    region       = "us-east-1"
    encrypt      = true
    use_lockfile = true   # this replaces dynamodb_table
  }
}

Step 3: Initialize and Verify

Run terraform init to initialize the backend:

terraform init

Expected output:

Initializing the backend...
 
Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.
 
Initializing provider plugins...
 
Terraform has been successfully initialized!

Run a plan to confirm everything is working end-to-end:

terraform plan

If locking is working, you'll see a brief pause while Terraform acquires the lock before the plan output appears. You'll also see the lock information if you look at the S3 bucket – a .tflock file will appear temporarily alongside your state file during the operation and disappear when it completes.

Part 2: Migration – How to Move from S3 + DynamoDB to S3 Native Locking

Follow this section if you have an existing Terraform setup using an S3 bucket and DynamoDB table for state locking, and you want to migrate to S3 native locking.

Important: Migration requires a maintenance window or at minimum a period where no Terraform operations are running. You're changing the backend configuration, which means all team members and CI/CD pipelines must stop running terraform plan or terraform apply during the migration. The migration itself takes under 10 minutes.

Step 1: Verify Your Current Setup

Before making any changes, document your existing backend configuration and confirm the state file is accessible:

# Confirm your state file is in S3
aws s3 ls s3://your-existing-bucket/path/to/terraform.tfstate
 
# Confirm the DynamoDB table exists
aws dynamodb describe-table \
  --table-name your-dynamodb-lock-table \
  --query 'Table.TableStatus'

Check your current backend.tf and note the exact values:

# Your current backend.tf - note these values before changing anything
terraform {
  backend "s3" {
    bucket         = "your-existing-bucket"       # note this
    key            = "path/to/terraform.tfstate"   # note this
    region         = "us-east-1"                   # note this
    encrypt        = true
    dynamodb_table = "your-dynamodb-lock-table"    # this will be removed
  }
}

Run one final plan to confirm the current state is clean and there are no unexpected changes pending:

terraform plan

If the plan shows no changes, you're in a safe state to proceed.

Step 2: Enable Object Lock on the Existing S3 Bucket

This is the most important step in the migration. Object Lock can't normally be enabled on an existing bucket. It's a setting that must be configured at creation time.

But AWS provides a way to enable Object Lock on an existing bucket through a support request or through a direct API call that's not exposed in the standard console UI. AWS has officially documented this path for the Terraform migration use case.

Run the following AWS CLI command to enable Object Lock on your existing bucket:

aws s3api put-object-lock-configuration \
  --bucket your-existing-bucket \
  --object-lock-configuration '{"ObjectLockEnabled": "Enabled"}'

Note: This command enables Object Lock in governance mode with no default retention, meaning it enables the locking capability without setting a default retention period on all objects. This is exactly what Terraform's native locking needs: the ability to create and delete lock files, not permanent object retention.

Verify Object Lock is now enabled:

aws s3api get-object-lock-configuration \
  --bucket your-existing-bucket

Expected output:

{
    "ObjectLockConfiguration": {
        "ObjectLockEnabled": "Enabled"
    }
}

Also verify that versioning is already enabled (it should be if you are running a production Terraform setup):

aws s3api get-bucket-versioning \
  --bucket your-existing-bucket

Expected output:

{
    "Status": "Enabled"
}

If versioning isn't enabled, enable it before proceeding:

aws s3api put-bucket-versioning \
  --bucket your-existing-bucket \
  --versioning-configuration Status=Enabled

Step 3: Update the Terraform Backend Configuration

Update your backend.tf to remove the dynamodb_table argument and add use_lockfile = true:

terraform {
  backend "s3" {
    bucket = "your-existing-bucket"
    key    = "path/to/terraform.tfstate"
    region = "us-east-1"
    encrypt = true
 
    # Add this:
    use_lockfile = true
 
    # Remove this line entirely:
    # dynamodb_table = "your-dynamodb-lock-table"
  }
}

Your updated backend.tf should look like this:

terraform {
  backend "s3" {
    bucket       = "your-existing-bucket"
    key          = "path/to/terraform.tfstate"
    region       = "us-east-1"
    encrypt      = true
    use_lockfile = true
  }
}

Step 4: Reinitialize Terraform

Run terraform init with the -reconfigure flag. This flag tells Terraform that the backend configuration has changed intentionally and to reinitialize without prompting you to copy state (the state is already in the same bucket):

terraform init -reconfigure

Expected output:

Initializing the backend...
 
Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.
 
Initializing provider plugins...
- Reusing previous version of hashicorp/aws from the dependency lock file
 
Terraform has been successfully initialized!

If you see an error here: The most common cause is that Object Lock wasn't successfully enabled on the bucket. Re-run the verification from Step 2 before proceeding.

Step 5: Verify the Migration

Run a plan to confirm Terraform is working correctly with the new backend configuration:

terraform plan

The plan should:

• Complete successfully

• Show the same result as the plan you ran in Step 1 (no changes, or the same changes as before)

• NOT mention DynamoDB anywhere in its output

To confirm that locking is actually using S3 instead of DynamoDB, open a second terminal and run a plan while the first one is running. You should see the second terminal output a lock error that mentions S3, not DynamoDB:

╷
│ Error: Error acquiring the state lock
│
│Error message: operation error S3: PutObject, https response       error StatusCode: 409,
│ RequestID: ..., api error Conflict: Object lock already exists for this key.
│
│ Lock Info:
│   ID:        a1b2c3d4-e5f6-7890-abcd-ef1234567890
│   Path:      your-existing-bucket/path/to/terraform.tfstate.tflock
│   Operation: OperationTypePlan
│   Who:       user@hostname
│   Version:   1.10.0
│   Created:   2026-05-06 14:22:01 UTC
│   Info:
╵

The Path field shows .tfstate.tflock, a file in your S3 bucket, not a DynamoDB record. This confirms that locking is now handled entirely by S3.

Step 6: Clean Up the DynamoDB Table

Once you've confirmed the migration is working correctly and your team has run at least one successful plan and apply cycle using the new backend, you can remove the DynamoDB table.

Wait at least 24-48 hours before deleting the DynamoDB table if you have CI/CD pipelines or multiple team members. This gives time to catch any pipeline that wasn't updated with the new backend configuration.

When you're ready, delete the DynamoDB table:

aws dynamodb delete-table \
  --table-name your-dynamodb-lock-table

Confirm the deletion:

aws dynamodb describe-table \
  --table-name your-dynamodb-lock-table

Expected output:

An error occurred (ResourceNotFoundException) when calling the DescribeTable operation:
Requested resource not found

This error confirms that the table is gone. The migration is complete.

If you provisioned the DynamoDB table using Terraform (which is the recommended pattern), remove the resource from your Terraform configuration and run terraform apply to destroy it via Terraform rather than the CLI directly. This keeps your state clean:

# Remove this entire block from your Terraform configuration:
resource "aws_dynamodb_table" "terraform_state_lock" {
  name         = "terraform-state-lock"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "LockID"
 
  attribute {
    name = "LockID"
    type = "S"
  }
}

After removing the block, run:

terraform apply

Terraform will detect that the DynamoDB table resource has been removed from configuration and will destroy the table.

How to Verify That Locking Is Working

After completing either the fresh setup or the migration, use this procedure to independently verify that locking is functioning correctly.

Method 1: Observe the lock file during an operation

In one terminal, start a long-running plan against a configuration with many resources:

terraform plan

While it's running, in a second terminal, check for the lock file in S3:

aws s3 ls s3://your-bucket/path/to/ | grep tflock

You should see a file like:

2026-05-06 14:22:01        512 terraform.tfstate.tflock

After the plan completes, run the same command again. The .tflock file should be gone.

Method 2: Read the lock file contents

While a plan is running, download and read the lock file to see its contents:

aws s3 cp \
  s3://your-bucket/path/to/terraform.tfstate.tflock \
  /tmp/current.lock && cat /tmp/current.lock

Expected output (formatted for readability):

{
  "ID": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "Operation": "OperationTypePlan",
  "Info": "",
  "Who": "tolani@dev-machine",
  "Version": "1.10.0",
  "Created": "2026-05-06T14:22:01.123456789Z",
  "Path": "your-bucket/path/to/terraform.tfstate"
}

This is the same lock information that Terraform displays when a lock is held. It's now a JSON file in S3 rather than a record in DynamoDB.

How to Handle a Stuck Lock

With the DynamoDB backend, resolving a stuck lock meant deleting a record from the DynamoDB table. With S3 native locking, it means deleting the .tflock file from S3.

A lock can get stuck if:

• A terraform apply or plan process was killed mid-execution

• A CI/CD pipeline runner crashed during a Terraform operation

• A network interruption prevented the lock release from completing

Here's how you can check for a stuck lock:

aws s3 ls s3://your-bucket/path/to/ | grep tflock

If a .tflock file exists and no Terraform operation is currently running, it is a stuck lock.

You can also read the lock to understand who held it:

aws s3 cp \
  s3://your-bucket/path/to/terraform.tfstate.tflock \
  /tmp/stuck.lock && cat /tmp/stuck.lock

This tells you who (Who field) was running the operation, what operation it was (Operation field), and when it was acquired (Created field).

And you can force-unlock using Terraform like this:

terraform force-unlock LOCK-ID

Replace LOCK-ID with the ID value from the lock file contents. For example:

terraform force-unlock a1b2c3d4-e5f6-7890-abcd-ef1234567890

Terraform will confirm:

Do you really want to force-unlock?
  Terraform will remove the lock on the remote state.
  This will allow local Terraform commands to modify this state, even though it
  may be still be in use. Only 'yes' will be accepted to confirm.
 
  Enter a value: yes
 
Terraform state has been successfully unlocked!

An alternative is to delete the lock file directly via CLI. If terraform force-unlock doesn't work (for example, because you are running in a CI environment without Terraform available), delete the lock file directly:

aws s3 rm s3://your-bucket/path/to/terraform.tfstate.tflock

Only delete the lock file if you are certain no Terraform operation is currently running. Deleting a lock that is actively held by a running operation will allow a second concurrent operation to start, which is exactly the race condition locking is designed to prevent.

Rollback Plan: If Something Goes Wrong

If you encounter problems after migrating, you can roll back to the S3 + DynamoDB setup with these steps.

Step 1: Stop all Terraform operations in your team and CI/CD pipelines.

Step 2: Recreate the DynamoDB table if you already deleted it:

aws dynamodb create-table \
  --table-name terraform-state-lock \
  --attribute-definitions AttributeName=LockID,AttributeType=S \
  --key-schema AttributeName=LockID,KeyType=HASH \
  --billing-mode PAY_PER_REQUEST

Step 3: Revert backend.tf to the previous configuration:

terraform {
  backend "s3" {
    bucket         = "your-existing-bucket"
    key            = "path/to/terraform.tfstate"
    region         = "us-east-1"
    encrypt        = true
    dynamodb_table = "terraform-state-lock"   # restored
    # Remove: use_lockfile = true
  }
}

Step 4: Reinitialize:

terraform init -reconfigure

Step 5: Verify:

terraform plan

The state file hasn't moved, so there's no data loss during a rollback. The only change is which locking mechanism Terraform uses.

Note: Object Lock being enabled on the S3 bucket doesn't prevent the rollback. Object Lock and DynamoDB locking can coexist, Object Lock simply adds a capability to the bucket. Using dynamodb_table in your backend config tells Terraform to use DynamoDB regardless of whether Object Lock is enabled on the bucket.

Security Best Practices for Your State Bucket

Migrating to S3 native locking is a good opportunity to review the overall security configuration of your state bucket. Here are the practices every production Terraform state bucket should implement:

Enable Versioning (Required)

Versioning is a hard requirement for S3 native locking to work safely. It ensures that if a state file is accidentally overwritten or corrupted, you can restore a previous version.

aws s3api put-bucket-versioning \
  --bucket your-state-bucket \
  --versioning-configuration Status=Enabled

Block All Public Access (Non-Negotiable)

Your state file contains resource ARNs, IP addresses, and may contain sensitive values passed through Terraform variables. It must never be publicly accessible.

aws s3api put-public-access-block \
  --bucket your-state-bucket \
  --public-access-block-configuration \
    "BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true"

Enable Server-Side Encryption

Always encrypt state files at rest. AES256 is the minimum. If your organization requires KMS key management:

aws s3api put-bucket-encryption \
  --bucket your-state-bucket \
  --server-side-encryption-configuration '{
    "Rules": [
      {
        "ApplyServerSideEncryptionByDefault": {
          "SSEAlgorithm": "aws:kms",
          "KMSMasterKeyID": "arn:aws:kms:us-east-1:123456789012:key/your-kms-key-id"
        },
        "BucketKeyEnabled": true
      }
    ]
  }'

Apply Least-Privilege IAM Permissions

The role or user that Terraform uses to access the state bucket should have only the permissions it needs. Here's a minimal IAM policy for S3 native locking:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "TerraformStateAccess",
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject"
      ],
      "Resource": [
        "arn:aws:s3:::your-state-bucket",
        "arn:aws:s3:::your-state-bucket/*"
      ]
    },
    {
      "Sid": "TerraformStateLocking",
      "Effect": "Allow",
      "Action": [
        "s3:GetObjectLegalHold",
        "s3:PutObjectLegalHold",
        "s3:GetObjectRetention",
        "s3:PutObjectRetention"
      ],
      "Resource": "arn:aws:s3:::your-state-bucket/*.tflock"
    }
  ]
}

Notice what is absent: there are no DynamoDB permissions. This is a cleaner, smaller permission set than the old approach required.

Enable Access Logging

Log all access to your state bucket in CloudTrail or S3 server access logs. This gives you an audit trail of every time state was read, written, or locked:

aws s3api put-bucket-logging \
  --bucket your-state-bucket \
  --bucket-logging-status '{
    "LoggingEnabled": {
      "TargetBucket": "your-logging-bucket",
      "TargetPrefix": "terraform-state-access/"
    }
  }'

Conclusion

AWS S3 native state locking removes the need for a DynamoDB table from your Terraform backend setup. The result is simpler infrastructure, a smaller IAM permission surface, and one fewer service to provision, monitor, and pay for across every environment your team manages.

Here's a summary of what you accomplished:

• Understood what state locking is and why it's required for safe Terraform operations

• Compared S3 native locking to the existing S3 + DynamoDB approach

• Set up a fresh Terraform backend using S3 native locking with correct bucket configuration

• Migrated an existing backend from S3 + DynamoDB to S3 native locking safely

• Learned how to verify locking, handle stuck locks, and roll back if needed

• Applied security best practices to the state bucket

This pattern – using S3 native locking – is the recommended approach for all new Terraform projects on AWS going forward. If you're managing a large estate with multiple Terraform backends, consider automating the migration using a script or Terraform module that applies the pattern across all your state buckets.

If you are building or optimizing cloud infrastructure for a startup and want a complete reference for production-ready Terraform modules, CI/CD pipeline patterns, and infrastructure runbooks, check out The Startup DevOps Field Guide. It covers the full lifecycle of AWS infrastructure from initial setup to production reliability.

References

• HashiCorp - S3 Backend Configuration: use_lockfile

• HashiCorp: Terraform 1.10 Release Notes

• AWS Docs: S3 Object Lock Overview

• AWS Docs: PutObjectLockConfiguration API

• AWS Docs: S3 Conditional Writes

• HashiCorp: Backend State Locking

• HashiCorp: terraform force-unlock Command

• AWS Docs: Enabling S3 Versioning

• AWS Docs: S3 Server-Side Encryption

To follow along, you should have:

• An AWS account.

• A basic understanding of AWS architecture (You can gain insights here.)

• Knowledge of AWS IAM (Identity and Access Management for secure resource access control)

• Familiarity with Git and GitHub

Let’s dive in and set up our static site hosting step by step.

Table of Contents:

1. What is AWS S3?

   • Fine-Grained Access Controls in S3

   • Encryption at Rest in S3

   • How to Upload a Static Website to Amazon S3

2. What is a Bucket?

   • How to Create a Bucket

   • How to Upload Files and Folders to an S3 Bucket

   • How to Set Permissions and Properties in AWS S3

   • What is a Bucket Policy?

   • How to Set Up a Bucket Policy

3. How to Enable Static Hosting

4. Amazon CloudFront

   • Key Features of Amazon CloudFront

   • Why Amazon S3 Alone is Not Enough

   • Why You Should Serve Your Website with CloudFront

   • What is a CloudFront Distribution?

   • Understanding the Amazon Resource Name (ARN)

   • Updated S3 Bucket Policy

5. Conclusion

What is AWS S3?

Amazon Simple Storage Service (Amazon S3) is an object storage service designed to store and retrieve any amount of data from anywhere.

Using Amazon S3 is straightforward. You start by selecting a region, creating a storage container called a "bucket," and then uploading your data. There is no limit to how much data you can store, and you can retrieve it at any time.

Amazon S3 automatically creates backups of your data by storing copies across multiple devices. It also allows you to preserve different versions of your files, helping you recover data if it’s accidentally lost. If you delete a file by mistake, you can restore it using Amazon S3’s versioning feature.

Amazon S3 offers configurable lifecycle policies to help manage data efficiently throughout its lifecycle. Security is a top priority for AWS, ensuring that data uploads and retrievals are protected using SSL encryption for secure transmission. AWS also provides multiple security features to safeguard your data, including fine-grained access controls and encryption at rest. Let’s explore these two features in a bit more detail.

Fine-Grained Access Controls in S3

Amazon S3 provides fine-grained access control, allowing you to define who can access your data and what actions they can perform. This is managed through:

1. AWS Identity and Access Management (IAM): Controls user permissions at the AWS account level. You can grant specific users or roles permissions to access S3.

2. S3 bucket policies: JSON-based policies applied at the bucket level to control access for all objects in a bucket.

3. Access Control Lists (ACLs): Defines permissions for individual objects within a bucket (less commonly used since bucket policies are more powerful).

4. Block public access settings: Prevents accidental public exposure of S3 data by restricting open access.

Encryption at Rest in S3

Encryption at rest ensures that stored data remains secure, even if unauthorized access occurs. S3 supports multiple encryption options:

1. Server-Side Encryption (SSE):

   • SSE-S3: AWS manages encryption keys automatically.

   • SSE-KMS: Uses AWS Key Management Service (KMS) for additional control over encryption keys.

   • SSE-C: Customers provide their own encryption keys.

2. Client-Side Encryption:

   • Data is encrypted before being uploaded to S3 using customer-managed encryption keys.

We could spend endless time researching and exploring the theory behind Amazon S3, but practical application solidifies learning. Now, let’s move on to uploading a static website to Amazon S3.

As mentioned earlier, you should have a basic understanding of how AWS works, including signing up, signing in, and creating IAM users. This is crucial because we will use an IAM user to perform operations securely. Understanding Identity and Access Management is essential in AWS, as it ensures proper access control and security while managing resources.

How to Upload a Static Website to Amazon S3

To upload a static website, you first need a static site. If you do not have one, you can use a free template from Free CSS.

I’ve also provided a ready-to-use template that you can clone from this GitHub repository: Mediplus Free Template.

Now that your static project is ready, let’s go to AWS and upload it to an Amazon S3 bucket.

Login to your AWS account using your IAM user credentials. Once logged in, you will be redirected to the AWS Management Console.

Your AWS Dashboard should look similar to this:

Navigate to the S3 service by clicking on the S3 link in the AWS dashboard. If you don’t see it, which is unlikely, use the search bar at the top of the dashboard. Simply type "S3", and it will appear in the results. Click on it to proceed.

Once you arrive at the Amazon S3 page, you will see the Create Bucket button.

S3 buckets are the backbone for many applications, including content delivery, data backup, archiving, static website hosting, and big data storage for analytics.

What is a Bucket?

Amazon S3 buckets are the fundamental storage containers within AWS Simple Storage Service that provide secure, scalable repositories for digital assets.

Each bucket features a globally unique name, regional deployment, and flat object storage architecture identified by unique keys.

With 99.999999999% durability through built-in redundancy, S3 buckets support crucial infrastructure needs including content distribution, data archiving, and static website hosting. Admins can implement comprehensive data governance through configurable access controls (policies, ACLs, IAM), lifecycle management, versioning capabilities, and encryption protocols to meet organizational security requirements.

How to Create a Bucket

To create a bucket, click on the "Create bucket" button. You will then be redirected to the bucket creation page.

Choose any name you like, as long as it follows AWS bucket naming rules.

You will also see various configuration options, but for now, leave them as default. We will make the necessary adjustments later in the project.

Finally, click on "Create bucket", and just like that your bucket is created! You should now see your bucket, which acts as a container for storing your files or data:

How to Upload Files and Folders to an S3 Bucket

Now, let’s upload the static site we created or cloned to our S3 bucket.

1. Click on your bucket:

After clicking on your bucket, you will be taken to the bucket details page, where you can manage various settings and configurations. This page provides options to adjust permissions and properties, monitor metrics, manage access points, and upload files using the "Upload" button.

2. Upload project files:

If your static site project is ready, it should contain the essential files needed for deployment. While the structure may vary, it should include an index.html file, along with necessary assets such as images (or an image folder), CSS files (or a CSS folder), and JavaScript files (or a JS folder) to ensure proper functionality and styling.

Let’s start by uploading the necessary files according to our project structure. Start with the root-level files such as index.html, along with any other essential files on the root-level. Ensure you follow your project structure carefully and upload all required files first to maintain proper structure.

Then click on the upload button:

On the Upload page, you will see the "Add files" and "Add folder" buttons. Let's start by uploading individual files. Click on "Add files" to begin selecting and uploading the necessary files.

After successfully uploading your files, ensure you follow your project structure. If your static site only requires individual files, proceed accordingly. But if your project relies on specific folders for assets like images and CSS, you’ll need to upload those as well. In my case, my project structure includes folders for images and CSS e.t.c, so I will be uploading both files and folders.

3. Upload project folders

To upload your folders click on the “Add folder” button.

Now, upload your folders by clicking on the "Add folder" button and selecting the necessary folders, such as those containing images, CSS, or JavaScript files, based on your project structure.

You will notice that as you upload folders, S3 automatically extracts and structures the files by name, type, and size. This can sometimes be confusing for beginners, as the files are displayed in a structured format.

Once you have successfully uploaded your files and folders, scroll down and click the "Upload" button to complete the process.

After clicking on "Upload", AWS S3 will begin uploading your files. You will see a progress indicator showing the status of each file being uploaded in real time.

Upon completing the upload, you will see a confirmation message stating "Upload Succeeded." AWS S3 will then generate a URL which you can find in the Summary section of the bucket’s details page. (Refer to the image below for reference.)

Congratulations! you have successfully uploaded your project into AWS S3. Now let’s make this accessible on the web.

Click on the "Close" button to exit the upload page. Now, while still in your bucket, let's configure the necessary settings and permissions to ensure proper access and functionality for our static website.

How to Set Permissions and Properties in AWS S3

After uploading your files, the next step is to configure permissions and properties to ensure your static website functions correctly.

• Permissions: By default, S3 objects are private. To make your website publicly accessible, you need to adjust the bucket policy and object permissions accordingly.

• Properties: You can configure various settings such as versioning, encryption, and static website hosting under the Properties tab of your bucket, but we won’t be going too deep as we are just going through the basics.

Let’s start with setting up permissions:

Permissions

By default, public access is blocked in S3 for security reasons. But for this tutorial, we will be enabling public access to ensure our static website is accessible to users.

In the image below, you can see that the public access is blocked by default:

To enable public access, click on the "Edit" button in the top-right corner. Then, uncheck the "Block all public access" option. After that, click on "Save changes" to apply the update.

Now you can see that the public access has been blocked:

Public access is now enabled. Click on "Save changes" to confirm and apply the update.

Enabling public access

AWS S3 will prompt you to enter a confirmation text to validate the settings. Input the required text and confirm, and public access will be enabled successfully.

So now that we have enabled public access, let’s set a bucket policy.

What is a Bucket Policy?

A bucket policy is a JSON-based access control policy that defines permissions for your S3 bucket. It allows you to specify who can access your bucket and what actions they can perform.

With a bucket policy, you can:

• Grant or restrict public access to objects in your bucket.

• Allow specific AWS users or services to interact with your bucket.

• Define read, write, or delete permissions for different users.

In this next section, we will configure a bucket policy to make our static website publicly accessible.

How to Set Up a Bucket Policy

Setting up a bucket policy is simple. Click on the "Edit" button in the Bucket Policy section and paste the following JSON policy into the editor:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "PublicReadGetObject",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::your-bucket-name/*"
    }
  ]
}

Understanding the policy attributes:

• Version: Defines the policy language version. The "2012-10-17" version is the latest and most commonly used for S3 policies.

• Statement: A list of rules that define what actions are allowed or denied.

• Sid (Statement ID): A unique identifier for the policy statement (optional but useful for reference).

• Effect: Specifies whether the rule allows or denies the specified action. In this case, it’s "Allow".

• Principal: Defines who has access. The "*" means anyone (public access).

• Action: Specifies the allowed operation. "s3:GetObject" allows users to retrieve (read) objects from the bucket.

• Resource: Defines the specific bucket and objects the policy applies to. Replace "your-bucket-name" with your actual bucket name. The "/*" means all objects within the bucket.

This policy makes all objects in the bucket publicly readable, allowing users to access your static website files via a browser.

if you followed closely, your bucket policy page should look like this:

Now, click on the "Save changes" button, usually located at the bottom right of the page, to apply your changes.

After following the required steps Bucket policy has been set and public access has been enabled.

Congratulations! You have successfully set up your bucket policy and enabled public access, meaning anyone can now access your website. But wait, where's the URL?

There's one last step: we need to enable static website hosting. To do this, you’ll need to navigate to the Properties tab and configure static website hosting so your site can be accessed on the web. I’ll walk you through the process in the next section.

How to Enable Static Hosting

Enabling static website hosting is simple. Scroll to the bottom of the Properties tab, and you will find the Static website hosting section. By default, this option is disabled. Let's enable it to make our website accessible on the web.

Click on the "Edit" button on the right side of the Static website hosting section. By default, this option is disabled, so change it to Enabled. Once enabled, you will see several configuration options, most of which are optional for this tutorial.

The most important setting here is the Index document, which specifies the default file that loads when someone accesses your site. The placeholder text indicates that it expects an index.html file. Simply type index.html in the Index document field to proceed.

After entering index.html in the Index document field, scroll down and click on "Save changes" to apply the configuration.

After successfully applying the changes, you should find your Static Website URL at the bottom of the Static website hosting section in the Properties tab of your bucket.

Congratulations! You have just hosted your website on AWS S3. That’s a solid step into the world of DevOps.

Your URL should look something like this: http://your-bucket-name.s3-website.your-region.amazonaws.com/.

Simply copy and paste your Static Website URL into your browser to see your hosted site live.

Now, you have the skills to host a static website for a client and share the URL with them.

So go grab some coffee and banana bread, you have earned it. Then we’ll move on to the next part of the tutorial.

Amazon CloudFront

Amazon CloudFront is a fast, highly secure, and programmable content delivery network (CDN) that improves website performance and security.

Our static website is hosted on S3(http://your-bucket-name.s3-website.your-region.amazonaws.com/) – but you might notice that the site is accessible via HTTP but lacks proper security measures such as SSL/TLS encryption. CloudFront addresses these limitations by providing a secure and scalable way to serve both static and dynamic content globally.

It delivers content to users with low latency by caching copies of your website at edge locations worldwide. When a user requests a resource, CloudFront serves it from the nearest edge location, significantly improving speed and availability.

Key Features of Amazon CloudFront

1. Secure content delivery: CloudFront supports SSL/TLS encryption, ensuring data is transferred securely between clients and servers.

2. DDoS protection: Integrated with AWS Shield, CloudFront helps mitigate Distributed Denial of Service (DDoS) attacks.

3. Global content caching: CloudFront caches content at multiple edge locations, reducing server load and latency.

4. Customizable distribution: Users can configure cache behavior, origin settings, and security policies.

5. Seamless integration with AWS tools: CloudFront integrates with AWS Lambda@Edge, S3, EC2, and API Gateway, supporting both static and dynamic content delivery.

6. Cost optimization: Reduces data transfer costs by caching and serving content from edge locations instead of directly from the origin.

Why Amazon S3 Alone is Not Enough

Amazon S3 is a scalable and durable object storage service, but it lacks key features required for securely serving web content.

First of all, it doesn’t have HTTPS by default. When hosting a site on S3, it is only accessible over HTTP unless additional configurations are made.

Second, it has higher latency. S3 buckets are hosted in a specific AWS region, which may result in slower content delivery for users in different locations.

It also doesn’t have built-in caching, which means that every request is served from S3, increasing response time and potential costs.

And finally, there’s no DDoS protection – unlike CloudFront, S3 does not provide native protection against cyberattacks.

Why You Should Serve Your Website with CloudFront

While Amazon S3 is a great storage solution, it lacks the security and performance optimizations needed for web hosting. Amazon CloudFront enhances security with SSL/TLS encryption, improves performance with global caching, and provides robust security features like DDoS protection.

By leveraging CloudFront, you can ensure your website is not only fast but also secure, scalable, and cost-effective.

To get started with hosting your site on CloudFront, navigate to the CloudFront service page in AWS. You can do this by going to your AWS Management Console homepage or dashboard and searching for “CloudFront” using the search bar in the top left corner. Click on “CloudFront” from the search results to proceed.

Once you navigate to the AWS CloudFront page, you will see a Create Distribution button. Click on it to begin setting up CloudFront for your website.

What is a CloudFront Distribution?

A CloudFront distribution is the configuration that defines how CloudFront delivers content to users. It acts as a link between your origin server (such as an S3 bucket) and CloudFront’s global network of edge locations. When a user requests your site, CloudFront retrieves content from the nearest edge location instead of always fetching it from the origin, ensuring faster load times, reduced latency, and improved security.

There are two types of distributions in CloudFront:

1. Web Distribution: Used for websites, APIs, and dynamic or static content.

2. RTMP Distribution (Deprecated): Previously used for streaming media (now replaced by modern streaming services).

For our S3-hosted website, we will be creating a Web Distribution to securely serve content over HTTPS while improving speed and reliability.

After clicking the Create Distribution button, you will be directed to the Create Distribution page, where you can configure various settings. In this tutorial, we will focus on the essential options.

1. Under the Origin section, select your S3 bucket as the origin domain.

2. If your S3 bucket has static website hosting enabled, AWS recommends using the S3 website endpoint instead of the default bucket endpoint.

3. Enter the correct S3 website endpoint in the Origin domain field. For example:

    freecodecampbuckettutorial.s3-website.ca-central-1.amazonaws.com
   

4. Ensure you use the S3 website endpoint rather than the standard S3 bucket URL to avoid issues with accessing your site.

By following these steps, you ensure that CloudFront correctly serves your static website from the S3 bucket.

Make sure that you use the S3 website endpoint rather than the standard S3 bucket URL to avoid issues with accessing your site. This will be suggested to you descriptively while filling in the origin name. Click on “Use website endpoint” and it will populate the field with the S3 bucket website endpoint instead of the bucket URL.

Next, scroll down to the Web Application Firewall (WAF) section at the bottom of the page and enable security protections to safeguard your website from common web threats. Select “Create Distribution” to deploy your CloudFront distribution.

Your CloudFront Distributions page should now display the newly created distribution. The page will include key details such as:

• Distribution ID: A unique identifier for your CloudFront distribution.

• Domain Name: The CloudFront-provided URL (for example, d1234abcd.cloudfront.net), which you can use to access your site.

After creating your CloudFront distribution, navigate to your Distributions page and check the Last Modified section for its status. If the status shows Deploying, you will need to wait until it changes, which may take several minutes.

Once the deployment is complete, the status will typically update to a timestamp, indicating that the distribution is ready for use. Ensure the status has changed before proceeding with further configurations or accessing your CloudFront distribution.

Your website is now successfully hosted on CloudFront!

We now have our CloudFront Domain Name (for example, d1234abcd.cloudfront.net), which you can find in the Details section of your distribution. Before making any further changes, let’s preview the site by copying and pasting the domain name into a web browser.

At this stage, when you try to access your website using the CloudFront Domain Name, you’ll notice that the site cannot be reached. This happens because CloudFront does not yet have permission to fetch content from your S3 bucket.

To fix this, you need to update your S3 bucket policy to explicitly allow CloudFront to access your objects. You can do this by adding a condition that grants access to requests coming specifically from your CloudFront distribution.

Understanding the Amazon Resource Name (ARN)

An Amazon Resource Name (ARN) is a unique identifier assigned to AWS resources. Every CloudFront distribution has its own ARN, which you can find at the top of the CloudFront distribution details page. It looks like something like this:

arn:aws:cloudfront::123456789012:distribution/E2ABC3XYZ456

This ARN is crucial because we use it in our bucket policy to restrict access to only our CloudFront distribution, ensuring no other services or users can retrieve data directly from our S3 bucket.

Updated S3 Bucket Policy

To allow CloudFront to serve content from our S3 bucket, we update the S3 bucket policy as follows:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "cloudfront.amazonaws.com"
      },
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::your-bucket-name/*",
      "Condition": {
        "StringEquals": {
          "AWS:SourceArn": "arn:aws:cloudfront::[ACCOUNT_ID]:distribution/[DISTRIBUTION_ID]"
        }
      }
    },
    {
      "Effect": "Allow",
      "Principal": "*",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::your-bucket-name/*"
    }
  ]
}

Breaking down the bucket policy:

1. First statement that grants CloudFront access:

   • “Effect”: “Allow”: This permits access to the specified resource.

   • "Principal": { "Service": "cloudfront.amazonaws.com" }: Grants access specifically to CloudFront.

   • "Action": "s3:GetObject": Allows CloudFront to retrieve objects from the S3 bucket.

   • "Resource": "arn:aws:s3:::your-bucket-name/*": Grants access to all objects in the S3 bucket.

   • “Condition”: Ensures that only requests originating from our CloudFront distribution are allowed using:

       AWS:SourceArn": "arn:aws:cloudfront::123456789012:distribution/E2ABC3XYZ456
     

2. Second statement (optional, and grants public access)

   • This statement allows all users (Principal: "*") to access the S3 objects.

   • If you want to restrict access only to CloudFront, you can remove this second statement.

After editing and updating your S3 bucket policy to allow CloudFront access, you can refresh the page where you preview your CloudFront domain name (for example, d1234abcd.cloudfront.net). Open it in a browser, and if you have carefully followed all the instructions, you have successfully hosted a static site on S3 and CloudFront.

Your website is now fully protected, well done! Congratulations, DevOps pro!

FIST BUMPS!

Conclusion

In this guide, we’ve only scratched the surface of what AWS can do. S3 and CloudFront are powerful services, but there’s still so much more you can explore, from advanced security settings to automation and performance optimizations.

As you continue your AWS journey, you can dive deeper into topics like caching strategies, custom domain configurations, and integrating AWS Lambda@Edge for dynamic content. The possibilities are endless.

This is just the beginning, and you’re off to a great start. Keep experimenting, keep learning, and soon we will be mastering even more advanced AWS capabilities. Happy building! 🚀

Using traditional single-part uploads can be cumbersome and inefficient for large files, often leading to timeout errors or the need to restart the entire upload process if any part fails. This is where the Amazon S3 multipart upload feature comes into play, offering a robust solution to these challenges.

In this article, you'll explore how to efficiently handle large files with Amazon S3 multipart upload. We'll discuss the benefits of using this feature, walk through the process of uploading files in parts, and provide code examples using the AWS SDK for full-stack Node and React project.

By the end of this article, you should have a good understanding of how to leverage the Amazon S3 multipart upload to optimize file uploads in your applications.

Prerequisites

Before we start, ensure you have the following:

• An AWS account with IAM user credentials.
• Node.js installed on your development machine.
• Basic knowledge of JavaScript, React, and Node.js.

Table of Contents:

• Introduction
• Prerequisites
• Table of Contents
• How it works
• Step 1: How to Set Up AWS S3
• How to Create an S3 Bucket
• How to Configure s3 Bucket Policy
• Step 2: How to Set Up AWS S3 Backend with Node.js
• How to Initialize a Node.js Project
• Install Required Packages
• Create Server file
• Imports and configuration
• Middleware and AWS Configuration
• Routes
• Start/Initialize Upload Endpoint
• Upload Part Endpoint
• Complete Upload Endpoint
• Start the Server
• Environment Variables
• Running the Server
• Step 3: How to Set Up the Frontend with React
• How to Initialize a React Project
• Install Required Packages
• Create Components
• App Component
• Testing
• Part Upload
• Complete Part Upload
• Full Code on GitHub
• Conclusion

How It Works

A large file upload is divided into smaller parts/chunks, each part is uploaded independently to Amazon S3. Once all the parts have been uploaded, they are combined to create the final object.

Example: Uploading a 100MB file in 5MB parts would result in 20 parts being uploaded to S3. Each part is uploaded with a unique identifier, and the order is maintained to ensure that the file can be reassembled correctly.

Retries can be configured to automatically retry failed parts, and the upload can be paused and resumed at any time. This makes the process more robust and fault-tolerant, especially for large files.

multipart AWS s3 uploads

Learn more on the Amazon S3 multipart upload docs.

Let's get started!

Step 1: How to Set Up AWS S3

How to Create an S3 Bucket

First, log into the AWS Management console

• Navigate to the S3 service.

How to create an s3 bucket

Create a new bucket and take note of the bucket name.

Uncheck the Public Access settings for simplicity We'll also configure bucket access using IAM policies after creating the bucket.

How to create an s3 bucket

• Leave other settings as default and create the bucket.

How to Configure S3 Bucket Policy

Now, that you have created the bucket, let's set up the policy to allow users read your objects(file/videos) url.

• Click on the bucket name and navigate to the Permissions tab.

How to configure s3 bucket policy

Navigate to the Bucket Policy section and click on Edit.

Input the following policy, and replace your-bucket-name with your actual bucket name:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": "*",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::your-bucket-name/*"
    }
  ]
}

Version: Amazon S3 object version number for the bucket policy language.

Statement: An array of one or more individual statements that define the policy.

Effect: The effect determines whether the statement allows or denies access.

Principal: The entity that the policy is applied to. In this case, we are allowing all principals. In production, you should specify the IAM user or role that needs access.

Action: The action that the policy allows or denies. In this case, we are allowing the s3:GetObject action, which allows users to retrieve objects from the bucket.

Resource: The Amazon Resource Name (ARN) of the bucket and objects that the policy applies to. In this case, we are allowing access to all objects in the bucket.

Click on Save changes to apply the policy.

Step 2: How to Set Up AWS S3 Backend with Node.js

Next, let's set up the backend server with AWS SDK to handle the file upload process.

How to Initialize a Node.js Project

Create a new directory for your project and initialize a new Node.js project:

mkdir s3-multipart-upload
cd s3-multipart-upload
npm init -y

Install Required Packages

Install the following packages using npm:

 npm install express dotenv multer aws-sdk

Create Server File

Create a new file named app.js (For simplicity, we are going to use this file only for all the upload logic) and add the following code:

Imports and Configurations

const cors = require("cors");
const express = require("express");
const AWS = require("aws-sdk");
const dotenv = require("dotenv");
const multer = require("multer");

const multerUpload = multer();
dotenv.config();

const app = express();
const port = 3001;

Imports

cors: Middleware for enabling Cross-Origin Resource Sharing (CORS). This is necessary to allow your frontend application interact with the backend hosted on a different domain or port.

express: A minimal and flexible Node.js web application framework.

AWS: The AWS SDK for JavaScript, which allows you to interact with AWS services.

dotenv: A module that loads environment variables from a .env file into process.env.

multer: Middleware for handling multipart/form-data, which is primarily used for uploading files.

Configurations

multerUpload: Initializes multer for handling file uploads.

dotenv.config(): Loads the environment variables from a .env file.

app: Initializes an Express application.

port: Sets the port on which the Express application will run.

Middleware and AWS Configuration

Next, add the following code to configure middleware and AWS SDK:

app.use(cors());

AWS.config.update({
  accessKeyId: process.env.AWS_ACCESS_KEY,
  secretAccessKey: process.env.AWS_SECRET_KEY,
  region: process.env.AWS_REGION,
});

const s3 = new AWS.S3();
app.use(express.json({ limit: "50mb" }));
app.use(express.urlencoded({ limit: "50mb", extended: true }));

app.use(cors()): Enables CORS for all routes, allowing your frontend to communicate with the backend without issues related to cross-origin requests.

AWS.config.update({ ... }): Configures the AWS SDK with the access key, secret key, and region from the environment variables.
const s3 = new AWS.S3(): Creates an instance of the S3 service.

app.use(express.json({ limit: '50mb' })): Configures Express to parse JSON bodies with a size limit of 50MB.

app.use(express.urlencoded({ limit: '50mb', extended: true })): Configures Express to parse URL-encoded bodies with a size limit of 50MB.

Routes

It's time to start creating our routes. The routes required for the multipart upload process are as follows:

• Initialization of the upload process.
• Uploading parts of the file.
• Completing the upload process.

Start/Initialize Upload Endpoint

This route puts the upload process in play. Add the following code to create an endpoint for initializing the multipart upload process:

app.post("/start-upload", async (req, res) => {
  const { fileName, fileType } = req.body;

  const params = {
    Bucket: process.env.S3_BUCKET,
    Key: fileName,
    ContentType: fileType,
  };

  try {
    const upload = await s3.createMultipartUpload(params).promise();
    // console.log({ upload });
    res.send({ uploadId: upload.UploadId });
  } catch (error) {
    res.send(error);
  }
});

The function above creates a POST endpoint /start-upload that expects a JSON body with fileName and fileType properties. It then uses the createMultipartUpload method from the S3 service to initialize the multipart upload process. If successful, it returns the uploadId to the user, which will be used to upload parts of the file.

Upload Part Endpoint

This is the route where the different smaller parts of the large file upload are received and tagged. Add the following code to create an endpoint for uploading parts of the file:

app.post("/upload-part", multerUpload.single("fileChunk"), async (req, res) => {
  const { fileName, partNumber, uploadId, fileChunk } = req.body;

  const params = {
    Bucket: process.env.S3_BUCKET,
    Key: fileName,
    PartNumber: partNumber,
    UploadId: uploadId,
    Body: Buffer.from(fileChunk, "base64"),
  };

  try {
    const uploadParts = await s3.uploadPart(params).promise();
    console.log({ uploadParts });
    res.send({ ETag: uploadParts.ETag });
  } catch (error) {
    res.send(error);
  }
});

The function above creates a POST endpoint at /upload-part that expects a form-data body with uploadId, partNumber, and fileName properties. It uses the uploadPart method from the S3 service to upload the part of the file. If successful, it returns the ETag of the uploaded part to the client.

The ETag is a unique identifier for the upload part that will be used to complete the multipart upload.

Complete Upload Endpoint

Once the part has been uploaded, the final step is to combine all the parts to create the final object.

Add the following code to create an endpoint for completing the multipart upload process:

app.post("/complete-upload", async (req, res) => {
  const { fileName, uploadId, parts } = req.body;

  const params = {
    Bucket: process.env.S3_BUCKET,
    Key: fileName,
    UploadId: uploadId,
    MultipartUpload: {
      Parts: parts,
    },
  };

  try {
    const complete = await s3.completeMultipartUpload(params).promise();
    console.log({ complete });
    res.send({ fileUrl: complete.Location });
  } catch (error) {
    res.send(error);
  }
});

The function above creates a POST endpoint at /complete-upload that expects a JSON body with uploadId, fileName, and parts properties. It uses the completeMultipartUpload method from the S3 service to combine the uploaded parts and creates the final object. If successful, it returns the data object containing fileUrl about the completed upload.

Start the Server

Finally, add the following code to start the Express server:

app.listen(port, () => {
  console.log(`Server running on port ${port}`);
});

This code starts the Express server on port 3001 and logs a message to the console when the server is running.

Environment Variables

Create a new file named .env in the root directory of your project and add the following environment variables:

AWS_ACCESS_KEY=your-access-key
AWS_SECRET_KEY=your-secret-key
AWS_REGION=your-region
S3_BUCKET=your-bucket-name

Replace your-access-key, your-secret-key, your-region, and your-bucket-name with your actual AWS credentials and bucket name.

Running the Server

To run the server, execute the following command in your terminal:

node app.js

This will start the server on port 3001.

Step 3: How to Set Up the Frontend with React

Now that the backend is set up, let's create a React frontend to interact with the server and upload files to S3 using the multipart upload process.

The frontend will be in charge of splitting the file into parts, uploading each part to the server, and completing the upload process.

How to Initialize a React Project

Create a new React project using Create React App:

npx create-react-app s3-multipart-upload-frontend
cd s3-multipart-upload-frontend

Install Required Packages

Install the following packages using npm:

  npm install axios

Create Components

Create a new file named Upload.js in the src/components directory and add the following code:

import React, { useState } from "react";
import axios from "axios";

const CHUNK_SIZE = 5 * 1024 * 1024; // 5MB

const FileUpload = () => {
  const [file, setFile] = useState(null);
  const [fileUrl, setFileUrl] = useState("");

  const handleFileChange = (e) => {
    setFile(e.target.files[0]);
  };

  const handleFileUpload = async () => {
    const fileName = file.name;
    const fileType = file.type;
    let uploadId = "";
    let parts = [];

    try {
      // Start the multipart upload
      const startUploadResponse = await axios.post(
        "http://localhost:3001/start-upload",
        {
          fileName,
          fileType,
        }
      );

      uploadId = startUploadResponse.data.uploadId;

      // Split the file into chunks and upload each part
      const totalParts = Math.ceil(file.size / CHUNK_SIZE);

      console.log(totalParts);

      for (let partNumber = 1; partNumber <= totalParts; partNumber++) {
        const start = (partNumber - 1) * CHUNK_SIZE;
        const end = Math.min(start + CHUNK_SIZE, file.size);
        const fileChunk = file.slice(start, end);

        const reader = new FileReader();
        reader.readAsArrayBuffer(fileChunk);

        const uploadPart = () => {
          return new Promise((resolve, reject) => {
            reader.onload = async () => {
              const fileChunkBase64 = btoa(
                new Uint8Array(reader.result).reduce(
                  (data, byte) => data + String.fromCharCode(byte),
                  ""
                )
              );

              const uploadPartResponse = await axios.post(
                "http://localhost:3001/upload-part",
                {
                  fileName,
                  partNumber,
                  uploadId,
                  fileChunk: fileChunkBase64,
                }
              );

              parts.push({
                ETag: uploadPartResponse.data.ETag,
                PartNumber: partNumber,
              });
              resolve();
            };
            reader.onerror = reject;
          });
        };

        await uploadPart();
      }

      // Complete the multipart upload
      const completeUploadResponse = await axios.post(
        "http://localhost:3001/complete-upload",
        {
          fileName,
          uploadId,
          parts,
        }
      );

      setFileUrl(completeUploadResponse.data.fileUrl);
      alert("File uploaded successfully");
    } catch (error) {
      console.error("Error uploading file:", error);
    }
  };

  return (
    <div>
      <input type="file" onChange={handleFileChange} />
      <button disabled={!file} onClick={handleFileUpload}>
        Upload
      </button>
      <hr />
      <br />
      <br />
      {fileUrl && (
        <a href={fileUrl} target="_blank" rel="noopener noreferrer">
          View Uploaded File
        </a>
      )}
    </div>
  );
};

export default FileUpload;

The FileUpload component above handles the file upload process using the multipart upload method. It splits the file into chunks, uploads each part to the server, and completes the upload process.

The component consists of the following key parts:

CHUNK_SIZE: The size of each part in bytes. In this case, we are using 5MB parts.

handleFileChange: A function that sets the selected file in the state.

handleFileUpload: A function that initiates the multipart upload process by sending the file to the server in parts.

• It starts the upload process by calling the /start-upload endpoint and retrieves the uploadId.
• It splits the file into chunks and uploads each part to the server using the /upload-part endpoint.
• It completes the upload process by calling the /complete-upload endpoint with the uploadId and parts array.

fileUrl: A state variable that stores the URL of the uploaded file.

The component renders an input field for selecting a file, a button to upload the file, and a link to view the uploaded file.

App Component

Update the App.js file in the src directory with the following code:

import React from "react";

import FileUpload from "./components/FileUpload";

function App() {
  return (
    <div className="App">
      <h1>Large File Upload with S3 Multipart Upload</h1>
      <FileUpload />
    </div>
  );
}


export default App;

The App component renders the FileUpload component, which handles the file upload process.

How to Start the Frontend

To run the frontend, execute the following command in your terminal:

npm start

This will start the React development server on port 3000 and open the application in your default web browser.

Testing

Let's test the application by uploading a large file using the frontend. You should see the file being uploaded in parts and then combined to create the final object on the server inspecting your network tab.

Part Upload

In the image below, the start-upload endpoint is called to initialize and start the upload process. The large file uploaded is broken into chunks and uploaded with the upload-part endpoint. You can see up to 10 or more (depending on the size of each chunk to the total file size).

Each upload part has a unique identifier Etag used for the complete upload.

Image uploading in parts

Complete Part Upload

The last and final step of the process is the complete-upload endpoint where the upload parts are combined to form a single object for the file uploaded.

Image uploads completed

You can click on the View Uploaded File to access your uploaded file.

Full Code on GitHub

Click the link below to access the full code on GitHub:

Multipart file uploads with react and NodeJS

Conclusion

In this article, we explored how to efficiently handle large files with Amazon S3 multipart upload. We discussed the benefits of using this feature, walked through the process of uploading files in parts, and provided code examples using Node.js and React.

This is a high-level implementation of the multipart upload process, you can further enhance it by adding more features like progress tracking, error handling, and resumable uploads.

By leveraging Amazon S3 multipart upload, you can optimize file uploads in your applications by dividing large files into smaller parts, uploading them independently, and combining them to create the final object. This approach not only enhances upload performance but also adds fault tolerance and flexibility to pause and resume uploads, making it ideal for handling large files over unstable networks.

This article will discuss the common S3 performance bottlenecks and how they impact the service's overall performance. It will also share some best practices to serve as guidelines for optimising the performance of your S3.

By the end, you'll have practical tips to ensure your AWS S3 runs at its best and handles all your data smoothly.

S3 Performance Bottlenecks

Here are some common bottlenecks that influence the overall performance of S3:‌‌

Network Latency:

Network latency is the delay incurred when data travels between your application and the S3 storage. This delay arises due to the physical distance, the number of network hops, and the speed at which data can be transmitted.

In simpler terms, it's like the time it takes for your request to reach the S3 server and the response to find its way back. ‌‌

Request Processing Time:

When you send a request to Amazon S3, the time it takes for the system to comprehend and fulfill that request is called request processing time. It involves authentication, authorisation, and any necessary computation before responding. ‌‌

Server-Side Processing:

Once your request reaches the S3 server, it undergoes processing to fulfill your requested action. This can involve tasks like encryption, access control checks, and other operations that impact the time it takes to serve your data.

While these operations are essential, they can become bottlenecks if not managed efficiently.

Impact of Bottlenecks on Overall S3 Performance

These bottlenecks, individually or collectively, can affect the overall performance of Amazon S3. Here are some of the impacts: ‌‌

Slower Data Retrieval:

Network latency and request processing time can lead to delays in retrieving data, affecting the overall speed of your applications relying on S3. Slow data retrieval can impact user experience and application responsiveness, such as waiting for a webpage to load.‌‌

Reduced Throughput:

Bottlenecks, especially in server-side processing and request processing time, can limit the amount of data transferred at a given time. This reduction in throughput can impact the speed at which your applications can read or write data to S3.‌‌

Increased Costs:

Inefficiencies in data transfer and processing can contribute to increased costs. Longer processing times and additional network usage can result in higher expenses for S3 usage. Identifying and mitigating bottlenecks can lead to cost savings and more efficient resource utilisation.‌‌

Understanding and addressing these common bottlenecks is essential for unlocking the full potential of Amazon S3. The following section will address the best practices to optimise S3 performance and ensure a seamless cloud storage experience.‌‌

Best Practices for S3 Performance Optimization

Amazon S3 Transfer Acceleration

Transferring data to and from Amazon S3 may take time due to the physical distance between your location and the S3 server.

Amazon S3 Transfer Acceleration is a bucket-level feature that enables fast, easy, and secure transfers of files over long distances between your client and an S3 bucket. ‌‌

When to opt for transfer acceleration

You might want to use Transfer Acceleration on a bucket for various reasons:

• If you're far from the S3 server, Transfer Acceleration is your shortcut. It's like taking a direct flight instead of a connecting one.
• For real-time applications or situations where time is of the essence, the speed boost from Transfer Acceleration can be a game-changer.
• You need to transfer gigabytes to terabytes of data regularly across continents.
• You can't use all of your available bandwidth over the internet when uploading to Amazon S3.‌‌

Things to keep in mind

• Cost Considerations: While Transfer Acceleration speeds up your data, it comes with additional costs.
• The bucket name used for Transfer Acceleration must be DNS-compliant and not contain periods (".").
• Transfer Acceleration must be enabled on the bucket.
• After you enable Transfer Acceleration on a bucket, it might take up to 20 minutes before the data transfer speed to the bucket increases.
• Transfer Acceleration is only supported in certain following regions.
• You must be the bucket owner to set the transfer acceleration state.‌‌

Benefits of Transfer Acceleration

• Faster Uploads and Downloads: Your data gets to its destination quicker, making uploads and downloads snappier. It's like sending a package with express shipping instead of regular mail.
• Improved User Experience: For applications relying on S3, users experience reduced wait times.
• Global Reach, Local Touch: Whether your users are in New York, Tokyo, or spread around the world, they experience speed as if the data is right next door. It's like having a local store in every city.‌‌

AWS provides a Transfer Acceleration speed comparison tool so that a comparison can be made between accelerated and non-accelerated S3 transfer.‌‌‌‌

Multi-part Uploads

By default, when you upload an object to S3, it is uploaded as a single blob of data in a single stream. S3 uses the PUT operation for uploads, which limits the speed and reliability of the upload because of the single stream of data constraint.

Instead of attempting to upload one colossal file, multi-part uploads divide it into smaller, manageable pieces. The minimum data size for an upload is 100MB, and you shouldn't use a multi-part upload if your data is smaller than this.

An upload can be split into a maximum of 10,000 parts, and each part can range between 5MB and 5GB. Each of the individual parts is isolated from the rest, which means the failure of one part doesn't affect the other parts.‌‌

Source

‌‌When to opt for multi-part uploads

• Large File Sizes: Multi-part uploads should become your go-to strategy for dealing with several gigabytes or more files. ‌‌
• Unstable Internet Connections: In areas where the internet connection may not be consistent, multi-part uploads act as a safety net. It's like having a backup plan for your data transfer.‌‌

Benefits of multi-part uploads

• Faster Uploads: Multi-part uploads speed up the process. It's like assembling a team to work on different project sections simultaneously, ensuring a quicker completion.
• Reliability in Unstable Connections: In scenarios where your internet connection might have a hiccup, multi-part uploads ensure the upload doesn't fail. It's similar to saving your progress in a game – even if the power goes out, you don't lose everything.
• Optimised for Large Files: When dealing with extensive files, multi-part uploads are like breaking them into manageable chapters. It accelerates the process and simplifies troubleshooting if an issue arises.‌‌

To implement multi-part uploads, you can find the detailed steps via AWS Documentation ‌‌‌‌

Measuring Performance and Monitoring

To ensure your Amazon S3 storage operates at its best, keeping an eye on its performance is like giving it a regular check-up. Here's a straightforward guide on how to measure performance and monitor your S3 environment efficiently:

• Network Throughput, CPU, and DRAM: Imagine S3 as a bustling marketplace. To optimise, check the "foot traffic" (network throughput), the "workers' efficiency" (CPU), and the "storage capacity" (DRAM). If any of these are congested, it might be time to consider different "worker" types – or, in AWS terms, Amazon EC2 instance types.‌‌
• Use HTTP analysis tools to ensure your data is moving swiftly and efficiently.‌‌
• Monitor the number of error 503 (Slow Down) Status Error Responses by using Amazon Cloudwatch, Amazon S3 Storage Lens, and Amazon S3 Server Access Logging.‌‌
• Use CloudWatch for Insights on your S3 Performance: CloudWatch uses metrics and dimensions to visualise S3's operational health. Metrics are the data points, such as the number of requests made to an S3 bucket, and dimensions represent a key-value pair of the identity of the metric, helping you filter and focus on specific aspects, like testing environments. CloudWatch presents your data visually, like a set of easy-to-read charts. It's like a health report for your S3. Alarms act as your alert system, notifying you if something needs immediate attention.‌‌

Use Byte-Range Fetches

Optimising the retrieval of objects from Amazon S3 involves a technique known as Byte-Range Fetches. Utilising the Range HTTP header within a GET Object request, you can selectively fetch specific byte ranges from an object, transmitting only the designated portion.‌‌

This approach is efficient when dealing with large objects. Fetching smaller ranges boosts aggregate throughput and facilitates more efficient retry times in case of interrupted requests.

For seamless integration with multi-part uploads, it is recommended to align the byte-range sizes with the sizes of the parts used during the upload process. This alignment ensures optimal performance, and GET requests can directly target individual parts, such as using the syntax GET ?partNumber=N. ‌‌‌‌

CloudFront

CloudFront is a global content delivery network (CDN) that provides viewers with swift content delivery marked by low latency and high transfer speeds.

Leveraging a network of strategically positioned edge locations, CloudFront intelligently caches copies of your S3 content, ensuring proximity to the viewer. This proximity reduces the distance travelled by cached content requests and responses compared to direct routes to your S3 region, resulting in significantly improved performance.

For content not cached, CloudFront seamlessly retrieves it from S3, caching it as necessary. ‌‌

CloudFront offers security enhancements by restricting direct access to S3 and allowing content access exclusively through CloudFront. This restriction is implemented using an origin access identity (OAI).

Additionally, HTTPS encryption can be enforced for situations requiring encryption-in-transit between CloudFront and the viewer, adding an extra layer of security to the content delivery process.‌‌

To achieve high performance, CloudFront incorporates various optimisations, including TLS session resumption, TCP fast open, OCSP stapling, S2N, and request collapsing. It supports a range of HTTP protocols, including HTTP/1.0, HTTP/1.1, HTTP/2, and HTTP/3.‌‌‌‌

S3 Select

S3 Select offers a streamlined approach to fetching specific data from object contents using SQL expressions. It enhances precision in data retrieval, contributes to cost savings, and improves overall performance by minimising the amount of data transferred.‌‌

S3 Select is versatile, operating seamlessly on objects stored in CSV, JSON, and Apache Parquet formats. This flexibility accommodates various data structures and types. It extends its capabilities to objects compressed with GZip and BZip2, ensuring that even compressed CSV and JSON files can be efficiently processed.‌‌

A diagram that illustrates S3 select method (from here).

‌‌By employing S3 Select, unnecessary data transfer is curtailed. This reduction in transferred data directly translates to cost savings, as only the required information is transmitted.

The streamlined data retrieval process enhances overall performance by ensuring that the SQL expressions used in S3 Select efficiently fetch the necessary data, contributing to a more responsive and agile system.‌‌‌‌

Conclusion

Optimising Amazon S3 performance is essential for faster storage and retrieval of data in the cloud.

Identifying and addressing common bottlenecks, such as network latency and server-side processing delays, forms the foundation of a high-performing S3 environment.

Then, implementing best practices like multi-part uploads, Amazon S3 Transfer Acceleration, and CloudFront integration streamlines operations and enhances efficiency.

The power of S3 Select provides precision and agility in data retrieval and monitoring tools like CloudWatch to respond to 503 error signals and keep S3 health in check.

As the cloud environment changes, adopting these practices ensures not just optimal S3 performance but a resilient and future-ready cloud storage infrastructure.

To support file uploads in your application, you will have to learn how to send files from the frontend and receive files on the backend.

This tutorial is going to take a step back and explore architectural changes that'll help you reduce costs when adding file uploads to your applications.

Here's what we'll cover:

1. What is Object Storage?
2. What is S3?
3. Start with an Existing Node.js Application
4. Set Up the S3 Client
5. How to Modify formidable
6. Walkthrough of the Whole Flow
7. Caveats
8. Closing Thoughts

And here's a video walkthrough you can use to supplement this tutorial if you like:

Before we get too far, you should already be familiar with sending and receiving a multipart/form-data request, parsing the request, accessing the file stream, and writing that file to the disk on the application server.

Note that the flow described above writes files to the application server. This is pretty common, but there are some issues with this approach.

First, this approach doesn’t work for distributed systems that may rely on several different machines. If a user uploads a file it can be hard (or impossible) to know which machine received the request, and therefore, where the file is saved. This is especially true if you’re using serverless or edge compute.

Secondly, storing uploads on the application server can lead to the server to run out of disk space. At which point, we’d have to upgrade our server. That could be much more expensive than other cost-effective solutions.

And that’s where Object Storage comes in.

What is Object Storage?

You can think of Object Storage like a folder on a computer. You can put any files (aka “objects”) you want in it, but the folders (aka “buckets”) live within a cloud service provider. You can also access files via URL.

Object Storage provides a couple of benefits:

• It’s a single, central place to store and access all of your uploads.
• It’s designed to be highly available, easily scalable, and super cost-effective.

For example, if you consider shared CPU servers, you could run an application for $5/month and get 25 GB of disk space. If your server starts running out of space, you could upgrade your server to get an additional 25 GB, but that’s going to cost you $7/month more.

Alternatively, you could put that money towards Object Storage and you would get 250 GB for $5/month. So 10 times more storage space for less cost.

Of course, there are other reasons to upgrade your application server. You may need more RAM or CPU, but if we’re talking purely about disk space, Object Storage is a much cheaper solution.

With that in mind, the rest of this article will cover connecting an existing Node.js application to an Object Storage provider. We’ll use formidable to parse multipart requests, but configure it to upload files to Object Storage instead of writing to disk.

If you want to follow along, you will need to have an Object Storage bucket set up, as well as the access keys. Any S3-compatible Object Storage provider should work.

Today, I’ll be using Akamai’s cloud computing services (formerly Linode). If you want to do the same, here’s a guide that shows you how to get going.

And here’s a link to get $100 in free credits for 60 days.

What is S3?

We'll get hands on with code shortly, but before we do, there’s one more concept that I should explain: S3. S3 stands for “Simple Storage Service”, and it’s an Object Storage product originally developed at AWS.

Along with their product, AWS came up with a standard communication protocol for interacting with their Object Storage solution.

As more companies started offering Object Storage services, they decided to also adopt the same S3 communication protocol for their Object Storage service, and S3 became a standard.

As a result, we have more options to choose from for Object Storage providers and fewer options to dig through for tooling. We can use the same libraries (maintained by AWS) with other providers. That’s great news because it means the code we write today should work across any S3-compatible service.

Today, we'll be working with a Node.js application and the libraries we’ll need are @aws-sdk/client-s3 and @aws-sdk/lib-storage:

npm install @aws-sdk/client-s3 @aws-sdk/lib-storage

These libraries will help us upload objects into our buckets.

Okay, let’s write some code!

Start with an Existing Node.js Application

We’ll start with an example Nuxt.js event handler that writes files to disk using formidable. It checks if a request contains multipart/form-data and if so, it passes the underlying Node.js request object (aka IncomingMessage) to a custom function parseMultipartNodeRequest. Since this function uses the Node.js request, it will work in any Node.js environment and tools like formidable.

import formidable from 'formidable';

/* global defineEventHandler, getRequestHeaders, readBody */

/**
 * @see https://nuxt.com/docs/guide/concepts/server-engine
 * @see https://github.com/unjs/h3
 */
export default defineEventHandler(async (event) => {
  let body;
  const headers = getRequestHeaders(event);

  if (headers['content-type']?.includes('multipart/form-data')) {
    body = await parseMultipartNodeRequest(event.node.req);
  } else {
    body = await readBody(event);
  }
  console.log(body);

  return { ok: true };
});

/**
 * @param {import('http').IncomingMessage} req
 */
function parseMultipartNodeRequest(req) {
  return new Promise((resolve, reject) => {
    const form = formidable({ multiples: true });
    form.parse(req, (error, fields, files) => {
      if (error) {
        reject(error);
        return;
      }
      resolve({ ...fields, ...files });
    });
  });
}

We’re going to modify this code to send the files to an S3 bucket instead of writing them to disk.

Set Up the S3 Client

The first thing we need to do is set up an S3 Client to make the upload requests for us, so we don’t have to write them manually. We’ll import the S3Client constructor from @aws-sdk/client-s3 as well as the Upload command from @aws-sdk/lib-storage. We’ll also import Node’s stream module to use later on.

import stream from 'node:stream';
import { S3Client } from '@aws-sdk/client-s3';
import { Upload } from '@aws-sdk/lib-storage';

Next, we need to configure our client using our S3 bucket endpoint, access key, secret access key, and region. Again, you should already have set up an S3 bucket and know where to find this information. If not, check out this guide ($100 credit).

I like to store this information in environment variables and not hard-code the configuration into the source code. We can access those variables using process.env to use in our application.

const { S3_URL, S3_ACCESS_KEY, S3_SECRET_KEY, S3_REGION } = process.env;

If you’ve never used environment variables, they’re a good place for us to put secret information such as access credentials. You can read more about them here.

With our variables set up, I can now instantiate the S3 Client we’ll use to communicate to our bucket.

const s3Client = new S3Client({
  endpoint: `https://${S3_URL}`,
  credentials: {
    accessKeyId: S3_ACCESS_KEY,
    secretAccessKey: S3_SECRET_KEY,
  },
  region: S3_REGION,
});

It’s worth pointing out that the endpoint needs to include the HTTPS protocol. In Akamai’s Object Storage dashboard, when you copy the bucket URL, but it doesn’t include the protocol (bucket-name.bucket-region.linodeobjects.com). So I just add the prefix here.

With our S3 client configured, we can start using it.

How to Modify formidable

In our application, we’re passing any multipart Node request into our custom function, parseMultipartNodeRequest. This function returns a Promise and passes the request to formidable, which parses the request, writes files to the disk, and resolves the promise with the form fields data and files data.

function parseMultipartNodeRequest(req) {
  return new Promise((resolve, reject) => {
    const form = formidable({ multiples: true });
    form.parse(req, (error, fields, files) => {
      if (error) {
        reject(error);
        return;
      }
      resolve({ ...fields, ...files });
    });
  });
}

This is the part that needs to change. Instead of processing the request and writing files to disk, we want to pipe file streams to an S3 upload request. So as each file chunk is received, it’s passed through our handler to the S3 upload.

We’ll still return a promise and use formidable to parse the form, but we have to change formidable’s configuration options. We’ll set the fileWriteStreamHandler option to a function called fileWriteStreamHandler that we’ll write shortly.

/** @param {import('formidable').File} file */
function fileWriteStreamHandler(file) {
  // TODO
}
const form = formidable({
  multiples: true,
  fileWriteStreamHandler: fileWriteStreamHandler,
});

Here’s what their documentation says about fileWriteStreamHandler:

options.fileWriteStreamHandler {function} – default null, which by default writes to host machine file system every file parsed; The function should return an instance of a Writable stream that will receive the uploaded file data. With this option, you can have any custom behavior regarding where the uploaded file data will be streamed for. If you are looking to write the file uploaded in other types of cloud storages (AWS S3, Azure blob storage, Google cloud storage) or private file storage, this is the option you’re looking for. When this option is defined the default behavior of writing the file in the host machine file system is lost.

As formidable parses each chunk of data from the request, it will pipe that chunk into the Writable stream that’s returned from this function. So our fileWriteStreamHandler function is where the magic happens.

Before we write the code, let’s understand some things:

1. This function must return a Writable stream to write each upload chunk to.
2. It also needs to pipe each chunk of data to an S3 Object Storage.
3. We can use the Upload command from @aws-sdk/lib-storage to create the request.
4. The request body can be a stream, but it must be a Readable stream, not a Writable stream.
5. A Passthrough stream can be used as both a Readable and Writable stream.
6. Each request formidable will parse may contain multiple files, so we may need to track multiple S3 upload requests.
7. fileWriteStreamHandler receives one parameter of type formidable.File interface with properties like originalFilename, size, mimetype, and more.

OK, now let’s write the code. We’ll start with an Array to store and track all the S3 upload requests outside the scope of fileWriteStreamHandler.

Inside fileWriteStreamHandler, we’ll create the Passthrough stream that will serve as both the Readable body of the S3 upload and the Writable return value of this function.

We’ll create the Upload request using the S3 libraries, and tell it our bucket name, the object key (which can include folders), the object Content-Type, the Access Control Level for this object, and the Passthrough stream as the request body.

We’ll instantiate the request using Upload.done() and add the returned Promise to our tracking Array. We might want to add the response Location property to the file object when the upload completes, so we can use that information later on.

Lastly, we’ll return the Passthrough stream from this function:

/** @type {Promise<any>[]} */
const s3Uploads = [];

/** @param {import('formidable').File} file */
function fileWriteStreamHandler(file) {
  const body = new stream.PassThrough();
  const upload = new Upload({
    client: s3Client,
    params: {
      Bucket: 'austins-bucket',
      Key: `files/${file.originalFilename}`,
      ContentType: file.mimetype,
      ACL: 'public-read',
      Body: body,
    },
  });
  const uploadRequest = upload.done().then((response) => {
    file.location = response.Location;
  });
  s3Uploads.push(uploadRequest);
  return body;
}

A couple of things to note:

• Key is the name and location where the object will exist. It can include folders that will be created if they do not currently exist. If a file exists with the same name and location, it will be overwritten (fine for me today). You can avoid collisions by using hashed names or timestamps.
• ContentType is not required, but it’s helpful to include. It allows browsers to create the downloaded response appropriately based on Content-Type.
• ACL: is also optional, but by default, every object is private. If you want people to be able to access the files via URL (like an <img> element), you’ll want to make it public.
• Although @aws-sdk/client-s3 supports uploads, you need @aws-sdk/lib-storage to support Readable streams.
• You can read more about the parameters on NPM.

This way, formidable becomes the plumbing that connects the incoming client request to the S3 upload request.

Now there’s just one more change to make. We are keeping track of all the upload requests, but we aren’t waiting for them to finish.

We can fix that by modifying the parseMultipartNodeRequest function. It should continue to use formidable to parse the client request, but instead of resolving the promise immediately, we can use Promise.all to wait until all the upload requests have resolved.

The whole function looks like this:

/**
 * @param {import('http').IncomingMessage} req
 */
function parseMultipartNodeRequest(req) {
  return new Promise((resolve, reject) => {
    /** @type {Promise<any>[]} */
    const s3Uploads = [];

    /** @param {import('formidable').File} file */
    function fileWriteStreamHandler(file) {
      const body = new PassThrough();
      const upload = new Upload({
        client: s3Client,
        params: {
          Bucket: 'austins-bucket',
          Key: `files/${file.originalFilename}`,
          ContentType: file.mimetype,
          ACL: 'public-read',
          Body: body,
        },
      });
      const uploadRequest = upload.done().then((response) => {
        file.location = response.Location;
      });
      s3Uploads.push(uploadRequest);
      return body;
    }
    const form = formidable({
      multiples: true,
      fileWriteStreamHandler: fileWriteStreamHandler,
    });
    form.parse(req, (error, fields, files) => {
      if (error) {
        reject(error);
        return;
      }
      Promise.all(s3Uploads)
        .then(() => {
          resolve({ ...fields, ...files });
        })
        .catch(reject);
    });
  });
}

The resolved files value will also contain the location property we included, pointing to the Object Storage URL.

Walkthrough of the Whole Flow

We covered a lot, and I think it’s a good idea to review how everything works together. If we look back at the original event handler, we can see that any multipart/form-data request will be received and passed to our parseMultipartNodeRequest function. The resolved value from this function will be logged to the console:

export default defineEventHandler(async (event) => {
  let body;
  const headers = getRequestHeaders(event);

  if (headers['content-type']?.includes('multipart/form-data')) {
    body = await parseMultipartNodeRequest(event.node.req);
  } else {
    body = await readBody(event);
  }
  console.log(body);

  return { ok: true };
});

With that in mind, let’s break down what happens if I want to upload a cute photo of Nugget making a big ol’ yawn.

1. For the browser to send the file as binary data, it needs to make a multiplart/form-data request with an HTML form or with JavaScript.
2. Our Nuxt.js application receives the multipart/form-data and passes the underlying Node.js request object to our custom parseMultipartNodeRequest function.
3. parseMultipartNodeRequest returns a Promise that will eventually be resolved with the data. Inside that Promise, we instantiate the formidable library and pass the request object to formidable for parsing.
4. As formidable is parsing the request when it comes across a file, it writes the chunks of data from the file stream to the Passthrough stream that’s returned from the fileWriteStreamHandler function.
5. Inside the fileWriteStreamHandler we also set up a request to upload the file to our S3-compatible bucket, and we use the same Passthrough stream as the body of the request. So as formidable writes chunks of file data to the Passthrough stream, they are also read by the S3 upload request.
6. Once formidable has finished parsing the request, all the chunks of data from the file streams are taken care of, and we wait for the list of S3 requests to finish uploading.
7. After all that is done, we resolve the Promise from parseMultipartNodeRequest with the modified data from formidable. The body variable is assigned to the resolved value.
8. The data representing the fields and files (not the files themselves) are logged to the console.

So now, if our original upload request contained a single field called “file1” with the photo of Nugget, we might see something like this:

{
  file1: {
    _events: [Object: null prototype] { error: [Function (anonymous)] },
    _eventsCount: 1,
    _maxListeners: undefined,
    lastModifiedDate: null,
    filepath: '/tmp/93374f13c6cab7a01f7cb5100',
    newFilename: '93374f13c6cab7a01f7cb5100',
    originalFilename: 'nugget.jpg',
    mimetype: 'image/jpeg',
    hashAlgorithm: false,
    createFileWriteStream: [Function: fileWriteStreamHandler],
    size: 82298,
    _writeStream: PassThrough {
      _readableState: [ReadableState],
      _events: [Object: null prototype],
      _eventsCount: 6,
      _maxListeners: undefined,
      _writableState: [WritableState],
      allowHalfOpen: true,
      [Symbol(kCapture)]: false,
      [Symbol(kCallback)]: null
    },
    hash: null,
    location: 'https://austins-bucket.us-southeast-1.linodeobjects.com/files/nugget.jpg',
    [Symbol(kCapture)]: false
  }
}

It looks very similar to the object formidable returns when it writes directly to disk. But this time it has an extra property, location, which is the Object Storage URL for our uploaded file.

Throw that sucker in your browser and what do you get?

That’s right! A cute photo of Nugget making a big ol’ yawn 🥰

I can also go to my bucket in my Object Storage dashboard and see that I now have a folder called “files” containing a file called “nugget.jpg”.

Caveats

I would be remiss if I didn’t mention the following. (In fact, I was remiss because I didn’t mention it until after someone pointed it out to me 🤣)

Streaming uploads through your backend to Object Storage is not the only way to upload files to S3. You can also use signed URLs.

Signed URLs are basically the same URL in the bucket where the file will live, but they include an authentication signature that can be used by anyone to upload a file, as long as the signature has not expired (usually quite soon).

Here’s how the flow generally works:

1. Frontend makes a request to the backend for a signed URL.
2. Backend makes an authenticated request to the Object Storage provider for a signed URL with a given expiry.
3. Object Storage provider provides a signed URL to the backend.
4. Backend returns the signed URL to the frontend.
5. Frontend uploads the file directly to Object Storage thanks to the signed URL.
6. Optional: Frontend may make another request to the Backend if you need to update a database that the upload completed.

This flow requires a little more choreography than Frontend -> Backend -> Object Storage, but it has some benefits.

• It moves work off your servers, which can reduce load and improve performance.
• It moves the file upload bandwidth off your server. If you pay for ingress and have several large file uploads all the time, this could add up.

It also comes with its own costs.

• You have much less control over what users can upload. This might include malware.
• If you need to perform functions on the files like optimizing, you can’t do that with signed URLs.
• The complex flow makes it much harder to build an upload flow with progressive enhancement in mind.

As with most things in web development, there is not one right solution. It will largely depend on your use case. I like going through my backend, so I have more control over the files and I can simplify the frontend.

I wanted to share this streaming option, largely because there is hardly any content out there about streaming. Most content uses signed URLs (maybe I’m missing something). If you’d like to learn more about using signed URLs, here is some documentation and here’s a handy tutorial by Mary Gathoni.

Closing Thoughts

Okay, we covered a lot today. I hope it all made sense. If not, feel free to reach out to me with questions. Also, reach out and let me know if you got it working in your own application.

I’d love to hear from you, because using Object Storage is an excellent architectural decision if you need a single, cost-effective place to store files.

Thank you so much for reading. If you liked this article, and want to support me, the best ways to do so are to share it, sign up for my newsletter, and follow me on Twitter.

If you have ever worked as a developer, you have likely come across file storage use cases. From simple images to large videos, uploading, storing, and accessing those files when you need them is always tricky.

The usual answer to file storage is to keep them on the same server where you host your web applications. But with the advent of serverless architectures and single-page applications, storing files on the same server is not a good idea.

You could argue that you can store files in databases. But trust me, it won’t be a pleasant experience.

So what's another option?

What is S3?

Let's look at AWS S3. S3 is an easy-to-use, scalable, and cheap storage service from Amazon. You can use S3 to store any amount of data for a wide range of use cases.

Static website hosting, data archival, and software delivery are a few general scenarios where S3 would be a perfect tool.

You can easily push and pull data with S3 using the AWS SDK. S3 also supports a number of popular programming languages, so you can use your existing stack and integrate S3 pretty easily.

AWS Console

S3 also offers a great user interface via the AWS console. You can use it to view the data pushed to S3 along with additional options such as security and version control.

Buckets

In S3, files are stored in buckets. Buckets are similar to folders on your computer.

Every bucket has its own unique name which can be used only once. For example, if there is a bucket called “freecodecamp”, neither you nor anyone else can re-use the same bucket name.

This is useful to uniquely identify resources and for static website hosting with domain names.

There are no limits on the number of files you can store in a bucket. Buckets also provide additional features such as version control and policies.

You can also use different buckets for a single application. For example, an app that stores medical records can use two buckets: one for private customer data and another public bucket that contains whitepapers.

S3 is also an object-based storage service which means S3 considers each file an object. Every object can have its own metadata that includes the name, size, date, and other information.

S3 Storage Types

S3 has three storage classes based on general use cases.

S3 Standard

S3 Standard is the default storage plan you will be put into when you start using S3. The standard storage class has excellent performance, durability, and availability.

S3 Standard is best if you have data that you have to access frequently.

S3 Infrequent Access (S3-IA)

S3 Infrequent Access offers a lower price for data compared to the standard plan. You can use S3-IA for data that you need less often.

S3-IA is great for use cases such as backups and disaster recovery.

Glacier

Glacier is the least expensive storage option in S3 but is designed for archival storage. You cannot fetch data from Glacier as fast as Standard or S3-IA, but it is a great option for long term data archival.

In addition to choosing one of these three storage classes, you can also set lifecycle policies in S3. This means that you can schedule files to be moved automatically to S3-IA or Glacier after a certain period of time.

Why Use S3?

Companies like Netflix, Dropbox, and Reddit are avid users of S3. The popular file storage system Dropbox built its entire storage capacity on top of Amazon S3.

Let’s look at some of the core features of S3 and understand why it's so popular among enterprises and startups alike.

It's Affordable

S3 is cheap. I mean super cheap compared to other storage solutions. And with S3, you only pay for what you use. There are no upfront costs, no setup. It's just plug and play.

In addition to affordable pricing, S3 offers a Free tier. This free tier comes with 5GB of storage space, 20,000 GET Requests, 2,000 PUT, COPY, POST, or LIST Requests and 15GB of Data Transfer. The free tier is available every month for the first year.

With S3 you can avoid paying for space or bandwidth you might not even need.

It's Scalable

S3 scales with your application. Since you pay only for that you use, there is no limit to the data you can store in S3.

This is helpful during multiple scenarios, especially during an unexpected surge in user growth. You don’t have to buy extra space. S3 has you covered.

It's Secure

One of the many reasons companies prefer S3 is its inclination towards security. While you have to secure custom server setups, S3 is secure by default.

This does not mean you cannot store publicly accessible information in S3. S3 locks up all your data with high security unless you explicitly configure not to.

S3 also maintains compliance programs, such as PCI-DSS, HIPAA/HITECH, FedRAMP, EU Data Protection Directive, and FISMA, to help you meet your industry’s regulatory requirements.

It Has Versioning

Versioning means keeping multiple copies of a file and tracking its changes over time. This is useful, especially when you handle sensitive data.

You can also retrieve accidentally deleted files when you enable versioning with S3.

However, if you enable versioning, you are storing multiple copies of the same document. This can have an effect on pricing as well as read/write requests you make.

So just take that into account while integrating versioning for your application.

Versioning is disabled by default for S3 but you can enable versioning using the AWS Console.

It's Durable

Data durability is an underrated feature of S3. Given how common data loss is among companies, data durability is a core factor to consider when building enterprise software.

S3 provides a highly durable storage infrastructure. S3 redundantly stores data in multiple facilities, making you data safe in the event of a system failure. S3 also performs regular data integrity checks to make sure your data is intact.

S3 offers 99.999999999% durability (called the 9s durability) and 99.99% availability of objects over a given year.

S3 Use Cases

Static Website Hosting

You can use S3 as a static website hosting platform. The difference between static and dynamic websites is that dynamic websites receive and process user input. Static websites are used only for displaying information.

With the advent of Single Page Applications, you can host a complete web app on S3, often free of charge.

Frameworks like React and Angular have made user input processing happen within the browser. You can build a SPA that listens to third party APIs and host it within S3.

S3 also has great support for routing, so you can use your own custom domain as well.

I recently wrote an article on hosting a React web app using S3 and you can find the article here.

Analytics

You can run queries on your S3 data without moving your data to an analytics platform. This makes S3 a great use case for building powerful analytics applications.

S3 offers multiple options including S3 Select, Amazon Athena, and Amazon Redshift Spectrum. You can also combine these with AWS Lambda to perform data processing on the fly.

File Sharing

Amazon S3 can be also used as a cheap file sharing solution. Like I mentioned earlier in the article, the famous file sharing service Dropbox was first built on top of S3.

With flexible security policies, you can configure your S3 buckets with custom permissions for different customers. S3 also offers transfer acceleration to speed up large file transfers across longer distances.

Summary

Amazon S3 is a great tool to work with for your web or mobile application storage requirements. With on-demand pricing and scalability at its core, S3 has been the favored cloud storage solution for small and large businesses alike.

Companies from Netflix to Pinterest trust S3 with their data, thanks to the 99.999999999% data durability promise from Amazon.

You can also use Amazon S3 as a personal storage solution or host your next project via static site hosting. In a nutshell, S3 is a great multi-purpose storage solution catering to a wide range of use cases.

I regularly write about Machine Learning, Cyber Security, and AWS. You can signup for my weekly newsletter here.

AWS is THE cool kid in the town. Every comparison of different cloud providers is incomplete unless you compare them with AWS at least once.

But S3, the most popular solution for storing on the cloud and the one everyone loves, should not always be your choice. In this article, I'll explain why.

Note: Please don't immediately yell at me about how and why AWS is best. I know they are at the top of cloud computing - and in no way am I trying to target any of their business practices and services.

I've just used CloudFront + S3 myself, along with DigitalOcean + Cloudflare too, and have laid down my observations. Please take my thoughts constructively, and if you think I've made any mistakes, tweet me at mehulmpt.

CloudFront + S3

CloudFront is another service often used (and recommended) with S3 when you're trying to distribute files digitally all over the globe. CloudFront is a CDN from Amazon with edge servers all over the world. This is how it works:

Your user, say from India, tries to load your website whose server is located in the USA. Let's say you're using a SPA like React or Angular. The first index.html page will load from your origin server (it is usually a good practice to never cache HTML pages, especially if you're using SSR applications to prevent cache mishaps).

After that, if you've hosted your JS/CSS files on CloudFront (S3), those calls will be made to a domain name from CloudFront which resolves to an IP address of a machine closest to your location. In this case, it's probably some server from AWS sitting in some data center in Mumbai, India.

From this point, that server has the responsibility of delivering that file. Two things can happen:

• your file is already available with that Mumbai server (cached), and that server returns you that file immediately (cache hit),
• or it does not has that file and has to perform a trip to your origin server (S3 bucket in this case) to get that file.

But even if there's a cache miss, chances are high that it will be still faster for a user compared to not having CloudFront in front.

Why? Because when there is a cache miss and the edge server is trying to reach the main server, it is using a Tier 1 internet connection line operated by Amazon - a trillion-dollar US company. They likely have much better internet connectivity and latency than what your ISP can offer.

Also, because they're on the same global Amazon network, they can do some neat optimizations to save more time.

Alright! Sounds great to me so far, so what's the problem? Hold your horses, we'll get to it.

Asset compression

CloudFront allows you to deliver compressed assets using GZIP. But there's even a cooler kid in the market: brotli compression. And it is supported by almost every major browser.

Brotli compresses your transmission data even more. This means it's not only good on your wallet, but it's also good for the end-user (because they'll spend less time seeing that loading/white screen).

Amazon CloudFront does not support brotli compression delivery, yet. And I won't blame them for this either. This is because brotli compression is slow to do on the fly (CloudFront does gzip on the fly), so they have not implemented it yet.

Sure, then let's do it ourselves and store it on S3 and deliver the compressed version, right? Unfortunately, it is not as simple, and we'll soon spin down into more of an architecture problem.

A typical asset URL would look like this: http://mysite/assets/javascript/file.js

When your browser makes a request, it sends a header: Accept-Encoding. This header can contain compression algorithms your browser can support, like gzip, deflate, brotli, etc. The server now has to act smart to have maximum efficiency.

1. If the client supports brotli, then always deliver the brotli compressed asset.
2. If the client supports gzip, then always deliver gzip.
3. Otherwise, deliver the original file.
4. Also, make sure that in the response type, the correct Content-Encoding is set so that browser can recognize the compression algorithm.

Now, firstly, you have to create 3 variants of every single asset file:

1. file.js
2. file.js.br - brotli
3. file.js.gz - gzip

And you have to conditionally deliver them depending on if the browser supports it or not. CloudFront is a "dumb" CDN - it will just map your request URL to the file on your server. It cannot perform any transformations unless.... you opt-in for another AWS service - Lambda@edge functions

We all likely know what Lambda is on AWS - you can run functions on the cloud without worrying about underlying infrastructure upscaling or downscaling. Per API request pricing, time-bounded, sweet. Lambda@edge is a similar service but was made for edge servers (CloudFront CDN datacenters)

You can technically configure a Lambda server to act as a "middle man" between the request made by your client and the CloudFront CDN. Lambda can open the request, see the supported content headers, modify the URL accordingly and forward it to the "dumb" CloudFront which is going to retrieve the modified URL file then.

For example, if Lambda sees that browser sent an Accept-Encoding: br then lambda can be used to modify the request URL from /javascript/file.js to /javascript/file.js.br without actually telling the user side. Cloudfront will now retrieve a smaller payload and return a response for a brotli encoding. WIN!

But that is good, isn't it? WHERE is the problem? The problem is... pricing.

AWS is ridiculously expensive (for this task)

Whatever you've done so far sounds and looks very good. But when you look at what's happening when you start to hit significant numbers, you'll realize that AWS isn't great when it comes to data transfer. Zoom just bounced AWS for the same reason.

Plus, with the asset compression, now you also have to pay for Lambda@edge calls. I figured out that implementing Lambda@edge will actually reduce your costs, otherwise you'll pay much more for AWS for traffic!

CloudFront works on data transfer pricing. It does not charge you when it retrieves data from the S3 bucket, it charges you when a user retrieves data from the edge servers.

Upper cost bound

In the most expensive country - India - CloudFront charges you $0.170 per GB of data transferred. This is huge!

Let's say you have a popular (mainly) Indian website with about 50,000 users visiting your site daily. Also, let's say you make some design changes every week on your site (pretty common for fast iterating products) so you have to invalidate the browser and CloudFront cache.

Also, let's assume on average, a single user downloads about 10MB of the static asset from your site (includes CSS/JS/images/fonts) hosted on S3 proxied through CloudFront.

Let's calculate the cost:

1. 50K Indian users
2. 0.17 USD per GB
3. 10MB per user
4. Every user retrieves this 4 times a month (you flush your cache 4 times - once every week)

Cost = 50000 0.17 (10/1024) 4 = 332 USD. That is your COST of just data transfer! I did not calculate the S3 storage cost and the hosting site cost. (I also did not include lambda pricing because it's not much => $(0.20 (50,000 * 4))/1 million = 4 cents.)

Lower cost bound

In this case, let's assume a US based traffic site. The parameters now would be:

1. 50K USA users
2. 0.085 USD per GB
3. 3 MB per user
4. Every user retrieves this 4 times a month (you flush your cache 4 times - once every week)

The cost = 500000.0853*4/1024 = 50 USD. That is the lowest you'll pay when using CloudFront with the mentioned traffic (given that all of your 50K users are from the USA only). And remember, that is the cost only for the data transfers! (Not including server costs for hosting your website.)

Alternative

Let's say now you host all these static assets on your main server only - reverse proxied by NGiNX and say, running on a $60 DigitalOcean instance.

Your data transfer per month = 50000 (10/1024) 4 = 1952 GB approximately 2TB - DigitalOcean covers your 1TB of transfer per droplet for free. And it is $10 per 1TB from then, so it'll be $70 net for running the server.

Sure, you'll get some latency now - because you're hosting it yourself (we'll even fix this later). NGiNX is a high performing web server and you can rely on it not to be a bottleneck in your static asset delivery.

So you just dropped the cost of "only asset transfer" from $332 to $70 for running the whole server! Bonus tip? We were focusing on running this only in India, so use a DigitalOcean server from India. This would mean less latency.

Not only this, but you can also opt for Cloudflare CDN too - which is FREE. Cloudflare won't respect your files to keep in the CDN if they're too big or too infrequently accessed. But we're assuming a hell of a popular site here, so we should be fine. If not, opt for any other CDN service, and I guarantee you it will be less than $332 a month.

TL;DR - If you're hosting a website with medium-large amounts of traffic with regularly scheduled updates, it is much more cost efficient to host assets yourself and use external CDNs (or even things like DigitalOcean CDN) instead of using S3 and CloudFront (where data traffic rates are through the roof).

Conclusion

I used this setup (CloudFront + AWS S3) on codedamn.com - a platform for developers to learn and grow. I soon realised that although it looks fancy and I've put codedamn into the big leagues - Amazon - it's just not efficient enough.

Do you agree with me? What do you think? Let me know by tweeting at me on my Twitter or reaching out to me on Instagram.

Peace!

This guide was created because it was so hard to find a way to play around with a sample database using AWS RDS MSSQL Server. I hope you find this helpful.

If you haven't set up your AWS RDS Microsoft SQL Server and Azure Data Studio, check this guide first: How to Connect your AWS RDS Microsoft SQL Server using Azure Data Studio.

We will be touching the technologies shown below:

• Database: AWS RDS Microsoft SQL Server Express Edition
• Database tool and GUI: Azure Data Studio
• Sample database backup copy: Amazon S3 Bucket

AdventureWorks sample database backup copy

To get the OLTP downloads of AdventureWorks, go to this link and choose any sample database. In my example, I choose AdventureWorks2017.bak. We will upload this to the S3 Bucket.

Amazon S3 Bucket

Creating the S3 Bucket

1. Create a bucket. You can choose any bucket name (example: yourname-sample-dbs).

2. Make sure the region is same as the AWS RDS instance.

3. Tick the following checkboxes:

4. Block public access to buckets and objects granted through new access control lists (ACLs)

5. Block public access and objects granted through any access control lists (ACLs)

4. Access your bucket again by clicking on your created bucket.

Uploading the file to the S3 bucket

1. Click Upload.

2. Choose the database backup file. For example: AdventureWorks2017.bak. Keep choosing Next and choose Upload at the Review section.

3. Update your Bucket Policy to allow access to your S3 Bucket. Note that your ARN will differ to mine. Hit Save afterwards.

{
    "Version": "2012-10-17",
    "Id": "Policy1548223592786",
    "Statement": [
        {
            "Sid": "Stmt1548223591553",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::changethis/*"
        }
    ]
}

AWS RDS - MSSQL Server Express

Creating an Option Group for your RDS instance

1. Click Option groups,

2. Create an option group. Choose any name and description. For the Engine, it should match your RDS instance. In my example, I used SQL Server Express Edition so I choose sqlserver-ex.

Here are the following Engines and their abbreviations:

• SQL Server Enterprise Edition: sqlserver-ee
• SQL Server Standard Edition: sqlserver-se
• SQL Server Web Edition: sqlserver-web
• SQL Server Express Edition: sqlserver-ex

3. Once you have created the option group, you'll need to Add option.

4. Choose SQLSERVER_BACKUP_RESTORE for your Option name. For the IAM role, it is best to create a new role.

5. Choose the S3 bucket where your database file is hosted. For scheduling, choose Immediately.

6. Go back to your AWS RDS MSSQL Server instance and click Modify.

7. Choose the created option group with sql-server-express-backup, then Click Continue.

8. Choose to Apply immediately for scheduling of modifications.

9. Go back to your AWS RDS MSSQL Server instance page and scroll down and modify Manage IAM Roles. Add the IAM role you have created in S3. For the Feature, choose S3_INTEGRATION.

Azure Data Studio

Importing the sample database in S3 bucket through restore function

1. In your connected AWS RDS MSSQL Server, create a new query and type in the following:

exec msdb.dbo.rds_restore_database 
@restore_db_name='AdventureWorks-test', 
@s3_arn_to_restore_from='arn:aws:s3:::clark-sample-dbs/AdventureWorks2017.bak';

Refresh your Azure Data Studio. Also, try restarting the application if your database did not appear or don't have permission to access it.

Now you are done! Good job! ???

Resources:

• https://aws.amazon.com/premiumsupport/knowledge-center/native-backup-rds-sql-server/

Connect with me on LinkedIn here

If you're hosting a static website in an S3 bucket and it's your first time buying a domain name, this simple guide is for you.

Summary - What You Need

Amazon S3

• Have an S3 bucket with the same name as your domain name
• Upload your website's code
• Allow public access
• Add a policy to enable S3 GetObject
• Enable static website hosting

Domain Name provider

• In your domain name's DNS Zone settings, delete all A records
• In DNS Zone settings, add www to subdomain and the S3 endpoint in hostname for CNAME records

Let's go through these steps one by one.

Step 1: Create an S3 bucket

Create an S3 bucket to host your files for your website

First you need to create a bucket for your website. The name for your bucket must be the same as your domain name. Let's say we bought the domain name www.clarkngo.net. Then my S3 bucket's name should be www.clarkngo.net as well.

After configuration, my endpoint should look similar to this:

http://www.clarkngo.net.s3-website-us-west-2.amazonaws.com

Go to your AWS console and login. Choose S3.

1. Click Buckets
2. Click Create bucket

3. Add your domain name in the bucket name

4. You may choose any Region

Creating the S3 bucket and general configuration

Follow the checkboxes below and click Create Bucket.

Only tick the following:

• Block public access to bucket and objects granted through new access control lists (ACLs)
• Block public access to bucket and objects granted through any access control lists (ACLs)

Uploading files to the S3 Bucket

1. Click Overview and Upload.

2. Upload your website files in Select Files

3. For Set permissions, hit Next.

4. For Set properties, hit Next. (The default is Standard S3.)

5. For Review, hit Upload.

Editing the Bucket Policy

1. Click Permissions, then Bucket Policy.

2. Add the policy. (Note: For your website you'll change arn:aws::s3:::www.clarkngo.net/*)

{
    "Version": "2012-10-17",
    "Id": "Policy1548223592786",
    "Statement": [
        {
            "Sid": "Stmt1548223591553",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::www.clarkngo.net/*"
        }
    ]
}

3. Hit Save.

Static website hosting

1. Click Properties, then Static website hosting.

2. Choose Use this bucket to host a website.

3. For Index document, type index.html.

4. For Error document, type index.html.

5. Hit Save.

Step 2: Add the S3 Endpoint to your Domain

Editing your DNS Zone

1. Login to your domain provider.
2. In this example, choose Name Servers/DNS, then Modify DNS Zone (or the equivalent).

3. Remove all A records in your domain. Usually it will have a default IP address for a 404 Not Found page.

4. Add a CNAME to point to the S3 Bucket:

5. add www for the Subdomain.

6. add www.clarkngo.net.s3-website-us-west-2.amazonaws.com (the S3 Endpoint) to the Hostname.

And you're done! Note that it might take a while for your new settings take effect.

Connect with me in LinkedIn here.

• A little about AWS
• What are the benefits of serving from S3 and CloudFront?
• Before we start, you’ll need an AWS account
• Storing your website on S3
• Serving your website on S3
• Distributing your website on CloudFront
• Custom domain names
• Advanced AWS Usage
• Resources

A little about AWS

If you’re not familiar, AWS (Amazon Web Services) is a cloud service provider that gives developers opportunities to build pretty much anything they can imagine in the cloud.

Though their services extend beyond the likes of machine learning and artificial intelligence, we’re going to stick with the entry level services for the purpose of this guide that will allow us to easily host an HTML website.

Types of AWS services available

Building a site with S3 and CloudFront is a common recipe that small and high scale companies across the web use, but let’s break down what each service actually does.

Object storage with S3

S3 (Simple Storage Service) acts as your hosting for your static website. Think of it like a hard drive in the cloud which we’re not able to use it for processing purposes, but rather for simple file storage and access.

List of files from a static site in an AWS S3 bucket

When an app or website is compiled in static form, this is all we need to serve it to the people visiting our site. The HTML is sent in the initial request “as is” (unless there’s processing with your provider) and any additional work occurs after the page loads in the browser usually by JavaScript. This allows us to take this simple (and cheap) approach by serving these files from S3.

Content Delivery Network with CloudFront

CloudFront works as a CDN (Content Delivery Network) that sits in front of your website, caching the files, and serving them directly to the people visiting your site.

CDN Diagram

Where you host and serve your website from, typically called the origin, is the main source of your files and can serve the website itself. But putting a CDN in front of it provides the people accessing your content a shorter and faster way to make their request.

What are the benefits of serving from S3 and CloudFront?

Given the rise in the JAMstack era, many services are popping up that provide similar services for static sites that make it really easy to deploy. Some even come with a generous free tier like Netlify and Zeit!

But sometimes developers need a little bit more control over their services or they need to integrate into a larger cloud pipeline that’s already 99% percent in AWS, which is exactly where S3 shines. Also, chances are, during your first year you might still qualify for AWS’s free tier.

Fitting in to the AWS Well-Architected Framework

As a lead provider in cloud services, AWS has published many guides to help developers and teams strive for excellence in their solutions in terms of performance, cost, and security.

One particular guideline is their 5 pillars of what they describe as a “well-architected" infrastructure.

AWS Well-Architected Framework

By default, we check all of these boxes with our hosting solution by using S3 and CloudFront. Out of the box, the HTML and assets you serve will be fast, cheap, secure, and reliable.

The beauty of static and JAMstack sites

Building on top of the pillars, what you’re actually serving is a static HTML file and group of assets that won’t require any type of rendering resources on the initial request. Before this, a common problem was having to worry about a site crashing due to heavy load. But with S3 and CloudFront, your website is infinitely scalable.

On a similar note, when that server scales up as it's trying to serve millions of hits on your post that went viral, so will your costs. Serving a static site is cheap and can greatly reduce the cost associated with running a web server.

Before we start, you’ll need an AWS account

To work through this guide, you’ll need an AWS account. Luckily, it's free to create an account – you’ll only pay for the services used.

On top of that, AWS provides a generous free tier for some of its services. Some services provide only 12 months of a free tier (like S3) where others are always eligible for the free tier (like Lambda), so make sure to do your homework so you don’t rack up an unexpectedly high bill.

To create your account, head over to the AWS website and then continue on to get started: https://aws.amazon.com/.

Storing your website on S3

To get started, we’re going to begin with a simple HTML file that will serve as our website. This will allow us to focus more on the process of hosting rather than the intricacies of the website itself.

Creating our website file

Begin by creating a new folder called my-static-site. Inside that folder, let's create a new file called index.html and add the following to the file:

<!DOCTYPE html>
<html lang=“en”>
<head>
  <meta charset=“UTF-8”>
  <meta name=“viewport” content=“width=device-width, initial-scale=1.0”>
  <title>My Static Website</title>
</head>
<body>
  <h1>Hello World!</h1>
  <p>This is my static website. ?</p>
</body>
</html>

If you open this file from your computer in your favorite browser, you should now be seeing this.

Hello World! Opening a local webpage

Creating a new bucket

Head on over to your AWS account, log in, and navigate to your S3 console.

Once there, let’s create our bucket by clicking on the blue Create bucket button:

Creating a bucket in AWS S3

The first thing AWS wants us to do is enter a Bucket name. The bucket name must be globally unique, meaning, the name you use can be the only one in the world, so let’s try something like [yourname]-static-website, where I’ll use colbyfayock-static-website.

Naming a bucket in AWS S3

Next, let’s set the Region. This is the geographic location where AWS will host the bucket and your website. You’re probably fine with the default, but if you’d like, you can select the location closest to you if it’s permitted. Since I’m in Virginia, I’m going to stick with my default of US East (N. Virginia).

Setting the region of a bucket in AWS S3

Finally, hit the Create button on the bottom left of the page.

Note: even if you use the [yourname]-static-website pattern, there’s a chance the name will be taken. If it’s taken, AWS will show an error stating “Bucket name already exists,” at which point you’ll want to try a new name of your choosing.

Alternatively, you can hit Next for advanced usage, but for this guide, we’re okay with all of the defaults S3 provides.

If successful, you should now see your bucket in the list on the S3 console dashboard.

New bucket in AWS S3

Uploading your website to the bucket

Let’s navigate to our new bucket by clicking the row of our bucket. You’ll be greeted with a message stating “This bucket is empty. Upload new objects to get started,” so that’s what we’ll do.

Click the Upload button to get started.

Uploading files to AWS S3

You’ll then see a popup that will ask you to upload a file. Click on the Add files button and select your index.html file we created earlier.

Once selected, click the Upload button on the bottom left.

Selecting files to upload in AWS S3

And now your file is uploaded to S3!

Serving your website on S3

If you try to navigate to your index.html file and open it, you’ll notice a big ugly "Access Denied" message.

Access Denied to bucket file

This is because your file doesn’t currently have the permissions and settings necessary to serve the file to the public, so let’s fix that.

Setting up your bucket as a website

Navigate to the Properties tab inside of your bucket, then click Static website hosting.

Setting up an AWS S3 bucket for statice website hosting

Once there, we want to do a few things:

• Note down the Endpoint at the top of the block. We’ll use this to access our site later (you can always find this here again)
• Select the “Use this bucket to host a website” option
• Enter index.html in the Index document field
• Finally hit Save

Configuring an AWS S3 bucket for static website hosting

Setting up your bucket policy and permissions

Next, navigate to the Permissions tab. Here we’ll want to do 2 things: unblock all public access and add a Bucket Policy.

First, on the main page, let’s click Edit to unblock all access.

Configuring an AWS S3 bucket permissions

Then, uncheck the “Block all public access” checkbox and hit Save.

Allowing public access to an AWS S3 bucket

AWS will ask you to confirm these settings, as this may not always be what you want to do with your bucket. But for the purposes of hosting a website, we want the whole world to see, so type in the word “confirm” and hit the Confirm button.

After confirming, click the Bucket policy button and you’ll be taken to a text editor.

In this text box, we’ll want to paste the following snippet. Within this snippet, make sure to replace [your-bucket-name] with the name of your bucket, otherwise you will not be able to save this file.

{
  "Version":"2012-10-17",
  "Statement":[{
    "Sid":"PublicReadGetObject",
        "Effect":"Allow",
      "Principal": "*",
      "Action":["s3:GetObject"],
      "Resource":["arn:aws:s3:::[your-bucket-name]/*”
      ]
    }
  ]
}

This policy states that it’s allowing the public to perform a GetObject request on the S3 resource, which is your S3 bucket.

Setting up a public policy for an AWS S3 bucket

After you add the policy, click the Save button. Your should now see a message stating "This bucket has public access.”

Previewing your new bucket website

If you noted down the Endpoint from your Properties page, you can now visit that address to see your website. The endpoint should look like this:

http://[your-bucket-name].s3-website-[region-id].amazonaws.com

If you didn’t, jump back up a few steps to remind yourself how to find it or look under the Properties tab.

Hello World! Opening an AWS S3 website

Congrats, you're halfway there! ?

Distributing your website on CloudFront

Now that we have our static website being served from a bucket on S3, let’s take it up another level and serve it across the world using CloudFront.

Creating a CloudFront distribution

Navigate to your CloudFront dashboard and click the Create Distribution button.

Creating a new distribution in AWS CloudFront

Next, select Get Started under the Web delivery method.

Getting started with an AWS CloudFront distribution with Web delivery

Here, we’ll enter a few custom parameters to get our distribution set up.

Click into the Origin Domain Name field. Once selected, a dropdown list should appear where you can select the S3 bucket you just created. Go ahead and select your S3 bucket.

Setting the origin domain name in AWS CloudFront to your bucket

While you can customize most of the settings to your liking, for our purposes, we’re going to leave all as their default values except for one.

Scroll down to the Default Root Object field and type index.html.

Setting the Default Root Object for a distribution in AWS CloudFront

After, scroll down to the bottom and click Create Distribution in the bottom right.

Creating an AWS CloudFront distribution

Previewing your new CloudFront distribution

After hitting the Create button, it will take some time for your distribution to be created and set up. You’ll notice on the CloudFront Distributions list page that the Status of your new distribution is In Progress.

AWS CloudFront distribution deployment is In Progress

Once this completes, it will say Deployed. Then you can find your Domain Name in the same row.

AWS CloudFront distribution is Deployed

Using the value in the Domain Name column, open your distribution in your browser and success! You now are viewing your S3 bucket through CloudFront’s distribution network.

Hello World! Opening an AWS CloudFront website

Custom domain names

While most of us will probably want to use a custom domain name with our website, we’re not going to dive too deep into that this guide, as there are many ways to set that up depending on where you purchase your domain name.

However, here are a few things to consider.

HTTPS / SSL Certificate

If you’re creating your CloudFront distribution to use with a custom domain name, you'll most likely want to configure your distribution with an SSL certificate using AWS’s Certificate Manager. Alternatively you can provide your own certificate with tools like Let's Encrypt, but by using ACM, AWS makes it easy to pull in the records for use with your distribution.

Once in ACM, you’ll want to configure the certificate, map what domains and subdomains should match (typically *.domain.com), and then create your certificate to use with your distribution.

To get started, you can check out the AWS guide for requesting a public certificate.

CNAMEs and Aliases

A common approach to setting up a custom domain is to use a CNAME. CloudFront makes this pretty painless, as you’ll add it as a configuration option when you’re configuring your distribution.

To get started with setting up a CNAME in CloudFront, see the AWS guide.

If you’re using Route53 to manage your DNS, you can then set up an A record (alias) to point to your distribution. You can learn more using this guide.

Advanced AWS Usage

For this guide, we walked you through setting up a new static website and app using the AWS console. But whether you want to learn more, improve your deploy efficiency, or want to automate this process, you’ll want to take a it a step further with the AWS CLI or CloudFormation.

While we won’t walk you through how to use these tools here, we’ll get you started with a little bit of an idea of what you’re up against.

AWS CLI

The AWS CLI allows someone to perform AWS operations from the command line. This can be incredibly powerful when you want to script out your resource creation or if you simply prefer to do all of your work from the terminal.

Once set up locally, you’ll be able to perform actions like creating a bucket using the following command:

aws s3api create-bucket —-bucket [your-bucket-name] —-region [bucket-region]

To get started, check out the AWS CLI Github page or the AWS CLI User Guide .

AWS CloudFormation

AWS preaches “infrastructure as code.” It’s the idea that you can spin up your infrastructure using something that’s written in a file, where in this particular case, it would be a CloudFormation template. This allows you to have a repeatable process that will be the same each time you perform the deploy.

CloudFormation allows you to set up a configuration file that will deploy the services and resources of your choosing by pointing to that file with the CLI or by uploading it in the console.

Here’s an example from AWS of what that looks like for a static S3 bucket that could serve as a website.

AWS CloudFront template example

To get started, check out AWS’s CloudFormation example templates or their Get Started guide.

Resources

If you’re interested in getting deeper into the AWS ecosystem, here are a few resources to get started:

• AWS Certified Cloud Practitioner Training 2019 - A Free 4-hour Video Course (freeCodeCamp.org)
• Introducing The #AWSCertified Challenge: A Path to Your First AWS Certifications (freeCodeCamp.org)
• 10-Minute Tutorials (AWS)
• A Cloud Guru (Paid courses)
• AWS Case Studies (AWS)

Hiya folks!

In this tutorial I’ll show you how to host a static website with HTTPS on AWS with a custom domain. All this is possible using AWS free tier.

However, the services we are going to use do incur some small charges. Generally speaking these shouldn’t exceed $1/month.

We’ll be using a combination of the following AWS services:
—S3
— Route53
— Certificate manager
— CloudFront

Let’s get into it!

Setup your S3 buckets

First, you’ll need two S3 buckets, both should match your custom domain name with the second including the www subdomain.

Bucket 1: mywebsite.com
Bucket 2: www.mywebsite.com

The first bucket (mywebsite.com) is the main bucket for your site. This contains all your files and assets for your static website.

Next we setup this bucket for static site hosting. You can find this under the Properties tab of the bucket, and we’re going to keep the defaults provided here with the index of the site set to index.html.

We also need to make this bucket publicly accessible as a user’s browser will need to access the bucket’s files in order to render the website. We can do this by setting a Bucket Policy under the Permissions tab.

{       "Version": "2012-10-17",       "Statement": [        {            "Sid": "PublicReadGetObject",            "Effect": "Allow",            "Principal": "*",            "Action": "s3:GetObject",            "Resource": "MY_BUCKET_ARN"        }    ]}

This is a simple policy that will only allow public read access of objects in the bucket. Now, if you head to the endpoint defined in the static hosting config of the bucket, you should see your website.

Progress! But we can do better than that.

The second bucket (www.mywebsite.com) we will leave empty but configure to redirect to our first bucket using HTTP as the protocol (we’ll make it HTTPS later).

Redirect requests back to the main bucket using HTTP protocol

Your buckets are now ready to go!

Configure Domains with Route53

So your website is up and running but only accessible via the bucket endpoint and not your custom domain. Let’s change that.

Head to Route53. If you’ve registered your domain with the Amazon Registrar you should see that a hosted zone has been setup for you with two record sets. One for Name Server (NS) and one for SOA.

All we need to do is to create two more record sets to point to the S3 bucket endpoints.

For each record set:
— Type: A — IPv4 address
— Alias: Yes
— Alias Target: the S3 website endpoint that matches what you set for Name.

Creating a record set for www subdomain

Now we can head to the custom url…and voilà!
We’re almost there, but there’s one last thing we’re missing…

Note: If your domain is registered with another domain registrar (not Amazon) you’ll need to follow some different steps to set this up. Usually you’ll need to add a CNAME record with a value of the main S3 buckets endpoint.

Troubleshooting:
If you deleted the hosted zone Amazon created when you first registered the domain (I’ve done this because hosted zones do incur some charges), you’ll need to create a new hosted zone from scratch.

1. Select “Create Hosted Zone” and set the domain name, for example “mywebsite.com”
2. This will generate some new record sets for types NS and SOA.
3. Go into your registered domain and update the Name Servers values to those generated in the new NS record set.

Requesting a Certificate

Awesome, the site is now hosted using the custom url! However we can only access it via HTTP protocol.
We should always ensure our sites are secured using HTTPS protocol. This protects our site and users from malicious injection attacks and guarantees authenticity.

Head to Certificate Manager in AWS Console and request a new public certificate (this is free). You’ll be prompted to enter the domain names you wish to secure.

Before the certificate can be issued, Amazon needs to be able to verify that you own the specified domains.

You can choose from two verification methods: Email or DNS.

Email is generally simpler, but you’ll need to ensure you can access the email used to register the domain. Alternatively, if you used Amazon Registrar and Route53, you can select the DNS method. This requires you to add some specific record sets to the hosted zone, but this is mostly automated for you so it’s quite simple.

It can take a few minutes for the certificate to be issued after validation.
When its all done we can continue to the final step!

Configuring CloudFront

For the final step we are going to use CloudFront which allows us to use the new SSL certificate to serve the website with HTTPS. CloudFront also speeds up the distribution of web content by storing it at multiple edge locations and delivering from the closest edge location to a user.

We need two new web distributions, one for each S3 bucket. Head to CloudFront in the AWS Console and create the first web distribution.
There are lots of settings available to create a web distribution, but for the basics we only need to change five:

1. Origin Domain Name: Set this to the S3 website endpoint for one of the buckets. Important: This field will give you some auto-complete options with your S3 bucket names. However, using these can cause issues with redirecting to the bucket endpoint. So instead use the bucket endpoint directly.
2. Origin Id: This populated for you when you enter Origin Domain Name.
3. Viewer Protocol Policy: Set to “Redirect HTTP to HTTPS”.
4. Alternate Domain Names: This should match the name of the S3 bucket you’re pointing to. For example “mywebsite.com”.
5. SSL Certificate: Select “Custom SSL Certificate” and select your new certificate from the dropdown.

Do this again for the second S3 bucket.

The distributions can take a while to spin up, so while we wait, let’s do the finishing steps.

Back in S3, go to your secondary bucket (www.mywebsite.com), in the Properties tab and under Static Website Hosting set the redirect protocol to HTTPS.

Finally, head back to Route53. We need to update the custom A records we created to now target the CloudFront distributions rather than the S3 buckets. For each record, change the Alias Target and select the CloudFront distribution available in the dropdown.

Note: Again, if you are using another DNS service you’ll need to go update the CNAME record from there to point to the CloudFront domain name.

Huzzah!

And there you have it! Your beautiful website is now available at the custom domain and served with HTTPS!

_[From Giphy](https://giphy.com" rel="noopener" target="blank" title=")

Thanks for reading! I hope this guide was useful and enjoyable, I’d love to know if you found it helpful.