But it has always carried a cost that rarely gets discussed openly. That cost isn't just money, though a DynamoDB table with on-demand billing adds up across multiple teams and environments.

The real cost is complexity. Every new AWS environment needs both resources provisioned before Terraform can manage anything else. Every engineer who sets up their first Terraform backend has to understand why two completely different AWS services are responsible for what is logically one thing: storing and protecting state. And every incident involving a stuck lock has required someone to manually delete a record from DynamoDB to unblock the team.

In November 2024, AWS announced that S3 now supports native object locking for Terraform state files, meaning DynamoDB is no longer required for state locking. Terraform 1.10 added support for this feature, and it's now generally available.

In this tutorial, you'll learn:

• What S3 native locking is and how it works

• How to set it up from scratch if you're starting a new project

• How to migrate an existing S3 + DynamoDB setup to S3 native locking safely

• How to verify locking is working and handle edge cases

By the end, you'll have a simpler, cleaner Terraform backend with one fewer AWS resource to manage.

Table of Contents

• What Is Terraform State Locking?

• What Is S3 Native State Locking?

• How S3 Native Locking Compares to the S3 + DynamoDB Approach

• Prerequisites

• Part 1: Fresh Setup – How to Configure S3 Native Locking from Scratch

  • Step 1: Create the S3 Bucket with Versioning and Encryption

  • Step 2: Configure the Terraform Backend with Native Locking

  • Step 3: Initialize and Verify

• Part 2: Migration – How to Move from S3 + DynamoDB to S3 Native Locking

  • Step 1: Verify Your Current Setup

  • Step 2: Enable Object Lock on the Existing S3 Bucket

  • Step 3: Update the Terraform Backend Configuration

  • Step 4: Reinitialize Terraform

  • Step 5: Verify the Migration

  • Step 6: Clean Up the DynamoDB Table

• How to Verify That Locking Is Working

• How to Handle a Stuck Lock

• Rollback Plan: If Something Goes Wrong

• Security Best Practices for Your State Bucket

• Conclusion

• References

What is Terraform State Locking?

Before looking at the new approach, it helps to understand what state locking is solving.

Terraform stores everything it knows about your infrastructure in a state file – a JSON document that maps your configuration to real AWS resources. When you run terraform apply, Terraform reads this file, calculates the difference between the current state and your configuration, and makes the necessary changes.

The problem arises when two engineers or two CI/CD pipelines run and try to apply changes at the same time. If both read the state file simultaneously, calculate changes independently, and both try to write back, you get a race condition. The second write overwrites changes from the first, and your state is now out of sync with reality. This is a serious problem that can cause resources to be untracked, doubled, or destroyed unexpectedly.

State locking solves this by creating a lock when any operation starts that could modify state. If a lock already exists, Terraform refuses to proceed and reports who holds the lock and when it was acquired. Only one operation can hold the lock at a time. When the operation completes, the lock is released.

Terraform Run A                 State File / Lock                Terraform Run B
(User 1)                         (S3/DynamoDB)                   (User 2)

   |                                   |                            |
   |------- 1. Acquire Lock ---------->|                            |
   |                                   |                            |
   |<------ 2. Lock Granted -----------|                            |
   |                                   |                            |
   |                                   |------- 3. Acquire Lock --->|
   |            [PROCESSING]           |                            |
   |      (Modifying Infrastructure)   |<------ 4. Lock Denied -----|
   |                                   |        (Wait / Retry)      |
   |                                   |                            |
   |------- 5. Release Lock ---------->|                            |
   |                                   |                            |
   |           [COMPLETED]             |<------ 6. Lock Granted ----|
   |                                   |                            |
   |                                   |       [PROCESSING]         |
   |                                   | (Modifying Infrastructure) |              
   |                                   |                            |

What Is S3 Native State Locking?

Previously, Terraform's S3 backend used a DynamoDB table as the locking mechanism. When a lock was needed, Terraform wrote a record to DynamoDB with a LockID primary key. DynamoDB's conditional writes guaranteed that only one process could create that record, which is what made the locking atomic.

S3 native locking uses S3 Object Lock instead. S3 Object Lock is an S3 feature originally designed to enforce WORM (Write Once, Read Many) compliance for regulatory requirements. AWS extended this capability to support Terraform's state locking workflow.

When S3 native locking is enabled in your Terraform backend:

1. Terraform writes your state to an .tfstate object in S3 (as before)

2. To acquire a lock, Terraform uses S3's conditional write operations – specifically the if-none-match conditional header to create a lock file atomically

3. If the lock file already exists, S3 rejects the write, and Terraform reports that a lock is held

4. When the operation completes, Terraform deletes the lock file to release the lock.

The key difference from DynamoDB: the entire locking mechanism lives inside S3. No second service. No second set of IAM permissions. No second resource to provision.

Note: This feature requires Terraform version 1.10.0 or later and an S3 bucket with Object Lock enabled. Object Lock must be enabled at bucket creation time. You can't enable it on an existing bucket through the console or CLI. But there is a supported workaround for existing buckets, which we'll cover in Part 2.

How S3 Native Locking Compares to the S3 + DynamoDB Approach

The functional behavior from Terraform's perspective is identical. Locking works the same way. The lock information displayed when a lock is held has the same structure. The only difference is what happens under the hood.

Prerequisites

Before you start, make sure you have the following in place:

• Terraform 1.10.0 or later installed. Check your version:

terraform version

If you need to upgrade, follow the official upgrade guide.

• AWS CLI installed and configured with credentials that have permission to create and manage S3 buckets.

aws --version
aws sts get-caller-identity   # confirm you're authenticated

• IAM permissions to perform the following S3 actions:

  • s3:CreateBucket

  • s3:PutBucketVersioning

  • s3:PutBucketEncryption

  • s3:PutObjectLegalHold

  • s3:PutObjectRetention

  • s3:GetObject

  • s3:PutObject

  • s3:DeleteObject

  • s3:ListBucket

• For the migration path: access to your existing Terraform project and the S3 bucket and DynamoDB table currently in use.

Part 1: Fresh Setup – How to Configure S3 Native Locking from Scratch

Follow this section if you're starting a new Terraform project and want to use S3 native locking from the beginning.

Step 1: Create the S3 Bucket with Versioning and Encryption

Object Lock must be enabled at bucket creation time. You can't add it afterward through the standard console flow. Create the bucket using the AWS CLI with Object Lock enabled:

aws s3api create-bucket \
  --bucket your-project-terraform-state \
  --region us-east-1 \
  --object-lock-enabled-for-bucket

Note: For regions other than us-east-1, add the --create-bucket-configuration flag.

aws s3api create-bucket \
  --bucket your-project-terraform-state \
  --region eu-west-1 \
  --create-bucket-configuration LocationConstraint=eu-west-1 \
  --object-lock-enabled-for-bucket

Now enable versioning on the bucket. Versioning is required alongside Object Lock and allows Terraform to recover previous state versions if something goes wrong:

aws s3api put-bucket-versioning \
  --bucket your-project-terraform-state \
  --versioning-configuration Status=Enabled

Enable server-side encryption so your state files are encrypted at rest:

aws s3api put-bucket-encryption \
  --bucket your-project-terraform-state \
  --server-side-encryption-configuration '{
    "Rules": [
      {
        "ApplyServerSideEncryptionByDefault": {
          "SSEAlgorithm": "AES256"
        },
        "BucketKeyEnabled": true
      }
    ]
  }'

Block all public access to the bucket. A Terraform state file contains resource IDs, IP addresses, and potentially sensitive values. It should never be publicly accessible:

aws s3api put-public-access-block \
  --bucket your-project-terraform-state \
  --public-access-block-configuration \
    "BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true"

Verify the bucket configuration:

# Confirm Object Lock is enabled
aws s3api get-object-lock-configuration \
  --bucket your-project-terraform-state
 
# Confirm versioning is enabled
aws s3api get-bucket-versioning \
  --bucket your-project-terraform-state
 
# Confirm encryption is configured
aws s3api get-bucket-encryption \
  --bucket your-project-terraform-state

Expected output for the Object Lock check:

{
    "ObjectLockConfiguration": {
        "ObjectLockEnabled": "Enabled"
    }
}

Step 2: Configure the Terraform Backend with Native Locking

In your Terraform project, create or update your backend.tf file:

terraform {
  backend "s3" {
    bucket = "your-project-terraform-state"
    key    = "production/terraform.tfstate"
    region = "us-east-1"
 
    # Enable S3 native state locking
    # Requires Terraform 1.10.0+ and a bucket with Object Lock enabled
    use_lockfile = true
 
    # Encryption at rest
    encrypt = true
  }
}

The critical difference from the old configuration is the use_lockfile = true parameter. Notice what is absent: there's no dynamodb_table argument. No DynamoDB table. No second service.

Here's a direct comparison of the old and new configurations:

Old configuration (S3 + DynamoDB):

terraform {
  backend "s3" {
    bucket         = "your-project-terraform-state"
    key            = "production/terraform.tfstate"
    region         = "us-east-1"
    encrypt        = true
    dynamodb_table = "terraform-state-lock"   # this goes away
  }
}

New configuration (S3 native locking):

terraform {
  backend "s3" {
    bucket       = "your-project-terraform-state"
    key          = "production/terraform.tfstate"
    region       = "us-east-1"
    encrypt      = true
    use_lockfile = true   # this replaces dynamodb_table
  }
}

Step 3: Initialize and Verify

Run terraform init to initialize the backend:

terraform init

Expected output:

Initializing the backend...
 
Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.
 
Initializing provider plugins...
 
Terraform has been successfully initialized!

Run a plan to confirm everything is working end-to-end:

terraform plan

If locking is working, you'll see a brief pause while Terraform acquires the lock before the plan output appears. You'll also see the lock information if you look at the S3 bucket – a .tflock file will appear temporarily alongside your state file during the operation and disappear when it completes.

Part 2: Migration – How to Move from S3 + DynamoDB to S3 Native Locking

Follow this section if you have an existing Terraform setup using an S3 bucket and DynamoDB table for state locking, and you want to migrate to S3 native locking.

Important: Migration requires a maintenance window or at minimum a period where no Terraform operations are running. You're changing the backend configuration, which means all team members and CI/CD pipelines must stop running terraform plan or terraform apply during the migration. The migration itself takes under 10 minutes.

Step 1: Verify Your Current Setup

Before making any changes, document your existing backend configuration and confirm the state file is accessible:

# Confirm your state file is in S3
aws s3 ls s3://your-existing-bucket/path/to/terraform.tfstate
 
# Confirm the DynamoDB table exists
aws dynamodb describe-table \
  --table-name your-dynamodb-lock-table \
  --query 'Table.TableStatus'

Check your current backend.tf and note the exact values:

# Your current backend.tf - note these values before changing anything
terraform {
  backend "s3" {
    bucket         = "your-existing-bucket"       # note this
    key            = "path/to/terraform.tfstate"   # note this
    region         = "us-east-1"                   # note this
    encrypt        = true
    dynamodb_table = "your-dynamodb-lock-table"    # this will be removed
  }
}

Run one final plan to confirm the current state is clean and there are no unexpected changes pending:

terraform plan

If the plan shows no changes, you're in a safe state to proceed.

Step 2: Enable Object Lock on the Existing S3 Bucket

This is the most important step in the migration. Object Lock can't normally be enabled on an existing bucket. It's a setting that must be configured at creation time.

But AWS provides a way to enable Object Lock on an existing bucket through a support request or through a direct API call that's not exposed in the standard console UI. AWS has officially documented this path for the Terraform migration use case.

Run the following AWS CLI command to enable Object Lock on your existing bucket:

aws s3api put-object-lock-configuration \
  --bucket your-existing-bucket \
  --object-lock-configuration '{"ObjectLockEnabled": "Enabled"}'

Note: This command enables Object Lock in governance mode with no default retention, meaning it enables the locking capability without setting a default retention period on all objects. This is exactly what Terraform's native locking needs: the ability to create and delete lock files, not permanent object retention.

Verify Object Lock is now enabled:

aws s3api get-object-lock-configuration \
  --bucket your-existing-bucket

Expected output:

{
    "ObjectLockConfiguration": {
        "ObjectLockEnabled": "Enabled"
    }
}

Also verify that versioning is already enabled (it should be if you are running a production Terraform setup):

aws s3api get-bucket-versioning \
  --bucket your-existing-bucket

Expected output:

{
    "Status": "Enabled"
}

If versioning isn't enabled, enable it before proceeding:

aws s3api put-bucket-versioning \
  --bucket your-existing-bucket \
  --versioning-configuration Status=Enabled

Step 3: Update the Terraform Backend Configuration

Update your backend.tf to remove the dynamodb_table argument and add use_lockfile = true:

terraform {
  backend "s3" {
    bucket = "your-existing-bucket"
    key    = "path/to/terraform.tfstate"
    region = "us-east-1"
    encrypt = true
 
    # Add this:
    use_lockfile = true
 
    # Remove this line entirely:
    # dynamodb_table = "your-dynamodb-lock-table"
  }
}

Your updated backend.tf should look like this:

terraform {
  backend "s3" {
    bucket       = "your-existing-bucket"
    key          = "path/to/terraform.tfstate"
    region       = "us-east-1"
    encrypt      = true
    use_lockfile = true
  }
}

Step 4: Reinitialize Terraform

Run terraform init with the -reconfigure flag. This flag tells Terraform that the backend configuration has changed intentionally and to reinitialize without prompting you to copy state (the state is already in the same bucket):

terraform init -reconfigure

Expected output:

Initializing the backend...
 
Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.
 
Initializing provider plugins...
- Reusing previous version of hashicorp/aws from the dependency lock file
 
Terraform has been successfully initialized!

If you see an error here: The most common cause is that Object Lock wasn't successfully enabled on the bucket. Re-run the verification from Step 2 before proceeding.

Step 5: Verify the Migration

Run a plan to confirm Terraform is working correctly with the new backend configuration:

terraform plan

The plan should:

• Complete successfully

• Show the same result as the plan you ran in Step 1 (no changes, or the same changes as before)

• NOT mention DynamoDB anywhere in its output

To confirm that locking is actually using S3 instead of DynamoDB, open a second terminal and run a plan while the first one is running. You should see the second terminal output a lock error that mentions S3, not DynamoDB:

╷
│ Error: Error acquiring the state lock
│
│Error message: operation error S3: PutObject, https response       error StatusCode: 409,
│ RequestID: ..., api error Conflict: Object lock already exists for this key.
│
│ Lock Info:
│   ID:        a1b2c3d4-e5f6-7890-abcd-ef1234567890
│   Path:      your-existing-bucket/path/to/terraform.tfstate.tflock
│   Operation: OperationTypePlan
│   Who:       user@hostname
│   Version:   1.10.0
│   Created:   2026-05-06 14:22:01 UTC
│   Info:
╵

The Path field shows .tfstate.tflock, a file in your S3 bucket, not a DynamoDB record. This confirms that locking is now handled entirely by S3.

Step 6: Clean Up the DynamoDB Table

Once you've confirmed the migration is working correctly and your team has run at least one successful plan and apply cycle using the new backend, you can remove the DynamoDB table.

Wait at least 24-48 hours before deleting the DynamoDB table if you have CI/CD pipelines or multiple team members. This gives time to catch any pipeline that wasn't updated with the new backend configuration.

When you're ready, delete the DynamoDB table:

aws dynamodb delete-table \
  --table-name your-dynamodb-lock-table

Confirm the deletion:

aws dynamodb describe-table \
  --table-name your-dynamodb-lock-table

Expected output:

An error occurred (ResourceNotFoundException) when calling the DescribeTable operation:
Requested resource not found

This error confirms that the table is gone. The migration is complete.

If you provisioned the DynamoDB table using Terraform (which is the recommended pattern), remove the resource from your Terraform configuration and run terraform apply to destroy it via Terraform rather than the CLI directly. This keeps your state clean:

# Remove this entire block from your Terraform configuration:
resource "aws_dynamodb_table" "terraform_state_lock" {
  name         = "terraform-state-lock"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "LockID"
 
  attribute {
    name = "LockID"
    type = "S"
  }
}

After removing the block, run:

terraform apply

Terraform will detect that the DynamoDB table resource has been removed from configuration and will destroy the table.

How to Verify That Locking Is Working

After completing either the fresh setup or the migration, use this procedure to independently verify that locking is functioning correctly.

Method 1: Observe the lock file during an operation

In one terminal, start a long-running plan against a configuration with many resources:

terraform plan

While it's running, in a second terminal, check for the lock file in S3:

aws s3 ls s3://your-bucket/path/to/ | grep tflock

You should see a file like:

2026-05-06 14:22:01        512 terraform.tfstate.tflock

After the plan completes, run the same command again. The .tflock file should be gone.

Method 2: Read the lock file contents

While a plan is running, download and read the lock file to see its contents:

aws s3 cp \
  s3://your-bucket/path/to/terraform.tfstate.tflock \
  /tmp/current.lock && cat /tmp/current.lock

Expected output (formatted for readability):

{
  "ID": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "Operation": "OperationTypePlan",
  "Info": "",
  "Who": "tolani@dev-machine",
  "Version": "1.10.0",
  "Created": "2026-05-06T14:22:01.123456789Z",
  "Path": "your-bucket/path/to/terraform.tfstate"
}

This is the same lock information that Terraform displays when a lock is held. It's now a JSON file in S3 rather than a record in DynamoDB.

How to Handle a Stuck Lock

With the DynamoDB backend, resolving a stuck lock meant deleting a record from the DynamoDB table. With S3 native locking, it means deleting the .tflock file from S3.

A lock can get stuck if:

• A terraform apply or plan process was killed mid-execution

• A CI/CD pipeline runner crashed during a Terraform operation

• A network interruption prevented the lock release from completing

Here's how you can check for a stuck lock:

aws s3 ls s3://your-bucket/path/to/ | grep tflock

If a .tflock file exists and no Terraform operation is currently running, it is a stuck lock.

You can also read the lock to understand who held it:

aws s3 cp \
  s3://your-bucket/path/to/terraform.tfstate.tflock \
  /tmp/stuck.lock && cat /tmp/stuck.lock

This tells you who (Who field) was running the operation, what operation it was (Operation field), and when it was acquired (Created field).

And you can force-unlock using Terraform like this:

terraform force-unlock LOCK-ID

Replace LOCK-ID with the ID value from the lock file contents. For example:

terraform force-unlock a1b2c3d4-e5f6-7890-abcd-ef1234567890

Terraform will confirm:

Do you really want to force-unlock?
  Terraform will remove the lock on the remote state.
  This will allow local Terraform commands to modify this state, even though it
  may be still be in use. Only 'yes' will be accepted to confirm.
 
  Enter a value: yes
 
Terraform state has been successfully unlocked!

An alternative is to delete the lock file directly via CLI. If terraform force-unlock doesn't work (for example, because you are running in a CI environment without Terraform available), delete the lock file directly:

aws s3 rm s3://your-bucket/path/to/terraform.tfstate.tflock

Only delete the lock file if you are certain no Terraform operation is currently running. Deleting a lock that is actively held by a running operation will allow a second concurrent operation to start, which is exactly the race condition locking is designed to prevent.

Rollback Plan: If Something Goes Wrong

If you encounter problems after migrating, you can roll back to the S3 + DynamoDB setup with these steps.

Step 1: Stop all Terraform operations in your team and CI/CD pipelines.

Step 2: Recreate the DynamoDB table if you already deleted it:

aws dynamodb create-table \
  --table-name terraform-state-lock \
  --attribute-definitions AttributeName=LockID,AttributeType=S \
  --key-schema AttributeName=LockID,KeyType=HASH \
  --billing-mode PAY_PER_REQUEST

Step 3: Revert backend.tf to the previous configuration:

terraform {
  backend "s3" {
    bucket         = "your-existing-bucket"
    key            = "path/to/terraform.tfstate"
    region         = "us-east-1"
    encrypt        = true
    dynamodb_table = "terraform-state-lock"   # restored
    # Remove: use_lockfile = true
  }
}

Step 4: Reinitialize:

terraform init -reconfigure

Step 5: Verify:

terraform plan

The state file hasn't moved, so there's no data loss during a rollback. The only change is which locking mechanism Terraform uses.

Note: Object Lock being enabled on the S3 bucket doesn't prevent the rollback. Object Lock and DynamoDB locking can coexist, Object Lock simply adds a capability to the bucket. Using dynamodb_table in your backend config tells Terraform to use DynamoDB regardless of whether Object Lock is enabled on the bucket.

Security Best Practices for Your State Bucket

Migrating to S3 native locking is a good opportunity to review the overall security configuration of your state bucket. Here are the practices every production Terraform state bucket should implement:

Enable Versioning (Required)

Versioning is a hard requirement for S3 native locking to work safely. It ensures that if a state file is accidentally overwritten or corrupted, you can restore a previous version.

aws s3api put-bucket-versioning \
  --bucket your-state-bucket \
  --versioning-configuration Status=Enabled

Block All Public Access (Non-Negotiable)

Your state file contains resource ARNs, IP addresses, and may contain sensitive values passed through Terraform variables. It must never be publicly accessible.

aws s3api put-public-access-block \
  --bucket your-state-bucket \
  --public-access-block-configuration \
    "BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true"

Enable Server-Side Encryption

Always encrypt state files at rest. AES256 is the minimum. If your organization requires KMS key management:

aws s3api put-bucket-encryption \
  --bucket your-state-bucket \
  --server-side-encryption-configuration '{
    "Rules": [
      {
        "ApplyServerSideEncryptionByDefault": {
          "SSEAlgorithm": "aws:kms",
          "KMSMasterKeyID": "arn:aws:kms:us-east-1:123456789012:key/your-kms-key-id"
        },
        "BucketKeyEnabled": true
      }
    ]
  }'

Apply Least-Privilege IAM Permissions

The role or user that Terraform uses to access the state bucket should have only the permissions it needs. Here's a minimal IAM policy for S3 native locking:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "TerraformStateAccess",
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject"
      ],
      "Resource": [
        "arn:aws:s3:::your-state-bucket",
        "arn:aws:s3:::your-state-bucket/*"
      ]
    },
    {
      "Sid": "TerraformStateLocking",
      "Effect": "Allow",
      "Action": [
        "s3:GetObjectLegalHold",
        "s3:PutObjectLegalHold",
        "s3:GetObjectRetention",
        "s3:PutObjectRetention"
      ],
      "Resource": "arn:aws:s3:::your-state-bucket/*.tflock"
    }
  ]
}

Notice what is absent: there are no DynamoDB permissions. This is a cleaner, smaller permission set than the old approach required.

Enable Access Logging

Log all access to your state bucket in CloudTrail or S3 server access logs. This gives you an audit trail of every time state was read, written, or locked:

aws s3api put-bucket-logging \
  --bucket your-state-bucket \
  --bucket-logging-status '{
    "LoggingEnabled": {
      "TargetBucket": "your-logging-bucket",
      "TargetPrefix": "terraform-state-access/"
    }
  }'

Conclusion

AWS S3 native state locking removes the need for a DynamoDB table from your Terraform backend setup. The result is simpler infrastructure, a smaller IAM permission surface, and one fewer service to provision, monitor, and pay for across every environment your team manages.

Here's a summary of what you accomplished:

• Understood what state locking is and why it's required for safe Terraform operations

• Compared S3 native locking to the existing S3 + DynamoDB approach

• Set up a fresh Terraform backend using S3 native locking with correct bucket configuration

• Migrated an existing backend from S3 + DynamoDB to S3 native locking safely

• Learned how to verify locking, handle stuck locks, and roll back if needed

• Applied security best practices to the state bucket

This pattern – using S3 native locking – is the recommended approach for all new Terraform projects on AWS going forward. If you're managing a large estate with multiple Terraform backends, consider automating the migration using a script or Terraform module that applies the pattern across all your state buckets.

If you are building or optimizing cloud infrastructure for a startup and want a complete reference for production-ready Terraform modules, CI/CD pipeline patterns, and infrastructure runbooks, check out The Startup DevOps Field Guide. It covers the full lifecycle of AWS infrastructure from initial setup to production reliability.

References

• HashiCorp - S3 Backend Configuration: use_lockfile

• HashiCorp: Terraform 1.10 Release Notes

• AWS Docs: S3 Object Lock Overview

• AWS Docs: PutObjectLockConfiguration API

• AWS Docs: S3 Conditional Writes

• HashiCorp: Backend State Locking

• HashiCorp: terraform force-unlock Command

• AWS Docs: Enabling S3 Versioning

• AWS Docs: S3 Server-Side Encryption

You might be wondering: what is Infrastructure-as-Code? How does it improve the development process and experience, and where does Terraform come into the picture? Well, we’ll explore all this and more in this guide. But before we start, here are some pre-requisites:

• Basic knowledge of the cloud and cloud terminologies

• Access to a PC to implement code examples

• A GCP account

With this, let's get started.

Here’s what we’ll cover:

1. Overview of Infrastructure as Code

2. What is Terraform?

3. Benefits of Terraform

4. Common Terms Used in Terraform

5. Demo Project: How to Write a Terraform Configuration

6. Conclusion

Overview of Infrastructure as Code (IaC)

Infrastructure as code refers to generating cloud infrastructure tools and applications with a code-based configuration document. This process, when running, automates the sequence and process of creating databases, virtual machines, and servers. This improves the user experience by reducing the frequency of manual cloud service deployments, especially for multiple identical services.

There are two distinct approaches to infrastructure as code: the Imperative approach and the Declarative approach.

When you’re using the Declarative approach to infrastructure generation, you simply detail your expected/desired outputs for the Infrastructure to be generated, and then the IaC tool you’re using figures out how to produce that output.

On the other hand, the Imperative approach involves specifying the exact steps needed to achieve the desired infrastructure state. While the Imperative approach seems more suited for complex infrastructure setups, the Declarative approach can work just as well.

Some tools are capable of both approaches while others are only suited to one or the other. Examples of some of the popular IaC tools used globally include Terraform IaC, AWS Cloud Formation, Ansible, and Pulumi, Chef, among others.

Like the name implies – infrastructure as code – the code creating the infrastructure is written in various template languages within the IaC space. Popular template languages include JSON, YAML, ARM template, HCL, Heat Scripts, and so on.

You can also use scripting tools to execute cloud infrastructure. Some popular ones include Bash and PowerShell. These sometimes come preinstalled on most personal computers.

Out of all these tools, though, Terraform is distinct for various reasons – and it’s the one we’ll be examining in this article.

What is Terraform?

Terraform is an open source tool developed by HashiCorp in 2014. It has evolved over the years and now serves as a cloud agnostic infrastructure tool that allows you to create infrastructure across multiple cloud service providers.

Terraform also offers Terraform Cloud, a cloud-based software as a service tool. It allows for cloud-based deployment of cloud tools, instead of using the old local-based methods we had in the defunct Terraform CLI tool.

Also, like other IaC tools which utilize template languages, the template framework used to create infrastructure in Terraform is the HashiCorp template language (HCL).

Benefits of Terraform

Now I’ll highlight some of the benefits of using Terraform as a cloud engineer, along with the tool’s key role in the cloud ecosystem.

1. Declarative Approach

This approach to cloud infrastructure automation ensures that all required infrastructure to be deployed (databases, servers, and so on) is stated explicitly and executed accordingly. This helps avoid conflicts.

2. Conflict Handling

In addition to its efficient cloud tool automation capabilities, Terraform has some robust conflict detection and handling properties. One of the ways it handles conflicts is via the Terraform plan function. This function highlights any perceived or potential conflicts of infrastructure orchestration which allows for easy correction before deployment. I’ll discuss this further in subsequent sections.

3. Cloud Agnostic

Terraform is a multipurpose, multi-cloud automation service provider with efficient infrastructure automation capabilities across the major cloud service providers (AWS, GCP and Azure). It also allows for hybrid and inter-provider automation.

4. User-friendly

Terraform is one of the largest cloud automation tools with the largest user communities out there. It has extensive beginner-friendly tutorials that help you get a quick hang of the tool. Here is a link to its documentation so you can dive in deeper.

5. File Management Capabilities

Terraform automatically creates a local backup of the automation states on your local computer to ensure immediate recall and file handling in case anything goes wrong. It also offers remote backup options to remote cloud service providers where necessary.

6. Version Control

Just like the Git version control system, Terraform has a built-in version control system which lets you track changes to a Terraform file. It also lets you go back to previous versions of your code if there are errors in the present version, for example.

7. Code Reusability

Terraform offers a wide variety of code templates for easy reuse on its developer documentation page.

Now that we’ve highlighted the benefits of Terraform, let’s learn some common terminologies used in Terraform and what they mean.

Common Terms Used in Terraform

Before you start using Terraform, you should be familiar with some key terms that come up a lot. Here’s what you need to know:

1. Providers: in Terraform, a Provider is a programming interface that lets Terraform interact with various APIs and cloud services. For example, you’d use a provider to interface with a cloud service provider like GCP or Azure.

2. Modules: Modules are specifically created within the Terraform framework and serve as reusable components that let you easily orchestrate cloud services. You can also store key information regarding cloud services in a module, and then modify it to ensure uniqueness using module variables.

3. Resources: Resources in Terraform refer to the cloud infrastructure components to be created. Examples include cloud networks, virtual machines, availability zones, and other infrastructures.

4. State: The concept of state in Terraform forms the basis for its efficiency. State keeps track of the current configuration of your infrastructure resources, and contains details about every resource Terraform has created, modified, or deleted. Terraforms version control system uses it to track any changes you make to a code file and uses that information to destroy and provision infrastructure as necessary.

5. Workspace: a Workspace functions sort of similarly to a version control system, as it creates a sort of constraint around a work file. Workspaces let you manage multiple instances of a single infrastructure configuration in a clean and isolated way within the same backend. You can use workspaces to separate environments like development, staging, and production while using the same Terraform configuration.

Demo Project: How to Write a Terraform Configuration

In this section, we’ll be diving deeper into writing our first Terraform file to orchestrate a Google Cloud program virtual machine with just a few lines of code. But before we begin, we’ll discuss the various commands that you should understand before we implement the demo project.

Common Terraform Commands

• Terraform init: This command initializes the Terraform tool and downloads essential cloud provider-specific files. It also establishes a connection between Terraform and the cloud provider in question. In our case, it’s between GCP and the Terraform provider.

• Terraform fmt: This command automatically ensures optimal code formatting and indentation. It ensures orderly execution of the code and minimizes any errors.

• Terraform plan: This command outlines the steps of execution of the Terraform code, and detects any errors that may occur during the process of execution. It also highlights any errors in the Terraform code that may hinder execution. Lastly, it works alongside Terraform state management to detect any change of state and de-provision or generate any additional cloud services if necessary.

• Terraform apply: This command executes the planned Terraform state implemented by the Terraform plan command.

• Terraform destroy: This command is the final command in the Terraform scheme which is used to deactivate or destroy all the cloud services created using the Terraform apply command. It's important to note that you should execute the commands listed above sequentially to ensure that your infrastructure gets created properly.

Creating an IaC-Powered GCP Virtual Machine

Now that you’ve learned these important commands, let’s test them all out by creating our first-ever IaC-powered GCP virtual machine.

In your code editor, type the following code:

provider "google" {
  project = "your-gcp-project-id"  # Replace with your GCP Project ID
  region  = "us-central1"          
  zone    = "us-central1-a"        
}

This code highlights the cloud provider we’re using to generate the cloud resources we need. In our case, it’s the Google cloud program. The name assigned to it is just “google”. Other cloud providers like AWS and Azure are “aws” and “azure” respectively.

The second line identifies the GCP subscription identifier, which is unique to each GCP account (and helps facilitate accurate integration). You should use yours in the space provided.

You’ll also need to include a suitable resource region and resource availability zone. This serves as the physical base for the virtual machine we’ll create so we can run it. In this scenario, I chose the USA central region and 1-a availability zone, respectively. You can read more here about cloud regions and availability zones.

resource "google_compute_instance" "vm_instance" {
  name         = "example-vm"      
  machine_type = "e2-medium"          

  boot_disk {
    initialize_params {
      image = "debian-cloud/debian-11" 
    }
  }

The code snippet above specifies the exact compute resource that’ll be orchestrated, which in our case is a virtual machine instance coded as “vm_instance”. 'example-vm’ is the name we want to assign to the virtual machine we will be creating for this project. It is important to note that the virtual machine name must be unique too. The type of the virtual machine we opted for was the E2 (General purpose)-medium type VM. You can get more information on Virtual machine types here.

Going further, we also specify the expected booted Operating system (“boot_disk”) which is an image of the Debian Linux Operating system version 11 in my case.

  network_interface {
    network = "default"  # Attach to the default VPC network
    access_config {

    }
  }

output "instance_ip" {
  value = google_compute_instance.vm_instance.network_interface[0].access_config[0].nat_ip
}

To complete the creation of our virtual machine, we need to set up a Virtual Network to allow remote access to the VM. The network interface block connects the virtual machine to the default VPC (Virtual Private Cloud) network provided by GCP. We won’t be able to interface with our virtual machine without the VPC network. The output block also displays the default access IP address in the terminal, which we can use to connect to the virtual machine.

Here is the final expected code:

provider "google" {
  project = "your-gcp-project-id"  # Replace with your GCP Project ID
  region  = "us-central1"          
  zone    = "us-central1-a"       
}

resource "google_compute_instance" "vm_instance" {
  name         = "example-vm"         
  machine_type = "e2-medium"          

  boot_disk {
    initialize_params {
      image = "debian-cloud/debian-11"  
    }
  }

  network_interface {
    network = "default"  # Attach to the default VPC network
    access_config {

    }
  }

output "instance_ip" {
  value = google_compute_instance.vm_instance.network_interface[0].access_config[0].nat_ip
}

Going on from there, we’ll now be executing this code using the commands highlighted in the image below:

The command terraform -v confirms that Terraform has been successfully installed on the terminal. The expected output will be the version of the Terraform tool installed.

The next command executed is the terraform init function which initializes a communication with the cloud service provider, which in our case is GCP. All needed dependencies are also installed.

The terraform fmt command is also run to ensure adequate code formatting and indentation. Then the terraform plan command is sequentially executed.

From the image above, you can see the steps Terraform intends to use to generate the expected Virtual machine.

On successful execution of Terraform plan, we will then execute the terraform apply function to execute the steps outlined by Terraform plan.

This will generate a prompt asking for a confirmation of the Terraform execution as shown above. Typing “Yes” will allow the operation to proceed smoothly.

On successful execution, a success message will be displayed as shown above. With that, we have created our Cloud infrastructure with just code. The terraform destroy command can then be called to remove the created Virtual machines.

Conclusion

In this article, you’ve learned the basics about infrastructure as code. We discussed Terraform, its benefits, and some of its key features and commands. We also illustrated its use in a demo project.

To further enhance your knowledge, you can consult Terraform‘s documentation for more code examples. I would also recommend utilizing your newly gained knowledge to automate a project with real-life uses.

Feel free to message me with any comments or questions. You can also check out my other articles here. Till next time, keep on coding!

When you're working with AWS, among the tools at the forefront of utilizing IaC are AWS CloudFormation, AWS Cloud Development Kit (CDK), and HashiCorp’s Terraform.

Each of these IaC tools offers unique features and approaches to infrastructure management. This makes them suitable for different scenarios and preferences, and they can help you automate and standardize your or your team's cloud resource deployments.

This article will provide a high-level comparison of these three tools, focusing on their capabilities, abstraction levels, and practical use cases. You'll explore how these tools enable you to programmatically create and manage complex cloud infrastructures.

Specifically, the focus will be on deploying a three-tier architecture networking infrastructure. It'll include deploying a Virtual Private Cloud (VPC) configured with multiple subnets, route tables, an internet gateway, and NAT gateways to showcase the unique capabilities and syntax of each IaC tool.

By the end of this article, you'll gain a thorough understanding of the functionalities of these tools so you can make an informed decision when selecting one to build resilient, scalable and efficiently managed cloud infrastructures.

Without further ado, let’s get this party started!

What We'll Cover:

1. What is Infrastructure as Code (IaC)?
2. Prerequisites
3. Use Case Scenario
4. IaC Tool Code Examples
5. Analysis and Comparison
6. Why Choose One Over the Other?

What is Infrastructure as Code (IaC)?

Infrastructure as Code is a key DevOps principle that involves managing and provisioning infrastructure resources by defining it as code in configuration files, instead of using manual processes and settings.

If you want to learn more about IaC basics, here's a helpful guide to get you started.

Now let's learn a bit more about the three tools we'll be comparing in this overview.

What Does AWS CloudFormation Do?

AWS CloudFormation uses YAML or JSON to describe as well as automatically and securely provision infrastructure resources needed for your applications – across all regions and accounts in your AWS cloud environment.

What Does AWS Cloud Development Kit (CDK) Do?

AWS Cloud Development Kit is a software development framework specifically used for defining cloud infrastructure in code. It ultimately provisions resources through AWS CloudFormation.

AWS CDK uses familiar programming languages like TypeScript, JavaScript, Python, Java, and others to define reusable cloud components known as constructs. These are then shared and used to create complex and scalable cloud architectures.

What's a construct?

In the context of AWS CDK, a construct represents a cloud component that encapsulates certain functionality and configuration in a reusable form.

What Does Terraform Do?

Terraform is a multi-tenant tool created by HashiCorp that allows you to define both low-level and high-level components of your cloud infrastructure using a declarative configuration language.

It is cloud-agnostic and is capable of managing multi-provider setups within a single configuration.

What does cloud-agnostic mean?

Cloud-agnostic refers to the ability of a tool or service to operate across different cloud providers without significant changes to its operational procedures or architecture.

Alright, now that you understand the tools we'll be discussing, let's dive in.

Prerequisites

• AWS Account with an IAM User with admin permissions
• Basic knowledge and use of AWS CloudFormation, AWS CDK, and Terraform
• Basic understanding of YAML, Python, and the HashiCorp Configuration Language
• Experience with an Interactive Development Environment (IDE)

Use Case Scenario

You’re a Cloud Network Engineer at REXTECH Corp, a startup on the verge of launching a new online service that offers digital content streaming. As the service is expected to attract a substantial user base right from the start, you need to deploy a highly scalable, reliable, and secure cloud infrastructure that can handle peak traffic and provide continuous availability.

Your manager has mandated a cloud network solution that not only meets these performance requirements but also allows for rapid scaling and efficient management.

In response to this, you are tasked with automating the deployment of a three-tier architecture networking infrastructure. It needs to have a Virtual Private Cloud (VPC) that includes multiple subnets across multiple Availability Zones (AZs), NAT gateways, and route tables to ensure resiliency and optimal configuration.

With the need for agility and maintainability in your infrastructure, you decide to evaluate and choose between AWS CloudFormation, AWS CDK, and Terraform for this project.

Before you evaluate each tool's application to the scenario, let’s break down the deployment resource components.

This deployment involves configuring a VPC with two public subnets for frontend facing web servers, two private subnets for servers in the application tier, and another two private subnets to host a multi-AZ database. All subnets will be deployed across multiple AZs and will include the connectivity configurations between components through route tables and an internet gateway.

Also, two NAT gateways in the public subnets will ensure that the resources in the private subnets of the application tier can securely access the internet for updates and inter-service communication without direct exposure to the outside world.

Now, let’s learn how you can automate the creation of this solution using all three IaC tools: AWS CloudFormation, AWS CDK and Terraform.

IaC Tool Code Examples

AWS CloudFormation Example

AWS CloudFormation allows you to define your desired infrastructure using a declarative JSON or YAML configuration file. But you must define the interdependencies and connections between resources using intrinsic functions like !Ref, referencing other resources or !GetAtt, to help select availability zones dynamically.

Below is how you define the three-tier networking solution using AWS CloudFormation:

AWSTemplateFormatVersion: '2010-09-09'
Resources:
  MyVPC:
    Type: 'AWS::EC2::VPC'
    Properties:
      CidrBlock: '10.0.0.0/16'
      EnableDnsSupport: true
      EnableDnsHostnames: true

  InternetGateway:
    Type: 'AWS::EC2::InternetGateway'

  AttachGateway:
    Type: 'AWS::EC2::VPCGatewayAttachment'
    Properties:
      VpcId: !Ref MyVPC
      InternetGatewayId: !Ref InternetGateway

  PublicSubnetOne:
    Type: 'AWS::EC2::Subnet'
    Properties:
      VpcId: !Ref MyVPC
      CidrBlock: '10.0.1.0/24'
      AvailabilityZone: !Select [0, !GetAZs '']
      MapPublicIpOnLaunch: true

  PublicSubnetTwo:
    Type: 'AWS::EC2::Subnet'
    Properties:
      VpcId: !Ref MyVPC
      CidrBlock: '10.0.2.0/24'
      AvailabilityZone: !Select [1, !GetAZs '']
      MapPublicIpOnLaunch: true

  PrivateSubnetAppOne:
    Type: 'AWS::EC2::Subnet'
    Properties:
      VpcId: !Ref MyVPC
      CidrBlock: '10.0.3.0/24'
      AvailabilityZone: !Select [0, !GetAZs '']

  PrivateSubnetAppTwo:
    Type: 'AWS::EC2::Subnet'
    Properties:
      VpcId: !Ref MyVPC
      CidrBlock: '10.0.4.0/24'
      AvailabilityZone: !Select [1, !GetAZs '']

  PrivateSubnetDBOne:
    Type: 'AWS::EC2::Subnet'
    Properties:
      VpcId: !Ref MyVPC
      CidrBlock: '10.0.5.0/24'
      AvailabilityZone: !Select [0, !GetAZs '']

  PrivateSubnetDBTwo:
    Type: 'AWS::EC2::Subnet'
    Properties:
      VpcId: !Ref MyVPC
      CidrBlock: '10.0.6.0/24'
      AvailabilityZone: !Select [1, !GetAZs '']

  EIPOne:
    Type: 'AWS::EC2::EIP'

  EIPTwo:
    Type: 'AWS::EC2::EIP'

  NATGatewayOne:
    Type: 'AWS::EC2::NatGateway'
    Properties:
      AllocationId: !GetAtt 'EIPOne.AllocationId'
      SubnetId: !Ref PublicSubnetOne

  NATGatewayTwo:
    Type: 'AWS::EC2::NatGateway'
    Properties:
      AllocationId: !GetAtt 'EIPTwo.AllocationId'
      SubnetId: !Ref PublicSubnetTwo

  PublicRouteTable:
    Type: 'AWS::EC2::RouteTable'
    Properties:
      VpcId: !Ref MyVPC

  PublicRoute:
    Type: 'AWS::EC2::Route'
    Properties:
      RouteTableId: !Ref PublicRouteTable
      DestinationCidrBlock: '0.0.0.0/0'
      GatewayId: !Ref InternetGateway

  PublicSubnetOneRouteTableAssociation:
    Type: 'AWS::EC2::SubnetRouteTableAssociation'
    Properties:
      SubnetId: !Ref PublicSubnetOne
      RouteTableId: !Ref PublicRouteTable

  PublicSubnetTwoRouteTableAssociation:
    Type: 'AWS::EC2::SubnetRouteTableAssociation'
    Properties:
      SubnetId: !Ref PublicSubnetTwo
      RouteTableId: !Ref PublicRouteTable

  PrivateAppRouteTableOne:
    Type: 'AWS::EC2::RouteTable'
    Properties:
      VpcId: !Ref MyVPC

  PrivateAppRouteTableTwo:
    Type: 'AWS::EC2::RouteTable'
    Properties:
      VpcId: !Ref MyVPC

  PrivateAppRouteOne:
    Type: 'AWS::EC2::Route'
    Properties:
      RouteTableId: !Ref PrivateAppRouteTableOne
      DestinationCidrBlock: '0.0.0.0/0'
      NatGatewayId: !Ref NATGatewayOne

  PrivateAppRouteTwo:
    Type: 'AWS::EC2::Route'
    Properties:
      RouteTableId: !Ref PrivateAppRouteTableTwo
      DestinationCidrBlock: '0.0.0.0/0'
      NatGatewayId: !Ref NATGatewayTwo

  PrivateSubnetAppOneRouteTableAssociation:
    Type: 'AWS::EC2::SubnetRouteTableAssociation'
    Properties:
      SubnetId: !Ref PrivateSubnetAppOne
      RouteTableId: !Ref PrivateAppRouteTableOne

  PrivateSubnetAppTwoRouteTableAssociation:
    Type: 'AWS::EC2::SubnetRouteTableAssociation'
    Properties:
      SubnetId: !Ref PrivateSubnetAppTwo
      RouteTableId: !Ref PrivateAppRouteTableTwo

  PrivateDBRouteTable:
    Type: 'AWS::EC2::RouteTable'
    Properties:
      VpcId: !Ref MyVPC

  PrivateSubnetDBOneRouteTableAssociation:
    Type: 'AWS::EC2::SubnetRouteTableAssociation'
    Properties:
      SubnetId: !Ref PrivateSubnetDBOne
      RouteTableId: !Ref PrivateDBRouteTable

  PrivateSubnetDBTwoRouteTableAssociation:
    Type: 'AWS::EC2::SubnetRouteTableAssociation'
    Properties:
      SubnetId: !Ref PrivateSubnetDBTwo
      RouteTableId: !Ref PrivateDBRouteTable

This YAML script creates the intended VPC, two public subnets, an internet gateway, two elastic IP addresses, and two NAT gateways. Here, you also leverage AWS CloudFormation’s capabilities to link resources and manage dependencies explicitly.

AWS CDK Example

When using the AWS CDK, you define cloud resources in an imperative programming style. It's offers an abstraction over AWS CloudFormation but offers more flexibility by using constructs, which can encapsulate multiple resources into a single logical unit. It also allows you to use of loops, conditionals, and other programming logic to dynamically generate your resources.

When configuration resources like subnets, it is simplified by grouping them under subnet_configuration in a VPC construct. This automatically handles subnet associations for you.

Below, you'll use the Python programming language to define the three-tier solution with AWS CDK:

from constructs import Construct
from aws_cdk import (
    Stack,
    aws_ec2 as ec2
)

class MyVpcStack(Stack):
    def __init__(self, scope: Construct, id: str, **kwargs):
        super().__init__(scope, id, **kwargs)

        # Create a VPC with specific configurations
        vpc = ec2.Vpc(self, "MyVpc",
                      ip_addresses=ec2.IpAddresses.cidr("10.0.0.0/16"),
                      max_azs=2,
                      subnet_configuration=[
                          ec2.SubnetConfiguration(
                              name="PublicSubnet",
                              subnet_type=ec2.SubnetType.PUBLIC,
                              cidr_mask=24
                          ),
                          ec2.SubnetConfiguration(
                              subnet_type=ec2.SubnetType.PRIVATE_WITH_EGRESS,
                              name="PrivateSubnet1",
                              cidr_mask=24
                          ),
                          ec2.SubnetConfiguration(
                              subnet_type=ec2.SubnetType.PRIVATE_ISOLATED,
                              name="PrivateSubnet2",
                              cidr_mask=24
                          )
                      ],
                      nat_gateways=2,  # Number of NAT Gateways
                      )

As you can see, this AWS CDK Python script is more concise and allows you to work with a very familiar high-level programming language, which provides powerful abstractions and leverages use of constructs.

Terraform Example

Terraform’s approach involves defining infrastructure using a declarative configuration language. But it differs from AWS CloudFormation in its approach to managing state and dependencies. It also allows more controlled resource creation, updating, and destruction with constructs like resource, provider and variable.

Here’s how you define the same solution with Terraform:

provider "aws" {
  region = "us-east-1"
}

resource "aws_vpc" "my_vpc" {
  cidr_block           = "10.0.0.0/16"
  enable_dns_support   = true
  enable_dns_hostnames = true
}

# Public Subnets
resource "aws_subnet" "public_subnet_one" {
  vpc_id                  = aws_vpc.my_vpc.id
  cidr_block              = "10.0.1.0/24"
  map_public_ip_on_launch = true
  availability_zone       = "us-east-1a"
}

resource "aws_subnet" "public_subnet_two" {
  vpc_id                  = aws_vpc.my_vpc.id
  cidr_block              = "10.0.2.0/24"
  map_public_ip_on_launch = true
  availability_zone       = "us-east-1b"
}

# Private Subnets for Application Tier
resource "aws_subnet" "private_app_subnet_one" {
  vpc_id            = aws_vpc.my_vpc.id
  cidr_block        = "10.0.3.0/24"
  availability_zone = "us-east-1a"
}

resource "aws_subnet" "private_app_subnet_two" {
  vpc_id            = aws_vpc.my_vpc.id
  cidr_block        = "10.0.4.0/24"
  availability_zone = "us-east-1b"
}

# Private Subnets for Database Tier
resource "aws_subnet" "private_db_subnet_one" {
  vpc_id            = aws_vpc.my_vpc.id
  cidr_block        = "10.0.5.0/24"
  availability_zone = "us-east-1a"
}

resource "aws_subnet" "private_db_subnet_two" {
  vpc_id            = aws_vpc.my_vpc.id
  cidr_block        = "10.0.6.0/24"
  availability_zone = "us-east-1b"
}

resource "aws_internet_gateway" "igw" {
  vpc_id = aws_vpc.my_vpc.id
}

resource "aws_nat_gateway" "nat_gateway_one" {
  allocation_id = aws_eip.nat_one.id
  subnet_id     = aws_subnet.public_subnet_one.id
}

resource "aws_nat_gateway" "nat_gateway_two" {
  allocation_id = aws_eip.nat_two.id
  subnet_id     = aws_subnet.public_subnet_two.id
}

resource "aws_eip" "nat_one" {
  domain = "vpc"
}

resource "aws_eip" "nat_two" {
  domain = "vpc"
}

# Public Route Table
resource "aws_route_table" "public_rt" {
  vpc_id = aws_vpc.my_vpc.id

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.igw.id
  }
}

# Private Route Tables for Application Tier
resource "aws_route_table" "private_app_rt_one" {
  vpc_id = aws_vpc.my_vpc.id

  route {
    cidr_block     = "0.0.0.0/0"
    nat_gateway_id = aws_nat_gateway.nat_gateway_one.id
  }
}

resource "aws_route_table" "private_app_rt_two" {
  vpc_id = aws_vpc.my_vpc.id

  route {
    cidr_block     = "0.0.0.0/0"
    nat_gateway_id = aws_nat_gateway.nat_gateway_two.id
  }
}

# Private Route Tables for Database Tier
resource "aws_route_table" "private_db_rt" {
  vpc_id = aws_vpc.my_vpc.id
}

# Route Table Associations
resource "aws_route_table_association" "public_subnet_one_association" {
  subnet_id      = aws_subnet.public_subnet_one.id
  route_table_id = aws_route_table.public_rt.id
}

resource "aws_route_table_association" "public_subnet_two_association" {
  subnet_id      = aws_subnet.public_subnet_two.id
  route_table_id = aws_route_table.public_rt.id
}

resource "aws_route_table_association" "private_app_subnet_one_association" {
  subnet_id      = aws_subnet.private_app_subnet_one.id
  route_table_id = aws_route_table.private_app_rt_one.id
}

resource "aws_route_table_association" "private_app_subnet_two_association" {
  subnet_id      = aws_subnet.private_app_subnet_two.id
  route_table_id = aws_route_table.private_app_rt_two.id
}

resource "aws_route_table_association" "private_db_subnet_one_association" {
  subnet_id      = aws_subnet.private_db_subnet_one.id
  route_table_id = aws_route_table.private_db_rt.id
}

resource "aws_route_table_association" "private_db_subnet_two_association" {
  subnet_id      = aws_subnet.private_db_subnet_two.id
  route_table_id = aws_route_table.private_db_rt.id
}

This script shows how Terraform allows for a modular approach to infrastructure as code, with explicit definitions and dependency management with syntax that is relatively easy to read and write.

Analysis and Comparison

When choosing between AWS CloudFormation, AWS CDK, and Terraform for managing cloud infrastructure, you have consider a number of factors. But in this article, will specifically focus on the ease of use, flexibility, scalability, language support, and the ability to handle complex environments.

Ease of Use and Learning Curve

AWS CloudFormation offers a JSON or YAML-based template format. This is straightforward for defining the infrastructure, but can become complex as infrastructure grows. It requires an understanding of specific syntax and AWS resource definitions, which might have a steeper learning curve for those not familiar with JSON or YAML.

AWS CDK uses familiar programming languages like Python, JavaScript, TypeScript and Java. This can make it more accessible for developers already familiar with these languages.

Also, since AWS CDK allows for defining infrastructure through code, it provides more intuitive logic, conditions, and loops, and it abstracts much of the boilerplate needed in AWS CloudFormation. This simplifies the development process.

Terraform uses its own domain-specific language, HashiCorp Configuration Language (HCL), which is designed to be easily readable and writable by humans. While it can be easy to learn, you'll need to be familiar with another new language. However, its declarative nature allows clear definitions of what the infrastructure should look like without the need of specifying how to achieve it.

Flexibility and Cloud Provider Support

AWS CloudFormation is tightly integrated with AWS and is updated in tandem with AWS services. But it’s inherently limited to AWS, making it less suitable for the possibilities for hybrid or multi-cloud environments.

AWS CDK also primarily targets AWS services but supports the use of AWS CloudFormation custom resources to manage resources outside of AWS. Still, it doesn’t naturally lend itself to managing multi-cloud resources as directly as Terraform.

Terraform is designed to be cloud-agnostic, supporting multiple providers including AWS, Microsoft Azure, Google Cloud Platform and others. This makes it an ideal choice for complex deployments spanning more than one cloud provider.

Scalability and Maintainability

AWS CloudFormation templates can become unwieldy and difficult to manage as projects scale. But AWS provides nested stacks as a solution to manage large infrastructures but even with this capability, managing many stacks can become cumbersome track.

AWS CDK provides high-level abstractions and modular constructs, making it easier to manage and scale large infrastructures by breaking them down into smaller, reusable components.

Terraform excels in managing large-scale infrastructures due to its modular approach. By using Terraform modules, you can reuse configurations and ensure consistency across deployments.

Community Support and Ecosystem

AWS CloudFormation has great adoption and support from AWS with a large user base, but its community contributions are limited to sharing templates.

AWS CDK is open-source and has a growing community, especially among developers preferring to use general-purpose programming languages for infrastructure management. The ecosystem includes a rich set of high-level constructs developed by both AWS and the community.

Terraform benefits from strong community engagement and a vast ecosystem of providers and modules shared publicly in the Terraform Registry. Its wide adoption across different platforms also helps foster a large and active community.

Code Length and Complexity

AWS CloudFormation scripts tend to be verbose, requiring detailed specifications of every property. This can lead to lengthy and complex templates for larger infrastructures.

AWS CDK scripts are typically shorter and less complex due to the use of programming constructs that abstract away much of the detailed specifications required in AWS CloudFormation.

Terraform configurations strike a balance, being more concise than AWS CloudFormation but typically more verbose than AWS CDK due to its declarative nature, which requires explicit resource and configuration definitions.

Why Choose One Over the Other?

When choosing between AWS CloudFormation, AWS CDK, and Terraform, consider each tools' unique features, operational principles, and your own personal preferences.

Now I'll share recommendations based on this article's information to help you figure out when it’s best to use each of these IaC tools.

• AWS CloudFormation is suitable for when you are looking for stable, native AWS tooling and don’t necessarily need to manage resources outside of AWS. It’s particularly great when compliance with specific AWS configurations is required.
• Choose AWS CDK when you prefer using standard programming languages and enjoy the benefits of object-oriented techniques to create reusable and modular cloud components. It is usually more appealing to developers who want to apply software development best practices to infrastructure provisioning.
• Terraform is the ultimate leader for multi-cloud environments, or if you need a tool that is both powerful and flexible enough to manage complex architectures. It is also the right choice if you anticipate integrating a variety of cloud services and need a unified approach to manage them.

Though these recommendations are based on the special makeup of each of these IaC tools, I advise you to gain some experience with each tool, so you can decide on the one that best aligns with the specific skills and needs of your team and projects.

If you’ve got this far, thanks so much for reading! I hope it was worthwhile to you.

If you want to learn more about me and my story of transitioning from a Pro Athlete to a Cloud Engineer, connect with me here on LinkedIn.

Terraform is a popular Infrastructure as Code (IaC) tool that allows users to define and manage cloud infrastructure in a declarative way. However, like any tool, Terraform can introduce security risks if not used properly.

In this article, we will explore the most common security risks when using Terraform, the threat landscape and attack surface, how it can be exploited, and how users can stay secure by following Terraform security best practices.

What is Terraform and why Should I Use it?

Terraform is an open-source tool developed by HashiCorp that enables users to define and provision infrastructure resources across multiple cloud providers and on-premises environments. It allows organizations to treat infrastructure as code, bringing the benefits of version control, collaboration, and automation to infrastructure management.

By adopting Terraform and embracing the principles of Infrastructure as Code, organizations can achieve several benefits:

• Consistency and repeatability: Infrastructure deployments become consistent and repeatable, eliminating the risk of manual errors and ensuring that the same configuration can be applied across different environments.
• Version control and collaboration: Infrastructure code can be stored in version control systems, enabling teams to collaborate, track changes, and roll back to previous versions if needed.
• Automation and scalability: Infrastructure deployments can be automated, allowing organizations to scale their infrastructure quickly and efficiently based on demand.
• Auditing and compliance: Infrastructure code can be audited and reviewed for compliance with security and regulatory standards, ensuring that best practices are followed.

While Terraform offers a powerful and flexible solution for managing infrastructure, it is essential to address the security considerations associated with using the tool effectively.

Security considerations

As with any tool or technology, Terraform is not immune to security threats and challenges. It is crucial to understand the threat landscape and the potential risks associated with using Terraform.

By identifying these risks, organizations can implement appropriate security measures to mitigate them effectively.

Configuration errors

One of the primary security risks associated with Terraform is misconfigurations in the infrastructure code.

Misconfigurations can lead to vulnerabilities and expose critical resources to unauthorized access or compromise. Common misconfigurations include weak access controls, open network ports, and incorrect permission settings on cloud resources.

Secrets management

Terraform relies on access keys and secret keys to authenticate with cloud providers and provision resources.

Storing these credentials insecurely can lead to security vulnerabilities such as unauthorized access and data breaches.

Handling access credentials securely is crucial to the overall security posture of Terraform deployments.

State security

Terraform uses state files to keep track of the resources it has created or modified.

These state files contain sensitive information, such as resource IDs, metadata, and secrets as referenced above. Inadequate management of state files can lead to security risks, including unauthorized access and data exposure.

State files can be stored either locally on a machine (suitable only for solo testing) or on a remote backend, such as a cloud storage resource, which should be encrypted and locked down.

Supply chain security

As with any software development process, the supply chain of Terraform modules and associated dependencies can introduce security risks.

Organizations must assess the trustworthiness and security of the modules they use and ensure they are regularly updated to address any vulnerabilities.

Permissions management

Appropriate permissions and access controls are essential to prevent unauthorized changes to infrastructure resources.

Managing permissions effectively can help reduce the risk of accidental or malicious actions that could compromise the security of the infrastructure.

Recommendations

To stay secure while using Terraform, users should follow these best practices:

1. Execute Terraform programmatically to minimize human error and enforce security policies.
2. Use safe Terraform modules and avoid using untrusted or vulnerable third-party components.
3. Secure the data store when remotely storing state data to prevent unauthorized access (for example, through encryption and restrictive access permissions).
4. Avoid storing secrets in state files; instead, use secret management solutions like AWS Secrets Manager, Azure Key Vault, or Google Cloud Secret Manager.
5. Use Terraform security scanners to identify and remediate potential vulnerabilities.
6. Implement centralized security policy and governance within Terraform code to improve visibility and enforce least privilege.
7. Require multi-factor authentication for collaborators to improve security posture.
8. Keep Terraform and all modules up to date.
9. Regularly audit terraform configurations for security vulnerabilities and misconfigurations, and build in appropriate automated tooling to detect violations prior to deployment.
10. Conduct regular drift detection to determine if resources deployed in your cloud provider match what should have been deployed in the terraform state file.

Conclusion

Terraform is a powerful tool for managing cloud infrastructure, but it can introduce security risks if not used properly.

By following Terraform security best practices, users can minimize the risk of security incidents and maintain a secure Terraform environment.

It's essential to keep Terraform configurations and infrastructure up-to-date, monitor for security threats, and adapt to the evolving threat landscape. By doing so, users can ensure that their cloud infrastructure is secure and compliant with industry standards.

The article will cover how infrastructure as code works using an analogy. We'll cover the different infrastructure as code tools available as well as declarative vs imperative code

I'll also introduce you to Terraform, which is an open source infrastructure as code tool you can use to create infrastructure across multiple cloud providers like AWS, GCP, Azure and others.

Infrastructure as Code in Practice

Imagine you are trying to create a three-tiered web application on AWS as you can see in the image below:

Three tiered web application example

The presentation tier is responsible for presenting the user interface to the user. It includes the user interface components such as HTML, CSS, and JavaScript running on EC2 instances.

The logic tier is responsible for processing user requests and generating responses, by communicating with the database layer to retrieve or store data. This is also deployed on EC2 instances

The database tier is responsible for storing and managing the application's data and allows access to its data through the logic tier. The database runs on AWS RDS.

Each of the instances are in an autoscaling group with a load balancer in front of it (except for the database tier).

If you want to create this infrastructure through the AWS console, you would have to manually click through various screens to spin up the infrastructure. This is fine if it is a one time activity.

But if you need to repeat this across different environments like development and test, or need to add additional infrastructure like caches, queues, firewall rules, IAM or SSL certificates, then it becomes increasingly more complex to manage through the AWS console.

Managing complex infrastructure through the console also introduces the possibility of human error.

Infrastructure as code expresses your desired infrastructure in the language of code. This brings all the benefits of code to managing your infrastructure like:

1. Version Control – allows you to store the history of your infrastructure and revert to a previous version if needed.

2. Faster & safer deployments – can recreate infrastructure in new environments quickly and with less errors since every part of the infrastructure is clearly defined in the code.

3. Documentation – your current infrastructure state is documented and kept up to date automatically whenever you make a change. This keeps your infrastructure documentation detailed and accurate, compared to having the infrastructure written in a document or on a confluence page that may not be updated whenever there is a change.

How Infrastructure as Code Works – Explained with an Analogy

Infrastructure as code allows you to create a detailed blueprint of your infrastructure. This blueprint gives instructions to your cloud provider about the infrastructure you want created.

This is similar to how an architecture blueprint works. It outlines the layout, dimensions, materials, and various components of the structure. The blueprint serves as a reference for architects and engineers to understand the desired construction.

how an architectural blueprint is analogous to infrastructure as code

The blueprint leaves little room for error. It will be interpreted in the same way by any architect or engineer. If you wanted to build exact copies of this house, all you need is the architecture blueprint.

Infrastructure as code, at a basic level, works in the same way as an architecture blueprint. It details the infrastructure you want to create as code in a number of different possible languages (JSON, YAML, HCL, Python, Ruby, JavaScript, and so on), instructing the cloud provider to create your infrastructure exactly as specified.

Declarative & Imperative Infrastructure as Code Tools

There are many IaC options to choose from, and all the major cloud providers have their own dedicated tools:

• AWS has CloudFormation

• GCP has Deployment Manager

• Azure has Resource manager

One limitation of these cloud provider-specific tools is that they can only create infrastructure in their respective clouds. So CloudFormation only works in AWS and Deployment Manager only works in GCP. IaC using these providers is usually written in JSON or YAML format.

Terraform, on the other hand, is open source and you can use it to create infrastructure across all the major cloud providers. It uses HCL (HashiCorp Configuration Language).

Infrastructure as code can also be written using popular languages like Python and JavaScript.

These scripting/programming languages lie on a spectrum of declarative and imperative code as shown below.

A spectrum of declarative & imperative languages and where Terraform HCL fits

The main difference between an imperative and declarative language is that imperative languages explicitly define the control flow. This is simply the order in which instructions are executed in a program. Control flow determines the path the program takes and how it responds to different conditions or events.

In imperative languages, control flow is explicitly defined using control structures such as loops, conditionals, and function calls. Imperative languages give you more flexibility in configuring your infrastructure. This is not necessarily a positive, as more flexibility means more opportunity to introduce errors into your infrastructure.

A declarative language focuses on describing the desired result without giving specific instructions on how to achieve it.

An illustration demonstrating the difference between declarative and imperative languages

An example JSON is shown below, used in AWS CloudFormation to create an EC2 instance:

"Type": "AWS::EC2::Instance",
      "Properties": {
        "ImageId": "ami-0123456789",
        "InstanceType": "t2.micro",
        "KeyName": "my-key-pair",
        "SecurityGroupIds": ["sg-0123456789"],
        "SubnetId": "subnet-0123456789",
        "Tags": [
          {
            "Key": "Name",
            "Value": "MyEC2Instance"
          }
        ]
      }

A declarative language like JSON abstracts away the underlying complexity that details how the EC2 instance will be created. All it cares about is the end state.

Terraform HCL is closer to the declarative end of the spectrum. Terraform allows you to describe the desired infrastructure's final state without specifying the exact steps to get there. Terraform internally manages the execution order, resource dependencies, and handles the infrastructure changes based on the desired configuration.

But Terraform does have support for some imperative features like variables and expressions, allowing dynamic behaviour based on inputs. So, it is not a completely declarative language like JSON.

How Terraform Works

There are two fundamental concepts that serve as a foundation for understanding Terraform:

1. The configuration file – this describes the desired infrastructure

2. The state file – this describes the current infrastructure as it exists in the real world

Terraform’s job is to create, modify or delete infrastructure as needed so that the desired infrastructure configuration is met. It does this by executing the necessary API calls to your cloud provider(s) to create, modify, or destroy the resources as specified.

Once the infrastructure has been created/modified/destroyed to match the configuration file, the state file is updated to reflect the current infrastructure.

The terraform plan command creates an execution plan, which lets you preview the changes that Terraform plans to make to your infrastructure.

By default, when Terraform creates a plan, it compares the desired configuration as described in the configuration file, with the current configuration as described in the state file. Terraform then proposes a list of changes needed what will ensure that the current configuration matches the desired configuration.

If you then run the terraform apply command, terraform will modify the real world infrastructure so that it matches the desired configuration, and updates the state file to show the new infrastructure configuration.

At a high level, this is what terraform does:

What happens when you run the terraform apply command

Let’s bring back the architectural blueprint analogy.

The configuration file is like the architectural blueprint. It details the infrastructure that needs to be built, that is the desired construction. The real world infrastructure is the existing construction in the physical world and the state file is a representation of what currently exists – the current blueprint. The engineers work to ensure that the existing construction matches the architecture blueprint.

In this analogy, engineers do the work of Terraform in ensuring that the existing construction matches the architecture blueprint. You don’t need to specify the details of how to build the house, you just need to specify what you want built and the engineers handle the rest.

An architectural analogy to running terraform apply

If you want to learn more about how Terraform works and how you can use it in your projects, you can check out this free course on freeCodeCamp's YouTube channel.

Bringing it Together

Infrastructure as code (IaC) is a great way of managing complex infrastructure configuration in the form of code. This naturally brings all the advantages of code to your infrastructure like version control, faster and safer infrastructure deployments across different environments and up to date documentation of your infrastructure.

Terraform is an open source IaC tool that allows you to work with multiple cloud providers to spin up infrastructure as defined in your configuration files.

Terraform HCL is a declarative language that allows you to describe your desired infrastructure configuration. All you have to do is specify what you want created and terraform handles the creation on your behalf by making API calls to your chosen cloud provider(s).

But managing your cloud infrastructure manually can be exhausting. You might start wondering, "how are the enterprise companies doing it?"

Well, they automate their cloud infrastructure management via code. And that's what AWS CloudFormation offers – a way to manage your cloud infrastructure as code.

In this tutorial, we'll explore the basics of AWS CloudFormation and how it can help you automate your cloud infrastructure management. Let's dive into the world of infrastructure as code.

What is CloudFormation?

AWS CloudFormation is a service that helps you automate creating and managing your cloud resources.

Imagine you're building a house, and you want to ensure everything is in the right place – the walls, the roof, the doors, and the windows. Before you build, you would create a blueprint for your house, and specify exactly what you want and where you want it.

Similarly, CloudFormation allows you to create a blueprint for your cloud infrastructure. You can specify what resources you want to create (for example EC2 servers, databases, storage, and so on) and how they should be configured. CloudFormation takes care of creating and managing those resources for you automatically.

CloudFormation can be helpful in many cases. I'll list a few of them here:

1. Managing the infrastructure changes in multiple environments (Development, Staging, Production)
2. Re-creating the same infrastructure in a different region / account
3. Re-creating a resource that's been accidentally deleted with the exact configuration in seconds (not manually, as you have all configurations in your code)

The best part is that CloudFormation makes updating your infrastructure super simple and automatic. If you want to add a new resource, change a configuration, or delete a resource, you can update your blueprint, and CloudFormation will handle the changes for you.

How CloudFormation Works

You may wonder how CloudFormation works. It's simple: we'll upload our CloudFormation templates in Amazon S3 (behind the scenes) which will be pulled by CloudFormation.

One important point to note is we cannot edit the template once uploaded. We'd need to re-upload the updated template to AWS which CloudFormation compares with the existing infrastructure and makes the necessary changes.

One awesome feature is that you can delete all resources created by CloudFormation in one click by deleting the stack. The CloudFormation stack is nothing but a collection of AWS resources that you can manage as a single unit.

You define a stack by creating a CloudFormation template that describes the resources you want to create, and run the template to create the stack.

How to Deploy CloudFormation Templates

We can deploy the CloudFormation template in two ways. One is using CloudFormation Designer and the other is writing the code in a YAML/JSON file.

If you're not familiar with YAML/JSON then you can go for CloudFormation Designer. This is what I'd recommended for those who don't know how to code. It allows you to create and edit templates graphically, making it easier to visualize your infrastructure and simplify the template creation process. But in this tutorial we'll be writing YAML code to deploy our app.

How to Create a CloudFormation Template to Deploy a NodeJS App

You can create a CloudFormation Template using either a YAML or JSON file – but we're going to use a YAML file in the tutorial.

In this template, we'll be creating an EC2 instance, we'll configure a Security Group for EC2, and add a script to deploy a simple NodeJS app.

CloudFormation Template to Create an EC2 Instance

There are over 224 types of resources, but we need to create an EC2 resource. Resources represent the different AWS Components that will be created and configured. We'll define the Resource type identifiers in the below format:

AWS::aws-product-name::data-type-name

The resource format for the EC2 instance is Aws::EC2::Instance. To learn more about AWS resources and syntax, checkout the AWS official documentation and play with it. Look at the EC2 documentation to understand the declaration of EC2 instance. Both JSON and YAML syntax is available but we'll stick with YAML for this tutorial.

There are a lot of properties available to customize the creation of our EC2 instances. To make things simple, we'll be configuring AvailabilityZone, ImageId, and InstanceType which are basic properties needed to create an EC2 instance.

Resources:
  SampleNodejsDeploy:
    Type: AWS::EC2::Instance
    Properties:
      AvailabilityZone: us-east-1a
      ImageId: ami-a4c7edb2
      InstanceType: t2.micro

Here SampleNodejsDeploy refers to the name of the resource we'll be creating. You can name your resource as your wish.

Let's see the process to deploy the NodeJS app.

How to Deploy a NodeJS App using a CloudFormation Template

We're going to deploy the NodeJS app using the UserData property in the EC2 resource.

If you don't know about EC2 user data, it is a feature of AWS EC2 which allows us to pass information during the launch of the EC2 instance. You can use it to perform custom actions, such as installing software and executing the script.

Let's write the bash script to deploy the NodeJS app and attach it to the user data.

Here is the simple script to deploy the NodeJS app:

#!/bin/bash 
set -e
curl -sL https://deb.nodesource.com/setup_16.x | bash -
sudo apt install nodejs
node -v
npm -v
curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -
echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list
sudo apt update && sudo apt install yarn
yarn --version
sudo -i -u ubuntu bash << EOF
set -e
cd /home/ubuntu
sudo npm install -g pm2
git clone https://github.com/5minslearn/node_with_docker.git
cd node_with_docker
yarn install 
pm2 start yarn --time --interpreter bash --name sample_node -- start -p 8000
EOF

The above script installs NodeJS, Yarn, and PM2. It clones a NodeJS project from Git, installs the dependencies, and starts the app with PM2.

Our next step is to attach this script to the CloudFormation template.

How to Attach UserData to the CloudFormation Template

Resources:
  SampleNodejsDeploy:
    Type: AWS::EC2::Instance
    Properties:
      InstanceType: t2.micro
      ImageId: ami-014d05e6b24240371
      UserData: 
        Fn::Base64:
          |
          #!/bin/bash 
          set -e
          curl -sL https://deb.nodesource.com/setup_16.x | bash -
          sudo apt install nodejs
          node -v
          npm -v
          curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -
          echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list
          sudo apt update && sudo apt install yarn
          yarn --version
          sudo -i -u ubuntu bash << EOF
          set -e
          cd /home/ubuntu
          sudo npm install -g pm2
          git clone https://github.com/5minslearn/node_with_docker.git
          cd node_with_docker
          yarn install 
          pm2 start yarn --time --interpreter bash --name sample_node -- start -p 8000
          EOF

You'll notice that the UserData property is added to the EC2 block. Fn::Base64 is a function in AWS CloudFormation that allows users to encode a string to base64 format. This function can be used to pass sensitive information, such as credentials, to AWS resources in a secure manner. Since EC2 user data is not encrypted it's always best practice to encode it.

Right below that line, you can see a small vertical bar (|). It is used for multi-line string support as our script is more than 1 line.

Alright. Now we have a script to deploy the NodeJS app. But, we have to remember one super important item. By default NodeJS applications run on port 8000. We should expose port 8000 from EC2. Now we need to create a security group configuration for our EC2 instance.

How to Create a Security Group using a CloudFormation Template

This process is similar to creating an EC2 instance, except we'll replace the type from Instance to SecurityGroup.

SampleNodejsDeploySG:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: for the app nodes that allow ssh, http, 8000
      SecurityGroupIngress:
      - IpProtocol: tcp
        FromPort: '80'
        ToPort: '80'
        CidrIp: 0.0.0.0/0
      - IpProtocol: tcp
        FromPort: '22'
        ToPort: '22'
        CidrIp: 0.0.0.0/0
      - IpProtocol: tcp
        FromPort: '8000'
        ToPort: '8000'
        CidrIp: 0.0.0.0/0

The above code should be pretty self explanatory – we defined a Security group, allowing ports 22 (SSH port), 80 (HTTP port), and 8000 (NodeJS). We named the Resource as SampleNodejsDeploySG.

How to Attach the Security Group to EC2

You may be wondering – "We've created a template for creating a Security group but how will this be linked to the EC2 instance?"

The solution is simple. CloudFormation provides an intrinsic function called !Ref that allows us to reference a resource or parameter within a CloudFormation template.

Resources:
  SampleNodejsDeploy:
    Type: AWS::EC2::Instance
    Properties:
      InstanceType: t2.micro
      ImageId: ami-014d05e6b24240371
      SecurityGroups:
        - !Ref SampleNodejsDeploySG
      UserData: 
        Fn::Base64:
          |
          #!/bin/bash 
          set -e
          curl -sL https://deb.nodesource.com/setup_16.x | bash -
          sudo apt install nodejs
          node -v
          npm -v
          curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -
          echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list
          sudo apt update && sudo apt install yarn
          yarn --version
          sudo -i -u ubuntu bash << EOF
          set -e
          cd /home/ubuntu
          sudo npm install -g pm2
          git clone https://github.com/5minslearn/node_with_docker.git
          cd node_with_docker
          yarn install 
          pm2 start yarn --time --interpreter bash --name sample_node -- start -p 8000
          EOF

  SampleNodejsDeploySG:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: for the app nodes that allow ssh, http 
      SecurityGroupIngress:
      - IpProtocol: tcp
        FromPort: '80'
        ToPort: '80'
        CidrIp: 0.0.0.0/0
      - IpProtocol: tcp
        FromPort: '22'
        ToPort: '22'
        CidrIp: 0.0.0.0/0
      - IpProtocol: tcp
        FromPort: '8000'
        ToPort: '8000'
        CidrIp: 0.0.0.0/0

You can see that the SecurityGroups property is added to the EC2 instance and the created Security Group configuration is linked to the EC2 instance by using the !Ref parameter.

Now we have the CloudFormation template. But we're not yet finished. We're still missing one more thing. Can you figure it out? We created an EC2 instance, and we allowed an SSH port...but to log in using SSH we need to attach a key-value pair, right? Let's do that.

We can attach the key-value pair name directly to the template. For example, let's assume your key-value pair name is 5minslearn you can attach the property KeyName directly to the EC2 resource block like what's shown below or we can pass it in via parameters.

Resources:
  SampleNodejsDeploy:
    Type: AWS::EC2::Instance
    Properties:
      InstanceType: t2.micro
      ImageId: ami-014d05e6b24240371
      KeyName: 5minslearn
      SecurityGroups:
        - !Ref SampleNodejsDeploySG

How to use parameters in the CloudFormation template

We can use parameters to get the name of the key-value pair from the user while creating the stack. Basically, parameters allow us to pass input values into CloudFormation templates at runtime. Let's see how to do that.

Parameters:
  SSHKey:
    Type: AWS::EC2::KeyPair::KeyName
    Description: name of the key pair to ssh into the instance
Resources:
  SampleNodejsDeploy:
    Type: AWS::EC2::Instance
    Properties:
      InstanceType: t2.micro
      ImageId: ami-014d05e6b24240371
      KeyName: !Ref SSHKey
      SecurityGroups:
        - !Ref SampleNodejsDeploySG
      UserData: 
        Fn::Base64:
          |
          #!/bin/bash 
          set -e
          curl -sL https://deb.nodesource.com/setup_16.x | bash -
          sudo apt install nodejs
          node -v
          npm -v
          curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -
          echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list
          sudo apt update && sudo apt install yarn
          yarn --version
          sudo -i -u ubuntu bash << EOF
          set -e
          cd /home/ubuntu
          sudo npm install -g pm2
          git clone https://github.com/5minslearn/node_with_docker.git
          cd node_with_docker
          yarn install 
          pm2 start yarn --time --interpreter bash --name sample_node -- start -p 8000
          EOF

  SampleNodejsDeploySG:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: for the app nodes that allow ssh, http 
      SecurityGroupIngress:
      - IpProtocol: tcp
        FromPort: '80'
        ToPort: '80'
        CidrIp: 0.0.0.0/0
      - IpProtocol: tcp
        FromPort: '22'
        ToPort: '22'
        CidrIp: 0.0.0.0/0
      - IpProtocol: tcp
        FromPort: '8000'
        ToPort: '8000'
        CidrIp: 0.0.0.0/0

In the above template, we added a parameter to get the key pair name and referenced it to KeyName property.

Great! We successfully created a CloudFormation template to create an EC2 instance and security group. In addition to that, we also added a script to deploy the NodeJS app. Now it's time to create a CloudFormation stack.

How to Create a CloudFormation Stack

The first step is to log in to the AWS console and search for CloudFormation in the search bar (see the below screenshot). Click on stacks in the left sidebar to get started with CloudFormation.

CloudFormation Getting Started

Click on the create stack button to create the CloudFormation stack.

Create CloudFormation Stack

As we have our template ready, select "Template is ready" and choose "Upload a template file" in the Template source section, and upload the template file.

Deploying CloudFormation Template

Once you upload the file, the "View in Designer" button will be enabled. Click on it to view your template design.

How to Validate the CloudFormation Template

Validate CloudFormation Template

To validate our template click on the "Tick" icon on top left in the designer. It will validate and show us errors if any. Once the validation is done, click on the "Cloud" icon to the left of "Tick" icon. It will take you to the create stack page.

In the stack details page, enter the stack name and select your key-value pair. If you don't have key-value pair, create one and select it.

Specify CloudFormation stack details

Leave the Configure stack options section as it is, and click continue since we don't need any IAM permissions or advanced options.

Finally, review the page and submit the template. The template will start creating the resources.

CloudFormation stack creating resources

Once that's done, click on the resources tab. You'll be able to see the resources we created (EC2 and Security group resources).

Resources created by CloudFormation Template

Click on the EC2 instance, and you can see that our instance will be up and running. Copy the public IPv4 address.

EC2 instance up and running

Open your browser and hit http://<ip_address>:8000 (In my case it is http://54.176.19.18:8000/). You should be able to see a page similar to the one below:

NodeJS app running

This represents that our NodeJS app is successfully deployed!

Note: EC2 user data will take some time to install dependencies. So for the first time, the page will take long time to load. Just be patient until the site is loaded.

How to Delete the CloudFormation Stack

If you no longer need the stack, you can delete it from the CloudFormation console.

Select the stack you want to delete, click "Delete Stack," and confirm the action. This action will delete all resources created using this stack. In our case, it'll delete both EC2 and Security Group. You don't need to delete the EC2 instance and Security Group individually.

Deleting CloudFormation stack

Conclusion

In this article, we learned about CloudFormation, how it works, and how to create and delete a template stack.

Hope you enjoyed reading this article! If you are stuck at any point feel free to drop your queries to me at my email. I’ll be happy to help you.

If you wish to learn more about AWS, subscribe to my newsletter (https://5minslearn.gogosoon.com/) and follow me on social media.

For AWS cloud development, the built-in choice for infrastructure as code is AWS CloudFormation.

Using IaC, developers can manage a project’s infrastructure efficiently, allowing them to easily configure and maintain changes within a project’s architecture and resources.

There are numerous IaC tools available such as Ansible, Puppet, Chef, and Terraform.

But for this guide, we will use CloudFormation, which was made specifically for AWS resources.

What You Will Learn in This Tutorial

After going through this tutorial, you will understand how to maintain your resources within one software file.

In addition to this, you will learn the benefits related to speed that Infrastructure as Code brings to the table. Without IaC, the time and cost of manual deployment of various infrastructures can be much greater compared to maintaining infrastructure as software.

In this article, we will consider an example. It will demonstrate manually provisioning resources vs deploying a CloudFormation script to create a serverless Lambda function and REST API on AWS.

Services We'll Use in This Tutorial

We will use the following services to implement Infrastructure as Code in AWS:

For those who are new to AWS, it's good to have some knowledge of it to understand the article. So, you can follow along with me by creating an account on AWS here and making sure you have AWS CLI installed to work with the example.

Overview of the Example

For the article, we will be implementing a REST API with an API gateway. It will be integrated with a serverless backend Lambda function that handles POST and gets requests made by our API.

The first step will show you how to manually build and deploy these resources using the AWS console. The second step will show you how to automate the process using CloudFormation.

How to Deploy Manually

In manual deployment, we will work inside the AWS console. It is a bit hard to track changes while working outside the local IDE, especially for large-scale projects.

In the first step, we will create a Lambda function.

If you want your Lambda function to work with some other service like Comprehend, you must give permissions for that service. So, make sure to create a role with these permissions.

Following is our Lambda function that will return “Hello World” when integrated with the API gateway.

import json

def lambda_handler(event, context):
    # TODO implement
    return {
        'statusCode': 200,
        'body': json.dumps('Hello World!')
    }

Now that we have configured our Lambda function, the next step is to create a REST API to interact with the Lambda function.

For this, go to Amazon API Gateway, Click create API, and select REST API from the provided options.

Now we will integrate Lambda with the API. For this, create a GET method from the actions menu and point our REST API to the Lambda function.

We are in a position to deploy and test our API for its proper integration with Lambda. Select any stage name you want – for this example, I'm using “prod”.

After deploying the API, you can see a URL on the “prod” stage. Hitting this URL will trigger the Lambda function. As we have returned “Hello World” from our Lambda function, so you should be able to see the desired result.

How to Deploy with CloudFormation

Up to this point, we have seen how manual deployment works, which usually takes a few minutes.

But let’s imagine that we have more than one API, method, and more than one developer working on them. In this scenario, tracking all the resources and changes would be challenging.

So, in this section, we will use AWS CloudFormation instead. It will give flexibility to the developers, allowing them to adjust their Infrastructure with a simple script.

How Does CloudFormation Work?

We will use the YAML file to provision and declare these resources and deploy them to AWS to create a CloudFormation stack. CloudFormation is a stack that contains all the resources required for the project.

We will be using the SAM template as described above in the services section. It is an abstraction of CloudFormation to build serverless applications with less YAML code.

For those who don’t know about YAML, you can think about it like JSON. But CloudFormation uses both of these file formats.

In the first step, we head to our local IDE and write the same Lambda function as we did in the AWS console.

helloworld.py:

import json

def lambda_handler(event, context):
    # TODO implement
    return {
        'statusCode': 200,
        'body': json.dumps('Hello World!')
    }

Next, we will create a template.yaml file containing our infrastructure. We will define our Lambda function and API Gateway in this file.

To build this file, we need to add some information that is common to all SAM templates.

template.yaml:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: First CloudFormation template

Now, we have to add “Globals” to this CloudFormation template.yaml file. Globals are the common configs for the resources that you are going to deploy. Globals allow you to declare information globally for a specific resource type rather than specifying it again and again for different resources.

template.yaml:

Globals:
    #Common to all Lambda functions you create
    Function:
      MemorySize: 128
      Runtime: python3.6
      Timeout: 5

We define have to define the Resources tag in our template.yaml file. The Lambda function and REST API will come under this tag.

template.yaml:

Resources:

    ##Lambda and API GW Integrated
    helloworld:
        Type: AWS::Serverless::Function
        Properties:
          #filename.functionname
          Handler: helloworld.lambda_handler

          #REST API created
          Events:
            PostAdd:
              Type: Api
              Properties:
                Path: /helloworld
                Method: get

In the above code, we define parameters for creating the Lambda function. For the event, we create a REST API that triggers the Lambda function.

Note: There is an array of parameters like CodeURI and description that you can specify for your serverless function. The best way to create a template file is to go through the CloudFormation docs and see the parameters available for your specified resource/service.

How to Deploy the Template File

We can deploy our template.yaml file using the following two AWS CLI commands:

##s3 bucket stores our sam template which we need to deploy
aws cloudformation package --template-file template.yaml --output-template-file sam-template.yaml --s3-bucket helloworld-sam

After running the above command, you will be able to see a SAM template file. We will use this file in the second command below.

In this command, give your appropriate path to the sam-template.yaml file:

#Deploy stack
#point to template file created by a previous command and a stack name as well as your region you're deploying

aws cloudformation deploy --template-file /path to sam-template.yaml file --stack-name test-stack --capabilities CAPABILITY_IAM --region us-east-1

After executing both of these commands, you will see the stack created in the CLI. You can verify it using CloudFormation in the console.

Here you will see all the resources provisioned through the code created and deployed using the template.yaml file.

You can click on API and access the URL to check the output as we did for the manual deployment.

Wrapping Up

That’s it – you have successfully implemented infrastructure as code in AWS using CloudFormation.

I hope this article has been helpful for anyone wanting to understand implementing infrastructure as code in AWS.

Connect with me on LinkedIn and Twitter

Hasta la vista!

In this article I will explain the concepts behind Infrastructure-As-Code (IaC) at a high level and dive into a category of IaC known as server templating.

For the purpose of this article, we will be using a server templating tool called Packer to create a custom Amazon Machine Image (AMI) and use it to launch an AWS EC2 instance.

What is Infrastructure-As-Code?

‌‌Infrastructure-As-Code (IaC) refers to the process of managing and supplying infrastructure using code rather than manual methods.

The core concept is that you define, deploy, update, and destroy your infrastructure (servers, databases, networks, configuration, and so on) by writing and executing code.

Types of Infrastructure

What is Mutable Infrastructure?

You can change and configure infrastructure that is mutable depending on your needs. You can log into servers, apply fixes, update configurations, and more.

‌‌Let's say we have a web application that we want to deploy to the cloud or to an onsite server.

We configure the program to install all necessary instructions and configurations after deployment in order for it to be production ready. We manually or automatically change the server whenever a new upgrade/configuration to the application is required.

This is fantastic for a few servers, but for many servers, it can result in application mismatches unless you use configuration tools like Ansible. This is referred to as mutable infrastructure. It follows the BUILD, DEPLOY, CONFIGURE pattern.

Benefits of Mutable Infrastructure

• Resolve issues more rapidly. Rather of having to build a new server from the ground up, IT staff gets to know each server on a "personal" level, allowing them to diagnose issues more quickly.
• Updates are usually quicker and can be tailored to each server individually.
• The infrastructure can be tailored to the exact requirements of the server-side applications.

Challenges of Mutable Infrastructure

• Because each server has a unique configuration, which is known as configuration drift, technical difficulties are difficult to diagnose or reproduce.
• Because you have to manually configure the servers, provisioning servers is frequently a lengthy procedure.
• Server changes aren't always recorded, making version tracking more complex.
• Update failures happen more often because of a range of unanticipated factors such as network connectivity, unresponsive repositories, DNS downtime, and so on.
• If updates aren't applied correctly, there's a higher danger and complexity in production workloads. Due to these unexpected situations, debugging is tough.

Fortunately, you can employ immutable infrastructure to avoid the issues that mutable infrastructure can cause.

What is Immutable Infrastructure?

In immutable infrastructure, components are recreated and replaced rather than updated after they're formed. This makes your product more consistent and reliable.

‌‌Making your infrastructure immutable is the best way to ensure scalability in the realm of IaC.‌‌

Now, from the use-case in mutable infrastructure, instead of configuring the server after deployment, how about configuring it before deployment, like in the mutable infrastructure above?

Immutable infrastructure is based on this concept, which you can implement with the help of technologies like Packer.‌‌

Benefits of Immutable Infrastructure

• There is no configuration drift because there are no adjustments to make.
• Tracking and rollbacks are significantly easier with discrete versioning. Each new server or virtual machine may be tracked by the IT department as it is deployed.
• Testing is made easier by the uniformity of setups across several servers.
• Predictable state due to the fact that the infrastructure is never changed, which reduces complexity.
• In a multi-threaded context, safe thread code means mutation is almost non-existent.

Challenges of Immutable Infrastructure

• Infrastructure can't be changed while it's in place. For example, if a zero-day vulnerability is discovered, all servers with the same configuration must be updated.
• Immutable infrastructure's increased agility and dynamism can occasionally conflict with standard IT security measures.
• Copying array data from one location to another incurs overhead. Instead of writing data to the local disk, data is externalized.

Types of Infrastructure-As-code Tools

There five five broad categories of IAC tools:

• Ad hoc scripts
• Configuration Management tools
• Orchestration tools
• Provisioning tools
• Server Templating tools

What are Ad Hoc Scripts?

‌‌An Ad hoc technique is when you build a script in your preferred programming language (like Bash or Python) to automate a task, configure a tool, and run it on the server.

What are Configuration Management Tools?

‌‌Large amounts of software is installed and managed using provisioning tools on existing servers.

When clustering computers, keeping the hardware and software as homogeneous as feasible is ideal. This helps make sure that performance is consistent and that the separate nodes get along with one another.

Configuration management tools make it simpler to manage the software side of clusters. Some of these tools include:

• Chef
• Puppet
• Ansible
• SaltStack

What are Orchestration Tools?

‌‌How do you manage VMs and containers once they've been created? You need to roll out updates, monitor the health of your VMs, and distribute traffic between VMs and containers in the real world. Even communication with one another over the internet is possible (server discovery).

Orchestration tools are designed to deal with these issues. ‌‌Orchestration is the simultaneous automation of many operations in order to reduce production difficulties and time to market.

Some of these tools include:

• Kubernetes
• Marathon/Mesos
• Amazon Elastic Container Service
• Docker Swarm
• Normad

What are Provisioning Tools?

The code that runs on each server is defined by configuration management tools, server templating tools, and orchestration tools.

Provisioning tools, on the other hand, are in charge of creating servers, databases, load balancers, monitoring, secure socket layer (SSL) certificates, and many other aspects of your infrastructure.

Example of provisioning tools are‌‌:

• Terraform
• CloudFormation
• OpenStack Heat

What are Server Templating Tools?

‌‌Server templating is a popular alternative to configuration management that has lately gained traction.

The goal behind server templating tools is to build an image of a server that takes a fully self-contained "snapshot" of the operating system (OS), software, files, and all other essential characteristics, rather than launching a number of servers and configuring them by running the same code on each one.

You can then install that image on all of your servers using another IaC program.

Examples of server templating tools are:

• Packer
• Docker
• Vagrant.

In this article, I'll go through how to generate a custom image using Packer server templating tools, and you'll see how to deploy the custom image on AWS instance .

Packer and the Custom Image

What is Packer?

‌‌Packer helps you create and customize images that already have specific applications installed, so that your programs are copied over, and so on.

These images are known as machine images, and each platform has its own image format name with preloaded applications.

For example, here's how some platforms identify these images:

• AMAZON (AMI)
• VMWare (VMDK/VMX)
• VirtualBox (OVF)

Packer lets you generate your own custom machine image and bake your code and configuration into it before deploying it to the server. It's finished. There's no need to alter the config because it's already baked into the image.

And for any subsequent updates, you destroy the server and spin up a new server with a load balancer to help maintain the destroying of one server and the starting of another.

Core Packer Building Blocks

There are three major building blocks in the Packer configuration file:

• builders
• provisioners
• post-processors

Builders create the machine and generate the image across platforms. This is a very important component. The builder block can take an array of different builders from different platform.‌‌

Provisioners help you install, customize and configure the machine image, either through third party software or built-in ones.‌‌

And post-processors run after the image has been build and customized.

Enough of the theory...

Now, it is time for us to build some immutable system with Packer, shall we?

Prerequisites and Installation

• Create an AWS account
• Create an IAM user
• Create and Download your user secret and access key
• Download and Install Packer

How to Build Your First Image with Packer

First, create a folder called packer_custom_image.

Open the folder and create the following files: packer.json, variable.json, setup.sh, and .gitignore.

• packer.json contains all the necessary code for creating the custom i=machine image
• variable.json contains sensitive information we want to keep alway from the public
• setup.sh contains shell scripts we will need to run instructions to our image
• .gitignore contains files/folders hidden from the Git remote server

Note that Packer files used the .json extension and have a format like the JavaScript JSON script.

Folder/File structure

Let's get started!

First let's define our variables and Packer builders inside variable.json like this:

 { 
     "description" : "myWebServer", 
    "access_key" : "enter-aws-your-key", 
    "secret_key" : "enter-aws-your-secret" 
    "source_ami" : "enter-yours" 
   }

Alright so what's going on here?

• descriptions: This variable detects the name of the machine image we are creating
• access_key and secret_key: Your IAM user credentials you created and downloaded. These are required by Packer for authentications and authorization.
• source_ami: The source AMI is the AMI you want to used as a base for your custom AMI. AWS has numerous AMIs such as Amazon Linux, Ubuntu, Windows, Redhat and so on. You can choose from these as a base image. In our case we will use the Amazon Linux AMI.

Now, log into your AWS console. Click services and search for EC2.

Click the launch instance button (we are not launching any instance, we just want to copy our base image).

Next, scroll to Application and OS Images (Amazon Machine Image). Click and choose Amazon image, then copy your most recent AMI.

Then close the instance (DO NOT CREATE/LAUNCH).

Next, let's update the packer.json file like this:

{ 
    "builders": [ 
        { 
            "type": "amazon-ebs", 
            "access_key": "{{user `access_key` }}", 
            "secret_key": "{{user `secret_key` }}", 
            "region" : "us-east-1", 
            "ami_name" : "myfirstami", 
            "source_ami" : "{{user `source_ami` }}", 
            "instance_type" : "t2.micro", 
            "ssh_username" : "ec2-user" 
         } 
    ] 
 }

Let's see what's going on here:

• builder: As previously stated, the builder creates the machine image and can produce identical images across multiple platforms, such as AWS and Azure, thus the array. It requires the builder.
• type: Builder type name
• access and secret keys: For authentication, it fetches data from the variable.json file we created for security, hence the "{{user variable-name }}" format.
• variable-name: the name we gave in our variable.json file (you should use this similar format for variables you don't intend to make public)
• region: identifies the AMI region.
• ami_name: Custom name for our custom machine image name.
• source ami: For the machine image we want to customize from the AWS AMI.
• Instance type: For the temporary instance required by Packer to create our image.
• ssh username: For credentials to create the temporary instance as related to the builder platform. AWS is the word that comes to mind in our case.

Next, we'll customize the image with provisioners. We will be customizing the source AMI with a Jenkins instance.

Update the packer.json, and add the following after the builder array block:

 {  ... 
         "provisioners": 
            [ 
                { 
                    "type": "shell", 
                    "script": "setup.sh" 
                 } 
             ] 
}

‌‌The provisioners come after the builders and take an array as well. We can use many provisioners for a single or multiple builders, giving us more flexibility.

• type: The shell type is the most basic and widely used provisioner. It can accept a script property (external file to run our code), inline properties to run as on the command line, or file properties to upload files, code, and other stuff our images.
• scripts: Takes an external script property to run our configurations.

Let’s setup the scripts file we created earlier.

setup.sh

sleep 30
sudo yum update –y
sudo wget -O /etc/yum.repos.d/jenkins.repo \
    https://pkg.jenkins.io/redhat-stable/jenkins.repo
sudo rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io.key
sudo yum upgrade
sudo amazon-linux-extras install java-openjdk11 -y
sudo yum install jenkins -y
sudo systemctl enable jenkins
sudo systemctl start jenkins
sudo systemctl status jenkins

The script installs Jenkins and its dependencies into our image and starts up the Jenkins instance.

Optionally, we can pass post-processors and sensitive-variables. There are numerous post processors you can use after an image has been built and altered.

‌‌packer.json

{  ... 
        "post-processors": 
            [ 
                { 
                    "type": "manifest", 
                    "output": "out.json" 
                  } 
             ] 
 }

• type: Takes the type of post processor instance, the manifest type defines the way we want our image architecture to be printed after it's created.
• output: the name of the file to print the after the baked image artifacts
• Sensitive-variables‌: Packer allows us specify sensitive variable to omit from the post processor manifest. Let's add our secret and access key:

{...
    "sensitive-variables": 
        [     
            "secret_key",    "access_key"  
         ]
 }

How to Run and Test the Code

‌‌In your terminal, in the root directory, run the code. You should see out.json and out.json.lock created at the root of your application. (These files contain your build and artifacts.)

‌‌packer build -var-file="variable.json" packer.json‌‌

AMI in the AWS Console

Now, let's log in to our AWS console to view our just created custom image. We will also need to spin up an EC2 instance to verify if our image has the proper configurations of the Jenkins instance baked into it.

• Go to your AWS console
• Click service and search for EC2 and scroll to image
• There click AMI and find your newly created custom image

Yay! Our image was created successfully. Now it's time to launch an instance from it.

• From the AMI page, select the recent created AMI and click the Launch instance from AMI at the top right corner. This takes you to the instance page.
• Name and tags – Give the instance a name. say ""My Jenkins Server"
• Application and OS Images (Amazon Machine Image) should remain same (that is, our image)
• Instance type – Leave as default "t2-micro". This allows us to use AWS's free tier
• Key pair – We will need to create a new key-pair to allow us login (SSH) to the instance from our computer terminal. Click the Create new key button and the following window should show:

create new keypair

• Give the key-pair a name and type. If you are using Windows, you can choose the .pem private key file format and use your Windows powershell to SSH into the server. Or you can use .ppk and you will need to download PuTTy, an application that helps give you SSH to remote servers.
• Click the Create key pair once and the key pair is automatically downloaded on your local system (take note of the file download location)
• Network settings – We will need to setup some networking to allow access to our Jenkins instance, and to allow SSH, HTTP, and HTTPS. So click edit and scroll to inbound security groups rules to add the following rule (custom TCP at port 8080) and allow access to anywhere. For more security you can add access from only your IP or customize to your desire.

Custom TCP at Port range will allow us view our Jenkins instance from our browser after SSHing into it via public IP or DNS.

Leave Configure storage and Advance Details as Default.

Then click launch and wait for a bit for the instance to fully launch. Once it's launched click on the instance to find all your configurations (DNS, IPs, and so on)

Now, for the final step, Remember the key-pair that was downloaded into our local system? We will need that now to SSH into this instance that has our Jenkins server.

Open your terminal (PowerShell for Windows users) and enter the following with your correct public-dns or (replace with public ip) and key-pair path:

ssh -i /path/key-pair-name.pem ec2-user@instance-public-dns-name

Follow and accept the prompt to successfully login.

Enter sudo systemctl status jenkins to check the status of the Jenkins instance. If it's not running, start it by entering sudo systemctl start jenkins on the terminal.

Start jenkins instance

Now, let's launch our instance from the browser. Open your favourite browser and Enter 'http://44.201.88.238:8080' to see the Jenkins instance:

Enter the following inside the EC2 instance terminal to see the password and login.

sudo cat /var/lib/jenkins/secrets/initialAdminPassword

Then unlock and enjoy all the features of your newly created Jenkins instance.

Congratulations, You Made It!

You can see the full code for this tutorial here:

• packer.json
• setup.sh

Clean Up

If you want to avoid unnecessary charges from AWS (based on resource use) and since we are not using this in production, we should clean up our resources.

To finish up, you can log into your AWS console‌‌ and terminate the EC2 instance launch with the custom image and delete the image if you are not using it.

Summary

In this tutorial, you have learned what infrastructure as a code is at a high level and how it apply in mutable/immutable practice.

You have also learned how to create a custom image with a powerful too called Hashicorp Packer. For our case we customized a Jenkins server into the image but you can extend it with any custom configurations.

And finally we tested our image by launching an EC2 instance.

I hope this article helps you understand what infrastructure as a code is and how you can use it tools in your applications. Happy learning!

In this article, you will learn how to use OpenStack to operate your own private cloud.

By the end of the tutorial, you will have a core understanding of what OpenStack is and you will know the basics of setting up and administering OpenStack using the OpenMetal platform. You will also understand some commonly used OpenStack services.

In addition to creating this article version of the tutorial, I also created a video version. You can watch the video on the freeCodeCamp.org YouTube channel.

To follow along with this tutorial, it can be helpful to have a basic understanding of the Linux command line, networking, and virtualization. But none of it is required.

Thanks to OpenMetal for sponsoring this tutorial.

What is OpenStack?

OpenStack is an open source cloud computing platform that is used by organizations to manage and control large scale deployments of virtual machines, such as in a cloud computing or virtual private server environment. OpenStack is a popular choice for organizations because it is scalable, reliable, and provides a high degree of control over the underlying infrastructure.

Besides being used to manage deployments of virtual machines, OpenStack can also be used to manage storage and networking resources in a cloud environment.

In some ways OpenStack can be compared to AWS but here are some key differences between the two:

• OpenStack is an open source platform, while AWS is a proprietary platform.
• OpenStack offers more flexibility and customization options than AWS.
• OpenStack typically requires more technical expertise to set up and manage than AWS since you basically have to set up everything yourself.

Let's go into more details about what OpenStack offers.

Beyond standard infrastructure-as-a-service functionality, additional components provide orchestration, fault & service management, and other services to ensure high availability of user applications.

OpenStack diagram.

OpenStack is broken up into services to allow you to plug and play components depending on your needs. The OpenStack map below shows common services and how they fit together.

OpenStack map.

I won't cover every service but here is what some of the more common OpenStack services do.

Object Storage: OpenStack Object Storage (Swift) is a highly scalable, distributed object storage system.

Compute: OpenStack Compute (Nova) is a cloud computing fabric controller, which manages the allocation of compute resources.

Networking: OpenStack Networking (Neutron) is a system for managing networks and IP addresses.

Dashboard: The OpenStack Dashboard (Horizon) is a web-based interface for managing OpenStack resources.

Identity: OpenStack Identity (Keystone) is a system for managing user accounts and access control.

Image: OpenStack Image (Glance) is a service for storing and retrieving virtual machine images.

Block Storage: OpenStack Block Storage (Cinder) is a service for managing block storage devices.

Telemetry: OpenStack Telemetry (Ceilometer) is a service for collecting and storing metering data.

Orchestration: OpenStack Orchestration (Heat) is a service for orchestration and cloud formation.

Bare Metal: OpenStack Bare Metal (Ironic) is a service for provisioning and managing bare metal servers.

Data Processing: OpenStack Data Processing (Sahara) is a service for provisioning and managing Hadoop and Spark clusters.

We will be demonstrating a few of the more common OpenStack services later int his course.

There are a bunch of different ways to deploy and configure OpenStack based on the needs of your application or organization.

In this course, we will learn how to get started with OpenStack and use many of the most common features.

One of the easiest ways to get started with OpenStack is by using the OpenMetal on-demand private cloud. This allows us to quickly deploy OpenStack to the cloud and simplifies the setup process. OpenMetal provided a grant that made this tutorial possible.

While we will be using OpenMetal to learn about OpenStack, the material covered in this tutorial applies to any OpenStack deployment, not just ones that use OpenMetal. So no matter how you want to use OpenStack, this tutorial is for you.

Setting Up OpenStack on OpenMetal

To get OpenStack setup, you need to provision and set up your cloud on OpenMetal. Just follow the prompts on this OpenMetal Central page to get everything setup.

When setting up, you will have to

OpenMetal Private Clouds are deployed with OpenStack to three bare metal servers. These three servers comprise the Private Cloud Core. To OpenStack, these three servers are considered the control plane. Private Clouds are deployed with Ceph, providing your cloud with shared storage. Ceph is an open source software-defined storage solution.

Let's view the hardware assets that were created on OpenMetal. If you just created a cloud, you may already be on on the cloud management page. If not, click "manage". Now click "Assets" on the left side menu.

This page contains a list of assets included with your Private Cloud Deployment. These include your Hardware Control Plane Nodes and IP blocks for Inventory and Provider IP addresses.

Assets Page of Cloud Management Dashboard for OpenMetal

The screenshot above is a list of assets in a Demo Private Cloud. Your Private Cloud can have different hardware based on the options you have selected in your deployment:

In this example, you will notice three main sections:

• 3 Cloud Core mb_small_v1 Control Plane Nodes
• Inventory IP Address Blocks
• Provider IP Address Blocks

With these Private Clouds, OpenStack is deployed with three hyper-converged control plane nodes.

You can access these Control Plane Nodes directly through SSH as the root user. This access is done through the SSH keys you provided during your Private Cloud Deployment. Use this command to connect (you will have to change the key name and IP to match your information):

ssh -i ~/.ssh/your_key_name root@173.231.217.21

Getting Started with OpenStack Horizon

Horizon is the name of the default OpenStack dashboard, which provides a web based user interface to OpenStack services. It allows a user to manage the cloud.

To access your new cloud's OpenStack dashboard (called Horizon) you will need to obtain Horizon's administrator password. The username is "admin".

To begin, SSH into one of the cloud's servers (you can use any IP address from the "Assets" page). For example:

ssh -i ~/.ssh/your_key_name root@173.231.217.21

Once you are logged in to the server, run this command:

grep keystone_admin_password /etc/kolla/passwords.yml

The password will be shown in the output as in this example:

keystone_admin_password: aB0cD1eF2gH3iJ4kL5mN6oP7qR8sT9uV

Next, launch Horizon. On OpenMetal, you can click the "Horizon" tab on the left menu.

Login using "admin" and the password you just accessed.

Horizon dashboard.

Create a Project in OpenStack Horizon

In OpenStack, the cloud is divided through the use of projects. Projects have associated with them users, who have differing levels of access, defined by roles. An administrator defines resource limits per project by modifying quotas.

Now we'll learn how to create a project and associate a user with it. And we will learn how project quotas can be adjusted. The Horizon interface will be similar no matter where you deploy OpenStack. This is not specific to OpenMetal.

There are three root-level tabs on the left menu in Horizon: Project, Admin, and Identity. Only users with administrative privileges can see the admin tab.

To create your first project, navigate to Identity -> Projects.

Projects.

Several projects already exist, including the admin project. These projects are deployed by default and generally should not be modified.

Click the Create Project button near the top right to create a new project.

Under the Name field, specify a name for the project. This example project is called Development. You can also add Project Members and Project Groups but we are not going to cover those yet. Click Create Project to finish creating the first project.

Once created, the project appears in the Project Listing page.

While in the project listing page, you can view and adjust quotas for this project as the admin user. Quotas are limits on resources, like the number of instances.

To view the quotas for this project while in Identity -> Projects tab, find the drop down to the right with the first option being Manage Members. From this menu, click Modify Quotas to view the default quota values.

You will see a form with several tabs and you are presented with the quotas for the Compute service. Quotas exist for the Volume and Network services as well.

You may want to adjust the parameters in this form depending on your workload. Setting a value to -1 means that quota is unlimited.

How to Create a User and Associate with Project

Now that you have a project, you can associate a user with it. There is already the default admin user but now let's see how to create a new user and login with the new user.

First navigate as admin to Identity -> Users. By default, there are several users already listed, and this is expected. These are created during cloud deployment and should generally not be modified.

Click the Create User button.

Users tab.

On the Create User form set values for User Name, Password, Primary Project, and Role. The Email field is optional but is helpful for password resets. For the Project choose the project we created earlier.

For Role there are several options depending on the level of access required. The default OpenStack roles are reader, member, and admin. Additional roles also exist in the drop down. Reader is the least authoritative role in the hierarchy. For this example, choose member for the role.

Press Create User to create the user.

Next, log out of Horizon as admin, and log back in with your new user. Upon logging back in you are by default in the newly created project. You can see the project you are currently in at the top left and your user can be seen at the top right of Horizon.

Managing and Creating Images

Now let's learn how to upload an image (not a graphical image but a copy of a Linux installation) into OpenStack as well as create images out of an existing instance.

Images contain a bootable operating system that is used to create instances. Within your OpenMetal Cloud, there are several different images that are readily available including CentOS, Debian, Fedora, and Ubuntu. In addition to this, you have the option to upload images from other sources or create your images.

We will learn how to upload images to Glance through Horizon and how to create an image from an instance snapshot. Glance is tool for managing images that allows users to discover, retrieve, and register VM (virtual machine) images and container images. Glance uses Ceph to store images instead of the local file system.

To access images from within your Horizon Dashboard, navigate to the Projects tab. Within the projects tab, select Compute and then select Images. This tab contains a list of all your images within OpenStack.

Images

Images can be uploaded through your Horizon dashboard by clicking the Create Image button. When creating an image you must choose the Format of the image. With our configuration, the recommended format for images is QCOW2 – QEMU emulator. QCOW2 is the most common format for Linux KVM, expands dynamically, and supports copy on write.

In order to upload an image on Horizon, you must first have the image locally on your machine. In this example, we will upload a CirrOS image. You can download a CirrOS image here.

Now click the Create Image button near the top right.

For this example we'll use the following values for the fields:

• Image Name: Name of the image
• Image Description: Optional description of the image
• File: The source file on your machine
• Format: QCOW2 – QEMU Emulator

Fill out the details as needed and submit the form. It may take some time to complete uploading the image.

Create an Instance in OpenStack Horizon

With OpenStack, instances, or virtual machines, play a large role in a cloud’s workload. OpenStack provides a way to create and manage instances with its compute service, called Nova.

Nova is the OpenStack project that provides a way to provision compute instances (aka virtual servers). Nova supports creating virtual machines, baremetal servers, and has limited support for system containers. Nova runs as a set of daemons on top of existing Linux servers to provide that service.

Now let's learn how to create an instance, including setting up a private network and router, creating a security group, and how to add an SSH key pair.

Create a Private Network

First, let's learn how to create a private network and router. Later we will create an instance on this private network. The router is created so the private network can be connected to your cloud’s public network, allowing you to assign a floating IP address to it, making the instance accessible over the Internet.

To create a private network, begin by navigating to Project -> Network -> Networks. Then click Create Network.

Networks tab.

For this example, we'll create a network with the following details:

• Network Name: Set a name for the network. This example is called Private.
• Enable Admin State: Leave this checked to enable the network.
• Create Subnet: Leave this checked to create a subnet.
• Availability Zone Hints: Leave this option as default.

Next, move on to the Subnet tab of this form and use these details:

• Subnet Name: Set a name for the subnet. This example subnet is called private-subnet.
• Network Address: Select a private network range. For example: 192.168.0.1/24
• IP Version: Leave this as IPv4.
• Gateway IP: This is optional. If unset, a gateway IP is selected automatically.
• Disable Gateway: Leave this unchecked.

Create a network.

For now we will keep the default details in the Subnet Details tab.

Click Create to create the network. Once created, it appears in the list of networks.

Create a Router

You next need to create a router to bridge the connection between the private network and the public network. The public network is called External.

To create a router, begin by navigating to Project -> Network -> Routers. Click Create Router.

Routers.

Input this data for this example:

• Router Name: Set a name for the router here. This example router is called Router.
• Enable Admin State: Leave this checked to enable the router.
• External Network: Choose the network External.
• Availability Zone Hints: Leave this as the default.

Once complete, create the router by pressing Create Router.

Connect Router to Private Network

Next, connect the router to the private network by attaching an interface. Performing this step allows network communication between the Private and External networks.

To attach an interface to the router, first navigate to the list of routers and locate the one previously created.

Click the name of the router to access its details page. This is where the interface is attached. There are three tabs: Overview, Interfaces, and Static Routes. To attach an interface, navigate to the Interfaces tab then load the form to attach an interface by clicking Add Interface near the top right.

On the new interface, choose the private-subnet for Subnet. If you don't set an IP address one is selected automatically. Press Submit to attach the Private network to this router. The interface is then attached and now listed.

You can visually see the network topology for your cloud by navigating to Project -> Network -> Network Topology.

The example above indicates the External network is connected to the Private network through the router called Router.

Security Groups

Security groups allow control of network traffic to and from instances. For example, port 22 can be opened for SSH for a single IP or a range of IPs.

Let's see how to create a security group for SSH access. Later we'll apply the security group we create to an instance.

To view and manage security groups, navigate to Project -> Network -> Security Groups.

You should notice a single security group called default. This security group restricts all incoming (ingress) network traffic and allows all outgoing (egress) network traffic. When an instance is created, this security group is applied by default. To allow the network traffic your instance requires, only open ports as required to just the needed IP ranges.

To create a security group for SSH, click Create Security Group near the top right.

Name the group SSH and then click Create Security Group

After creating the SSH security group, we need to add a rule allowing SSH traffic. We will allow SSH traffic from the first hardware node in this cloud to this instance.

To add a rule, load the form by navigating to Add Rule near the top right.

We'll need to obtain the IP address of the first hardware node of your cloud. You can find this using OpenMetal Central under your cloud’s Assets Page.

The IP address for the first hardware node.

In the Add Rule menu, add the following information:

• Rule: Select SSH. When adding rules you can choose from predefined options. In this case, we choose the SSH rule from the first drop down.
• Description: Optional. Provide a description of the rule.
• Remote: Select CIDR.
• CIDR: Specify the IP address of your first hardware node.

Press Add to add this rule to the security group.

Adding a rule.

Create an Instance

We now have almost everything in place to create an instance.

We will need an SSH public key. An SSH public key is required to access an instance over SSH. This key is injected into the instance when created. An SSH key cannot be added to an already running instance.

We will create an instance that can be accessed over SSH from one of the cloud’s hardware nodes. So we will have to create an SSH key pair in one of the hardware nodes. The public portion of that key pair is associated with the instance we'll create soon.

** To learn how to create this key pair, see the supplementary guide: Create SSH Key Pair for an OpenStack Control Plane Node.

To create an instance, begin by navigating to Project -> Compute -> Instances. Then click the Launch Instance button.

On the details tab, fill in the following details:

• Instance Name: Set a name for the instance. This example instance is called Jumpstation.
• Description: Optional. Set a description if this applies.
• Availability Zone: Leave as the default, which is nova.
• Count: Controls the number of instances spawned. Just create 1.

Next, move to the Source tab allowing you to specify an operating system image.

Source tab.

Fill in the following details:

• Select Boot Source: In this example, we use Image as the boot source.
• Create New Volume: Leave this checked as Yes. This creates a new Cinder volume where the specified operating system image is copied into it. The volume ultimately exists with the Ceph cluster, in the vms pool.
• Volume Size: Allow the system to determine this for you.
• Delete Volume on Instance Delete: Leave this option set as No. If checked, when the instance is deleted, the volume is as well.
• Under the Available section, select the appropriate operating system. This example uses CentOS 8 Stream (el8-x86_64). Clicking the up arrow will move it to the Allocated section.

This concludes configuring the instance’s source. Next, move to the Flavor tab.

Flavor tab.

Flavors are a way to define the VCPUs, RAM, and Disk space used by an instance. Pre-built flavors are available for you. For this step, select an appropriate flavor from the options under the Available heading. This example uses the m1.small flavor. Click the up arrow to move it to the Allocated section.

Next, move to the Networks tab.

Network tab.

In this section, you specify the network with which the instance is associated. For this example, select the Private network created previously. You can choose the External network as well, but this is generally recommended against in favor of using a floating IP should your instance require Internet connectivity.

You should only expose portions of your network as necessary. This reduces the attack surface and improves application security. If a private network is not created and an instance is created in a default cloud, it is associated with the External network. This means the instance consumes a public IP and it could be reached over the Internet.

Next, skip over the Network Ports tab and move to the Security Groups.

Security Groups tab.

This is where you select security groups for the instance. This example uses the SSH security group in the Available section. Click the up arrow to move the SSH security group to the Allocated section.

As the final step, move to the Key Pair tab.

In this section, you specify an SSH public key to inject into the instance. You can upload your key at this stage using this form using the Import Key Pair button. You can also create a key pair on this tab.

We will create a key pair from the first hardware node in our cloud so this instance will be accessible over SSH from that node.

To create the SSH key pair from the first hardware node, the first step is to login to the first hardware node. You can get the IP of the hardware node from the Assets tab on OpenMetal. We already connected to this node toward the beginning of the tutorial and the command is the same. It will look something like this:

ssh -i ~/.ssh/your_key_name root@173.231.217.21

After logging in to the node, use ssh-keygen to generate an SSH key pair.

For example:

[root@modest-galliform ~]# ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:BNIzHPcqCyjjZqWm88s0zqHrj8J8+gUnkF1cNOEDKZs root@modest-galliform.local
The key's randomart image is:
+---[RSA 3072]----+
|    o=**o        |
|  o..+Bo..       |
| o .+  =. .      |
|  .E   ...       |
|o .+... S        |
|.oo +. o         |
|o*+  ..          |
|BO.+.            |
|*B@+             |
+----[SHA256]-----+

The private key is saved in the default location of /root/.ssh/id_rsa and a passphrase is set for additional security.

To view the contents of the public key, use cat /root/.ssh/id_rsa.pub.

For example:

[root@modest-galliform ~]# cat /root/.ssh/id_rsa.pub
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCv6YOgYbRmXCEFxZP+t+pzh/RRKzsgWpvcnmKwF+uwiKDuihHadScCkgd8dE6ymCjP/+UVdVLGEzXfHXG5EfbcPQYOGjqqOGqOVCHIrhFMG3GjSPao99KaDIAvXsWyTDI9FmrXTiC+0WkmOLNb0UeDic+lQ6KJumw12O1niZjC19jMpWR5amRWEJo6oKFylC8JLHsdfhqr7EBcBzvUJkqh/1zY3qcsABHBrBCWOKC5oNiDAzctQ5MeHq6tv6w6YxdZLLdupczteERN6roroySMtR2JZnOIcnq1aUgD/YDJDeg9zpvUN7stsndONYVOH42+bBu7xEWsm8zobgdfLlmhv+8ab7dKVlYvJUkITqCoKpp8m0f3dbLtQSevCJ9qaeQvmxkjU9OHVPkkTolw4aUHvUsutpVynNfmErf3RGMjQRiQ3ZE7xGKVV7iSFDK9l0mMWBHpYu2OnVKQlP823IC0YKD2dP3qDd/nnvGXVlxfRI+C08n9ehoHwZAIz4SM3dU= root@modest-galliform.local

Copy the entire key. It starts with "ssh-rsa" and continues all the way until the end.

Now back to the Key Pair tab. Click Import Key Pair.

Input the following values:

• Key Pair Name: Set a name for the SSH public key. This example public key is called jumpstation-key but it can really be anything you like.
• Key Type: This example uses an SSH Key key type.
• Public Key: Paste in the public key you just copied.

Click Import Key Pair.

Once the public key is imported, create the instance by pressing Launch Instance. (The other tabs our outside the scope of this demonstration.)

The instance goes through a build process. Allow a few minutes for this to occur. When complete, the instance appears in the Instances Listing page.

Assign and Attach Floating IP

The instance created previously is associated with a private network. Presently, the only way to access this instance is to connect to it from with the cloud’s hardware nodes. Another option for connecting is to use a floating IP. In this section, we demonstrate how to allocate a floating IP and attach it to this instance.

To allocate a floating IP, first navigate to Project -> Network -> Floating IPs. Then click Allocate IP to Project.

Floating IPs tab.

In the popup, make sure Pool is set to External (and optionally add a description) and then click Allocate IP to add this floating IP address for use.

In the same section, allocate the IP to the Jumpstation instance by clicking the Associate button at the far right.

Fill in the details:

• IP Address: This field comes pre-selected with the floating IP so there’s no need to change anything here.
• Port to be associated: Select the instance created previously. In this case, we use the Jumpstation instance.

Click Associate. This instance is now accessible over SSH from the first hardware node of your cloud.

To login to this instance, after you login to your hardware node, run the following command [you will have to change the IP address to the one you just associated] :

ssh -i /root/.ssh/id_rsa centos@173.231.255.40

It should look something like this in your terminal:

[root@modest-galliform ~]# ssh -i /root/.ssh/id_rsa centos@173.231.255.40
The authenticity of host '173.231.255.40 (173.231.255.40)' can't be established.
ECDSA key fingerprint is SHA256:z45zzE8fPuKagtyrSGP9AWR4vIVpBppoaqkqj1Kx4SA.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '173.231.255.40' (ECDSA) to the list of known hosts.
Enter passphrase for key '/root/.ssh/id_rsa': 
Activate the web console with: systemctl enable --now cockpit.socket

[centos@jumpstation ~]$

In the next section, you will have to do something while logged in to this machine.

How to Install and Use OpenStack’s CLI

So far we've been learning how to manage OpenStack through a web browser. But it is also possible to manage through the command line using OpenStack’s CLI called OpenStackClient.

Using the command line to manage your cloud introduces more flexibility in automation tasks and can generally make an administrator’s life simpler. Let's learn how.

We'll now install the OpenStackClient on the instance we just created that we named Jumpstation.

Before installing OpenStackClient, you must obtain two files from Horizon, which are required to prepare your shell environment. Those two files are clouds.yaml and the OpenStack RC file.

• clouds.yaml: Used as a source of configuration for how to connect to a cloud
• OpenStack RC file: Used as a source of authentication for your user and project

To collect these files, log in to Horizon as your user. Navigate to Project -> API Access. Then click Download OpenStack RC File and download the OpenStack clouds.yaml and the OpenStack RC files to your machine. The files are associated with the current user and project that user is in.

Prepare and Install OpenStackClient

Next, use SSH to log in to the instance created previously. If you've been following along, then this instance can only be accessed from one of your control plane nodes. Use the instructions recently given to login to the instance.

Here are the steps to prepare and install the OpenStackClient, after you are logged in.

Step 1: Prepare clouds.yaml and OpenStack RC files

The clouds.yaml file obtained previously must be prepared in this instance. For this demonstration, we'll save it to ~/.config/openstack/clouds.yaml. We will have to copy the contents of clouds.yaml that we downloaded to our machine from Horizon and store it as ~/.config/openstack/clouds.yaml.

Here is how you create the directory and then edit the file. Run the commands after the $ on your command line.

# Create the following directory
$ mkdir -p ~/.config/openstack

# Create and open the clouds.yaml to edit
$ vi ~/.config/openstack/clouds.yaml

To get the contents from the clouds.yaml on your local computer into the instance, you will first have to open up the local version in a text editor. Then you will have to copy all the text and then paste it into the version you just created on the instance.

After you paste the text of the file into the vi editor in you terminal, use the command :wq to save and quit the editor.

The clouds.yaml file can actually be placed in several locations. For more see the Configuration Files heading of the OpenStack Victoria’s documentation.

Next, copy the contents of the OpenStack RC file from your local machine into the instance. The file can be placed anywhere so for this example we will store it in the user's home directory. The full path will be ~/Development-openrc.sh.

Create and start editing the file with the following command (don't use the $ when you copy and paste the command).

$ vi ~/Development-openrc.sh

Just like before, you will have to open up the local version of the file, copy all the text, then paste it into the instance version of the file that is open in your terminal. Then, use the command :wq to save and quit the editor.

Now you have to run the file. First change the permissions:

$ chmod +x Development-openrc.sh

Then, run the file:

$ ./Development-openrc.sh

You will have to enter your OpenStack password.

Then run the command:

$ source Development-openrc.sh

Step 2: Create a Python virtual environment

We'll create a virtual environment so we don't interfere with the system’s Python version.

In a default CentOS 8 Stream installation, the system’s Python executable is /usr/libexec/platform-python and is what will be used to create the virtual environment.

Use /usr/libexec/platform-python -m venv ~/venv to create a virtual environment in path ~/venv.

For example:

$ /usr/libexec/platform-python -m venv ~/venv

Step 3: Activate the Python virtual environment

Use source ~/venv/bin/activate to activate the virtual environment.

For example:

$ source ~/venv/bin/activate

After you have activated the virtual environment, the name of it will appear at the beginning of the command line. So it will look something like this:

[centos@jumpstation ~]$ /usr/libexec/platform-python -m venv ~/venv
[centos@jumpstation ~]$ source ~/venv/bin/activate
(venv) [centos@jumpstation ~]$

Step 4: Upgrade pip

Before installing OpenStackClient and to aid in a smooth installation, upgrade pip. Upgrade pip by using pip install --upgrade pip.

For example:

$ pip install --upgrade pip

Step 5: Install OpenStackClient

With everything prepared, OpenStackClient can be installed.

Note! – There exist two OpenStackClient packages: python-openstackclient and openstackclient. I recommend using python-openstackclient because it is maintained much more frequently than the prior package.

Install OpenStackClient using:

$ pip install python-openstackclient

Step 6: List servers associated with your project

For an initial command, list the servers associated with your project by running openstack server list.

For example:

(venv) [centos@jumpstation ~]$ openstack server list
+--------------------------------------+-------------+--------+---------------------------------------+--------------------------+----------+
| ID                                   | Name        | Status | Networks                              | Image                    | Flavor   |
+--------------------------------------+-------------+--------+---------------------------------------+--------------------------+----------+
| 412fc87f-a4d9-40f1-ba07-fe3eee216c38 | Jumpstation | ACTIVE | Private=173.231.255.40, 192.168.0.140 | N/A (booted from volume) | m1.small |
+--------------------------------------+-------------+--------+---------------------------------------+--------------------------+----------+

Here, we can see the server created previously.

Command Structure

When using OpenStackClient, there is typically a common command pattern for what you want to accomplish. All openstack commands begin with openstack. You can execute openstack by itself to enter into a shell, where commands no longer need to be prefixed by openstack:

(venv) [centos@jumpstation ~]# openstack
(openstack)

List all Available Subcommands

Use openstack --help to list all available subcommands. You initially see all the flags you can pass, but after scrolling a bit, the subcommand list starts:

Commands:
  access rule delete  Delete access rule(s)
  access rule list  List access rules
  access rule show  Display access rule details
  access token create  Create an access token
  acl delete  Delete ACLs for a secret or container as identified by its href. (py
thon-barbicanclient)
[...output truncated...]

Learn more about a Subcommand

After seeing available commands, learn more about a command by using openstack help <command>.

For example, to learn more about the openstack server command, use openstack help server:

$ openstack help server
Command "server" matches:
  server add fixed ip
  server add floating ip
  server add network
  server add port
  server add security group
[...output truncated...]

List Items and Show Details

It is very common when using OpenStackClient to list items and the command form is typically openstack <subcommand> list. For example, openstack server list, lists all servers for the currently configured project.

Furthermore, more information about an item can be found by typically running openstack <subcommand> show <item>. For example, openstack server show Jumpstation shows the details about the instance named Jumpstation.

How Private Clouds are Deployed

Now we'll learn more about how your Private Cloud was deployed and learn more about the environment. OpenStack can be deployed in several different ways and this section highlights the characteristics of your Private Cloud. We also explain some of the advantages for this type of deployment and areas that are unique to OpenMetal.

In OpenMetal, OpenStack is containerized through Docker using Kolla Ansible. This is done through an initial deployment container called FM-Deploy. FM-Deploy provides the initial setup changes during the provisioning process of your Private Cloud. The FM-Deploy Container is a necessary part of the current architecture of your Private Cloud. The FM-Deploy Container should remain running in your Private Cloud as it is used by our systems in the event you want to add or remove nodes from your cloud.

Containerization of OpenStack

OpenMetal uses Kolla Ansible to set up Docker containers for all running services. Should you need to make any configuration changes to your nodes, Kolla Ansible should be used to push these changes. If Kolla Ansible is not used then there is a risk of these changes being reverted during any system updates.

Some advantages of containerization through docker are:

• Containers create an isolated environment reducing software dependencies
• Containers can be scaled and allow for services to balance across your cluster
• Containers provide increased flexibility for test releases, patches, and automation
• Containers have a consistent and repeatable deployment and a shorter initialization time

Disk Storage and Ceph

In OpenMetal, disk storage is provided through Ceph. Ceph is an object storage interface that can provide interfaces for multiple different storage types on a single cluster. In OpenMetal Ceph is comprised of two elements: object storage and block storage.

Ceph object storage utilizes Ceph Object Storage Gateway daemon (RADOSGW). With OpenMetal clouds, Ceph’s RGW replaces Swift so there is no Docker container for Swift. Instead, Swift endpoints are connected directly to the RGW. Authentication for RGW is handled through Keystone in /etc/ceph/ceph.conf.

Ceph block storage connects to the Cinder service utilizing Ceph’s RADOS Block Device. Within your cloud, those objects are stored in Ceph pools. Ceph provides a layer of abstraction that allows objects to be recognized as blocks.

Some advantages of using ceph are:

• Data is self-healing and will redistribute data across your cluster in the event of power, hardware, or connectivity issues
• Data is replicated and highly available
• Ceph has the ability to run on commodity hardware and to mix hardware from different vendors

Introduction to Ceph

Ceph is an open-source, distributed storage system that provides object, block and file storage interfaces from a single cluster.

Ceph was selected as the storage solution for Private Cloud Core OpenStack clouds due to its ability store data in a replicated fashion. The data stored in the Ceph cluster is accessible from any of your cloud’s control plane nodes. The storage is considered shared across all nodes, which can make recovering an instance and its data trivial.

Let's learn how to check the status of your Ceph cluster and see available disk usage using the command line.

Check Ceph Status

First, make sure you are logged into one of your cloud’s control plane nodes (not an instance). To check the status of your Ceph cluster, use ceph status.

For example:

[root@modest-galliform ~]# ceph status
  cluster:
    id:     ac5c03ba-fcb8-4963-b235-e9020b5bfcc2
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum modest-galliform,gifted-badger,hopeful-guineafowl (age 7d)
    mgr: hopeful-guineafowl(active, since 7d), standbys: modest-galliform, gifted-badger
    osd: 3 osds: 3 up (since 7d), 3 in (since 7d)
    rgw: 3 daemons active (gifted-badger.rgw0, hopeful-guineafowl.rgw0, modest-galliform.rgw0)

  task status:

  data:
    pools:   12 pools, 329 pgs
    objects: 2.06k objects, 9.4 GiB
    usage:   31 GiB used, 2.6 TiB / 2.6 TiB avail
    pgs:     329 active+clean

  io:
    client:   2.2 KiB/s rd, 2 op/s rd, 0 op/s wr

Check Ceph Disk Usage

To check the available disk space in your Ceph cluster, use ceph df.

For example:

[root@modest-galliform ~]# ceph df
--- RAW STORAGE ---
CLASS  SIZE     AVAIL    USED    RAW USED  %RAW USED
ssd    2.6 TiB  2.6 TiB  28 GiB    31 GiB       1.15
TOTAL  2.6 TiB  2.6 TiB  28 GiB    31 GiB       1.15

--- POOLS ---
POOL                   ID  PGS  STORED   OBJECTS  USED     %USED  MAX AVAIL
device_health_metrics   1    1  418 KiB        3  1.2 MiB      0    839 GiB
images                  2   32  6.8 GiB      921   20 GiB   0.81    839 GiB
volumes                 3   32  2.4 GiB      662  7.3 GiB   0.29    839 GiB
vms                     4   32  4.2 KiB        6   48 KiB      0    839 GiB
backups                 5   32      0 B        0      0 B      0    839 GiB
metrics                 6   32  2.1 MiB      242  8.4 MiB      0    839 GiB
manila_data             7   32      0 B        0      0 B      0    839 GiB
manila_metadata         8   32      0 B        0      0 B      0    839 GiB
.rgw.root               9   32  2.4 KiB        6   72 KiB      0    839 GiB
default.rgw.log        10   32  3.4 KiB      207  384 KiB      0    839 GiB
default.rgw.control    11   32      0 B        8      0 B      0    839 GiB
default.rgw.meta       12    8      0 B        0      0 B      0    839 GiB

Maintaining OpenStack Software Updates

Software in the OpenStack ecosystem evolves over time, either through new feature additions, bug fixes, or when vulnerabilities are patched. Part of operating an OpenStack cloud involves maintaining its software through updates. In this section, we point out the sections in an OpenMetal cloud where software updates occur and explain best practices when performing updates.

The software of an OpenMetal cloud that can be updated include each hardware node’s package manager and the Kolla Ansible Docker images. Ceph updates are handled through the node’s package manager.

Now we will specifically cover the step to perform package manager updates.

1. Migrate Workload

Package manager updates requiring a server reboot to an OpenMetal control plane node can be disruptive to any workload running on it. Prior to performing disruptive actions, it may be possible to migrate instances another node running the Compute service. For information on how to migrate instances, see OpenStack Nova’s documentation.

2. Update One Node at a Time

While performing package manager updates, ensure updates occur successfully for one hardware node before updating another node.

3. Disable Docker

Before updating the package manager, ensure the Docker socket and service within SystemD are stopped and disabled. For example:

systemctl disable docker.socket
systemctl stop docker.socket
systemctl disable docker.service
systemctl stop docker.service

4. Upgrade Host OS Packages

After verifying the Docker socket and service are stopped, perform the package manager updates.:

dnf upgrade

5. Determine Reboot Need

Once package manager completes, check if a reboot is required with dnf-utils needs-restarting and the reboot hint flag (-r):

$ needs-restarting -r
Core libraries or services have been updated since boot-up:
  * kernel
  * systemd
Reboot is required to fully utilize these updates.
More information: https://access.redhat.com/solutions/27943
$

6. Ceph Maintenance

This step is optional and only required if the node needs to be rebooted.

Prior to reboot, if the node is part of the Ceph cluster automatic OSD removal and data rebalance should be temporarily suspended. To do so, perform:

ceph osd set noout
ceph osd set norebalance

This will reduce rebuild time and help ensure the node rejoins the cluster
automatically.

Once the node reboots and a healthy Ceph cluster is confirmed, these parameters must be unset. To unset this configuration, perform:

ceph osd unset noout
ceph osd unset norebalance

Reboot if Required

Reboot the node if required:

shutdown -r now

You may have to wait a minute or two before you can log back in to the control plan node.

Verify Successful Reboot

When the node comes back online, SSH into it to verify the OpenStack Docker containers have started. Additionally, if this node was part of the Ceph cluster, check Ceph’s cluster status.

To verify the Docker containers have started, use docker ps. You should see a number of Docker containers running. Under the STATUS column, each container should reflect the status Up.

For example:

[root@modest-galliform ~]# docker ps
CONTAINER ID   IMAGE                                                                        COMMAND                  CREATED        STATUS                          PORTS     NAMES
6f7590bc2191   harbor.imhadmin.net/kolla/centos-binary-telegraf:victoria                    "dumb-init --single-…"   20 hours ago   Restarting (1) 14 seconds ago             telegraf
67a4d47e8c78   harbor.imhadmin.net/kolla/centos-binary-watcher-api:victoria                 "dumb-init --single-…"   3 days ago     Up 6 minutes                              watcher_api
af815b1dcb5d   harbor.imhadmin.net/kolla/centos-binary-watcher-engine:victoria              "dumb-init --single-…"   3 days ago     Up 6 minutes                              watcher_engine
a52ab61933ac   harbor.imhadmin.net/kolla/centos-binary-watcher-applier:victoria             "dumb-init --single-…"   3 days ago     Up 6 minutes                              watcher_applier
[...output truncated...]

Next, if this node is part of a Ceph cluster, check Ceph’s status using ceph status.

For example:

[root@modest-galliform ~]# ceph status
  cluster:
    id:     06bf4555-7c0c-4b96-a3b7-502bf8f6f213
    health: HEALTH_OK
[...output truncated...]

The above output shows the status as HEALTH_OK, indicating the Ceph cluster is healthy. Ceph is naturally resilient and should recover from a node being rebooted.

View your Private Cloud’s Resource Usage

Now let's look at how to find the resource usage of your private cloud. We will explore how to utilize the Horizon Dashboard to determine the total memory and compute usage for a project as well as how to view instances stored on each node. Next, we'll look at disk usage by explaining how to briefly interact with your cloud’s Ceph cluster using the command line. Finally, we'll go over adding and removing nodes from your Ceph cluster.

There are currently three variations to private cloud deployments: Small, Standard, and Large. All private cloud deployments have a cluster of three hyper-converged servers but will have different allocations of memory, storage, and CPU processing power depending on the configuration and hardware. In addition, you have the option of adding additional hardware nodes to your cluster.

View Memory and Compute Usage in Horizon

To view the resources used by your cloud, you have to be the admin user and assigned to the admin project. Once you are in the admin project, navigate to Admin -> Compute -> Hypervisors. This section, lists the following items:

• VCPU Usage
• Memory Usage
• Local disk usage

View Instance State Across Cluster

There is also an option to see the location of your instances within your cluster. To view this information, navigate to Admin -> Compute -> Instances. You have the option to see the project, the host, as well as the IP address, and state.

Summary of Instances.

How to Access Resource Information from Ceph

To access information regarding your Ceph cluster’s resource pools, you will need to use Ceph’s CLI. These are a summary of some useful resource monitoring and health commands.

• ceph -s to check the status of Ceph
• ceph df to list the disk usage overview
• ceph health detail provides details about existing health issues
• ceph osd pool ls to list pools add detail for additional details regarding replication and health metrics

###

Create and Restore Volume Backups

With Private Clouds, you have the ability to create backups and snapshots of your volume data. If a volume’s data goes corrupt or is removed by mistake, having a copy of that data could be invaluable. Now we'll look at how to create and recover volume backups using Horizon.

First of all, let's review what a volume is. Volumes are block storage devices that you attach to instances to enable persistent storage. You can attach a volume to a running instance or detach a volume and attach it to another instance at any time. You can also create a snapshot from or delete a volume.

Create a Volume Backup

Navigate in Horizon to Project -> Volume -> Volumes.

Volumes List

Create a backup of your volume by selecting from the drop down Create Backup.

Access Create Volume Backup Form

You will need to fill in the following fields.

• Backup Name: Specify a name for the volume backup
• Description: Provide a description if necessary
• Container Name: Leave this blank otherwise the volume backup cannot be created. Horizon tells you if this field is blank, the backup is stored in a container called volumebackups, but this is not the case with our configuration. With Private Clouds, all volume backups created this way are stored in the Ceph pool called backups.
• Backup Snapshot: If applicable specify a snapshot to create a backup from

After submitting the form, you are navigated to Project -> Volume -> Volume Backups where you can see the volume you just created a backup of.

Volume Backup List

Test Volume Backups

Creating backup copies of your important data is only one part of having a solid backup and recovery plan. Additionally, consider testing backed-up data to ensure if something unexpected does happen that restoring your backups will actually be useful. To test volume backups, you can restore a volume backup within OpenStack alongside the original volume and compare the contents.

Restore a Volume Backup

To restore a volume backup, being by navigating in Horizon to Project -> Volume -> Volume Backups.

Next, find the volume backup you wish to restore and from its drop down on the right, select Restore Backup.

Restore Volume Backup

Choose the volume to restore to, or have the system restore the backup to a new volume.

Ceph, Volumes, and Data Durability

When volume backups are created, they are stored in your cloud’s Ceph cluster in a pool called backups. By default, the Ceph cluster is configured with host level replication across each of your cloud’s three control plane nodes. With this configuration, your cloud could suffer losing all but one Ceph node and still retain all of the cluster’s data.

Conclusion

You should now have a basic understanding of OpenStack and some of the things you can do with it. Good luck setting up your own OpenStack system.

##

Heroku makes it simple to do things like create virtual machines to host applications and to deploy websites.

Some of the features that Heroku offers can actually be created easily with other tools.

In this article, you will learn how to create a very simple web app that allows users to provision virtual machines and deploy static websites at the click of a button, all hosted on Amazon Web Services.

You will learn how to provision infrastructure with code and then you will be able to apply what you learn to your own applications.

This article is a companion to the full course we just posted on the freeCodeCamp.org YouTube channel that will teach you how to provision infrastructure programmatically using Python.

You can watch the course below or on the freeCodeCamp.org YouTube channel (1.5-hour watch).

Provisioning infrastructure is related to platform engineering. A platform engineering team serves an organization by planning, designing, and managing its cloud platforms. And often this can be done programatically.

The tools I teach in this course can be used for much more than just provisioning VMs and deploying websites. They can be used for platform engineering to make it simpler to manage cloud platforms.

Automation API

This course focusses on the Automation API from Pulumi. Pulumi provided freeCodeCamp a grant that made this course possible.

Pulumi's open source infrastructure as code SDK enables you to create, deploy, and manage infrastructure on any cloud, using many different programming languages.

Their Automation API makes it possible to provision infrastructure programmatically using the Pulumi engine. Basically, it makes it simple to write a program that automatically creates VMs, databases, VPCs, static websites and more on a variety of different cloud platforms.

I'm going to show you how to create our Heroku-clone web app using Flask and Python on the backend. However, you don't already have to know how to use Flask and Python to follow along. Also, everything I show you could also be done with many other different frameworks and programming languages and many of the steps are the same no matter what web framework you use.

Our app will provision resources on AWS, but Pulumi makes it simple to provision resources on most of the major cloud providers and it wouldn't take that much updating to the code to use a different provider.

Thanks to Komal Ali who created the code that my code in this course is based off of.

Creating the Heroku Clone

Pulumi CLI

First, make sure you have the Pulumi CLI installed. The way to install is different depending on your operating system.

If you have MacsOS and Homebrew, you can use the command brew install pulumi.

If you have Windows and Chocolatey, you can use the command choco install pulumi.

This page will give you additional ways of installing Pulumi.

AWS CLI

This project also uses AWS so you'll have to make sure you have an AWS account and have the CLI set up and authenticated.

You can sign up for a free AWS account here: https://aws.amazon.com/free/

Learn how to install the AWS CLI for your operating system here: https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html

For MacOS, you can use these commands:

curl "https://awscli.amazonaws.com/AWSCLIV2.pkg" -o "AWSCLIV2.pkg"
sudo installer -pkg AWSCLIV2.pkg -target /

For Windows there are a few extra steps and you should just follow the instructions here.

Next, you need to get an Access key ID and secret access key from AWS. Follow the instructions from Amazon to get these.

Now run the following in the command line:

aws configure

Enter your Access Key ID and Secret Access Key when prompted. You can keep the "Default region name" and "Default output format" as None.

Setup the Project

A Pulumi project is just a directory with some files in it. It's possible for you to create a new one by hand. The pulumi new command, however, automates the process:

pulumi new

If this is the first time you have used Pulumi, you will be directed to enter an access code or login. To get an access code, go to https://app.pulumi.com/account/tokens

The pulumi new command allows you to choose between many templates. Choose the "aws-python" template. You can select the defaults for the other options.

This command has created all the files we need, initialized a new stack named dev (an instance of our project).

Every Pulumi program is deployed to a stack. A stack is an isolated, independently configurable instance of a Pulumi program. Stacks are commonly used to denote different phases of development. In this case, we called it 'dev'.

Now we need to install two more dependencies for our project with

venv/bin/pip install flask requests

Create Flask App

We can now start creating the files for our Flask web app.

Pulumi created a file called "main.py". Change the name of that file to "app.py" because this is a Flask app and Flask will look for a file with that name.

There is some code in that file and we will eventually use code similar to that but for now let's just test that flask is working by replacing the code in "app.py" to the following:

from flask import Flask

app = Flask(__name__)

@app.route('/')
def hello_world():
    return 'Hello World'

if __name__ == '__main__':

    app.run()

This is the most basic Flask web app and we can test it by running the following command in the terminal:

env/bin/flask run

Then we can access the running app in the web browser at the URL "http://127.0.0.1:5000/".

If you go to that URL it should say "Hello World". If so, Flask is working correctly so we can update the "app.py" file to the following:

import os
from flask import Flask, render_template

import pulumi.automation as auto


def ensure_plugins():
    ws = auto.LocalWorkspace()
    ws.install_plugin("aws", "v4.0.0")


def create_app():
    ensure_plugins()
    app = Flask(__name__, instance_relative_config=True)
    app.config.from_mapping(
        SECRET_KEY="secret",
        PROJECT_NAME="herocool",
        PULUMI_ORG=os.environ.get("PULUMI_ORG"),
    )

    @app.route("/", methods=["GET"])
    def index():
        return render_template("index.html")

    from . import sites

    app.register_blueprint(sites.bp)

    from . import virtual_machines

    app.register_blueprint(virtual_machines.bp)

    return app

We import Pulumi's automation framework as a variable called auto. Then in the ensure_plugins function we get access to the LocalWorkspace. A Workspace is the execution context containing a single Pulumi project. Workspaces are used to manage the execution environment, providing various utilities such as plugin installation, environment configuration, and creation, deletion, and listing of stacks.

Because we are deploying AWS resources in this tutorial, we must install the AWS provider plugin within the Workspace so that the Pulumi program will have it available during execution.

The create_app function is what Flask will run when we star the flask app. We first run the ensure_plugins function and then create the Flask app. The app.config.from_mapping() function is used by Flask to set some default configurations that the app will use. This will not be a production ready app but if you ever deploy a Flask app make sure to change secret key. The other variables are used by pulumi.

The rest of the file is common to Flask apps. We set the default route (@app.route("/", methods=["GET"])) to render the template "index.html" (which we still have to create.

Finally, the file imports the sites and virtual_machines files and registers them as a blueprint.

A Blueprint in Flask is a way to organize a group of related views and other code. Rather than registering views and other code directly with an application, they are registered with a blueprint. Then the blueprint is registered with the application when it is available in the factory function.

Basically, we are using Blueprints so we can define additional routes for our web app in other files that we still have to create.

Create Templates

And speaking of creating other files, let's create some. Create a director called "templates" and then create a file inside that directory named "index.html".

Add the following code:

{% extends "base.html" %}

{% block content %}
  <div class="row row-cols-1 row-cols-md-2 g-4">
    <div class="col">
      <div class="card">
        <div class="card-body">
          <h5 class="card-title">Static Websites</h5>
          <p class="card-text">Deploy your own static website in seconds!</p>
          <a href="{{ url_for("sites.list_sites") }}" class="btn btn-primary">Get started</a>
        </div>
      </div>
    </div>
    <div class="col">
      <div class="card">
        <div class="card-body">
          <h5 class="card-title">Virtual Machines</h5>
          <p class="card-text">Set up a virtual machine for development and testing.</p>
          <a href="{{ url_for("virtual_machines.list_vms") }}" class="btn btn-primary">Get started</a>
        </div>
      </div>
    </div>
  </div>
{% endblock %}

I won't go into too much detail about this code. It is basic HTML but anything inside curly braces is an expression that will be output to the final document. Flask uses the Jinja template library to render templates. You will notice that it dynamically renders a list of sites and virtual machines. Soon we will create the Python code that will get those lists using Pulumi.

But next create a file called "base.html". You will notice that "index.html" extends "base.html". This will basically be the header that all our pages share.

Add this code:

<!DOCTYPE html>
<head>
  <title>Herocool - {% block title %}{% endblock %}</title>
  <link rel="stylesheet" href="{{ url_for("static", filename="bootstrap.min.css") }}">
  <script src="{{ url_for("static", filename="bootstrap.min.js") }}"></script>
</head>
<body>
  <div class="container p-2">
    <header class="d-flex flex-wrap justify-content-center py-3 mb-4 border-bottom">
      <a href="{{ url_for("index") }}" class="d-flex align-items-center mb-3 mb-md-0 me-md-auto text-dark text-decoration-none">
        <span class="fs-3">Herocool</span>
      </a>
      {% block nav %}{% endblock %}
    </header>
  </div>
  <section class="container-md px-4">
    <div>
      <header class="row gy-4">
        <span class="fs-4">{% block header %}{% endblock %}</span>
        {% for category, message in get_flashed_messages(with_categories=true) %}
        <div class="alert alert-{{ category }}" role="alert">{{ message }}</div>
        {% endfor %}
      </header>
    </div>
    {% block content %}{% endblock %}
  </section>
</body>

Now, we'll quickly create the rest of our HTML templates, then create the Python files that do the real heavy lifting in our app.

Inside our "templates" directory, create two more directories called "sites" and "virtual_machines". Inside each of those directories create the same three files: "index.html", "create.html", and "update.html".

Here is the code to add to each:

sites/index.html

{% extends "base.html" %}

{% block nav %}
  <ul class="nav nav-pills">
    <li class="nav-item fs-6"><a href="{{ url_for("sites.create_site") }}" class="nav-link active">Create static site</a></li>
  </ul>
{% endblock %}

{% block header %}
  {% block title %}Site Directory{% endblock %}
{% endblock %}

{% block content %}
  <table class="table">
    <tbody>
      {% if not sites %}
      <div class="container gy-5">
        <div class="row py-4">
          <div class="alert alert-secondary" role="alert">
            <p>No websites are currently deployed. Create one to get started!</p>
            <a href="{{ url_for("sites.create_site") }}" class="btn btn-primary">Create static site</a>
          </div>
        </div>
      </div>
      {%  endif %}
      {% for site in sites %}
        <tr>
          <td class="align-bottom" colspan="4">
            <div class="p-1">
              <a href="{{ site["url"] }}" class="fs-5 align-bottom" target="_blank">{{ site["name"] }}</a>
            </div>
          </td>
          <td>
            <div class="float-end p-1">
              <form action="{{ url_for("sites.delete_site", id=site["name"]) }}" method="post">
                <input class="btn btn-sm btn-danger" type="submit" value="Delete">
              </form>
            </div>
            <div class="float-end p-1">
              <form action="{{ url_for("sites.update_site", id=site["name"]) }}" method="get">
                <input class="btn btn-sm btn-primary" type="submit" value="Edit">
              </form>
            </div>
            <div class="float-end p-1">
              <a href="{{ site["console_url"] }}" class="btn btn-sm btn-outline-primary" target="_blank">View in console</a>
            </div>
          </td>
        </tr>
      {% endfor %}
    </tbody>
  </table>
{% endblock %}

sites/create.html

{% extends "base.html" %}

{% block nav %}
  <ul class="nav nav-pills">
    <li class="nav-item fs-6"><a href="{{ url_for("sites.list_sites") }}" class="nav-link">Back to site directory</a></li>
  </ul>
{% endblock %}

{% block header %}
  {% block title %}Create new static site{% endblock %}
{% endblock %}

{% block content %}
<section class="p-2">
  <form method="post">
    <div class="mb-3">
      <label for="site-id" class="form-label">Name</label>
      <input type="text" class="form-control" name="site-id" id="site-id" aria-describedby="nameHelp" required>
      <div id="nameHelp" class="form-text">Choose a unique name as a label for your website</div>
    </div>
    <div class="mb-3">
      <label for="file-url" class="form-label">File URL</label>
      <input type="text" class="form-control" id="file-url" name="file-url">
    </div>
    <div class="mb-3">
      <strong>OR</strong>
    </div>
    <div class="mb-3">
      <label for="site-content" class="form-label">Content</label>
      <textarea class="form-control" name="site-content" id="site-content" rows="5"></textarea>
    </div>
    <button type="submit" class="btn btn-primary">Create</button>
  </form>
</section>
{% endblock %}

sites/update.html

{% extends "base.html" %}

{% block nav %}
  <ul class="nav nav-pills">
    <li class="nav-item fs-6"><a href="{{ url_for("sites.list_sites") }}" class="nav-link">Back to site directory</a></li>
  </ul>
{% endblock %}

{% block header %}
  {% block title %}Update site '{{ name }}'{% endblock %}
{% endblock %}

{% block content %}
  <section class="p-2">
    <form method="post">
      <div class="mb-3">
        <label for="file-url" class="form-label">File URL</label>
        <input type="text" class="form-control" name="file-url" id="file-url">
      </div>
      <div class="mb-3">
        <strong>OR</strong>
      </div>
      <div class="mb-3">
        <label for="site-content" class="form-label">Content</label>
        <textarea class="form-control" name="site-content" id="site-content" rows="5">{{ content }}</textarea>
      </div>
      <button type="submit" class="btn btn-primary">Update</button>
    </form>
  </section>
{% endblock %}

virtual_machines/index.html

{% extends "base.html" %}

{% block nav %}
  <ul class="nav nav-pills">
    <li class="nav-item fs-6"><a href="{{ url_for("virtual_machines.create_vm") }}" class="nav-link active">Create VM</a></li>
  </ul>
{% endblock %}

{% block header %}
  {% block title %}Deployed Virtual Machines{% endblock %}
{% endblock %}

{% block content %}
  <table class="table">
    <tbody>
      {% if not vms %}
      <div class="container gy-5">
        <div class="row py-4">
          <div class="alert alert-secondary" role="alert">
            <p>No virtual machines are currently deployed. Create one to get started!</p>
            <a href="{{ url_for("virtual_machines.create_vm") }}" class="btn btn-primary">Create VM</a>
          </div>
        </div>
      </div>
      {%  endif %}
      {% for vm in vms %}
      <tr>
        <td class="align-bottom" colspan="4">
          <div class="p-1">
            <pre> ssh -i ~/.ssh/id_rsa.pem ec2-user@{{ vm["dns_name"] }} </pre>
          </div>
        </td>
        <td>
          <div class="float-end p-1">
            <form action="{{ url_for("virtual_machines.delete_vm", id=vm["name"]) }}" method="post">
              <input class="btn btn-sm btn-danger" type="submit" value="Delete">
            </form>
          </div>
          <div class="float-end p-1">
            <form action="{{ url_for("virtual_machines.update_vm", id=vm["name"]) }}" method="get">
              <input class="btn btn-sm btn-primary" type="submit" value="Edit">
            </form>
          </div>
          <div class="float-end p-1">
            <a href="{{ vm["console_url"] }}" class="btn btn-sm btn-outline-primary" target="_blank">View in console</a>
          </div>
        </td>
      </tr>
      {% endfor %}
    </tbody>
  </table>
{% endblock %}

virtual_machines/create.html

{% extends "base.html" %}

{% block nav %}
  <ul class="nav nav-pills">
    <li class="nav-item fs-6"><a href="{{ url_for("virtual_machines.list_vms") }}" class="nav-link">Back to Virtual Machines directory</a></li>
  </ul>
{% endblock %}

{% block header %}
  {% block title %}Create new virtual machine{% endblock %}
{% endblock %}

{% block content %}
<section class="p-2">
  <form method="post">
    <div class="mb-3">
      <label for="vm-id" class="form-label">Name</label>
      <input type="text" class="form-control" name="vm-id" id="vm-id" aria-describedby="nameHelp" required>
      <div id="nameHelp" class="form-text">Choose a unique name as a label for your virtual machine</div>
    </div>
    <div class="mb-3">
      <label for="vm-id" class="form-label">Instance Type</label>
      <select name="instance_type" class="form-control" id="instance_type">
      {% for instance_type in instance_types %}
        <option value="{{ instance_type }}" {% if instance_type == curr_instance_type %} selected {% endif %}>
          {{ instance_type }}
        </option>
      {% endfor %}
      </select>
    </div>
    <div class="mb-3">
      <label for="vm-keypair" class="form-label">Public Key</label>
      <textarea class="form-control" name="vm-keypair" id="vm-keypair-content" rows="5" aria-describedby="keypairHelp"></textarea>
      <div id="keypairHelp" class="form-text">The public key to use to connect to the VM </div>
    </div>
    <button type="submit" class="btn btn-primary">Create</button>
  </form>
</section>
{% endblock %}

virtual_machines/update.html

{% extends "base.html" %}

{% block nav %}
  <ul class="nav nav-pills">
    <li class="nav-item fs-6"><a href="{{ url_for("virtual_machines.list_vms") }}" class="nav-link">Back to Virtual Machines directory</a></li>
  </ul>
{% endblock %}

{% block header %}
  {% block title %}Update Virtual Machine '{{ name }}'{% endblock %}
{% endblock %}

{% block content %}
  <section class="p-2">
    <form method="post">
      <div class="mb-3">
        <label for="vm-id" class="form-label">Instance Type</label>
        <select name="instance_type" class="form-control" id="instance_type">
          {% for instance_type in instance_types %}
            <option value="{{ instance_type }}" {% if instance_type == curr_instance_type %} selected {% endif %}>
              {{ instance_type }}
            </option>
          {% endfor %}
          </select>
      </div>
      <div class="mb-3">
        <label for="vm-keypair" class="form-label">Public Key</label>
        <textarea class="form-control" name="vm-keypair" id="vm-keypair-content" rows="5">{{ public_key }}</textarea>
      </div>
      <button type="submit" class="btn btn-primary">Update</button>
    </form>
  </section>
{% endblock %}

Create Functionality with Python

Now it's time to create the functionality of our Heroku clone using Python. The "app.py" file imports from files we still have to create. In the same directory as "app.py", create "sites.py".

Add the following to "sites.py":

import json
import requests
from flask import (
    current_app,
    Blueprint,
    request,
    flash,
    redirect,
    url_for,
    render_template,
)

import pulumi
import pulumi.automation as auto
from pulumi_aws import s3

bp = Blueprint("sites", __name__, url_prefix="/sites")

Those are just the basic imports we need for Flask and Pulumi, including importing the automation framework.

Next, add the the following code to that file. This function defines our Pulumi s3 static website in terms of the content that the caller passes in. This allows us to dynamically deploy websites based on user defined values from the POST body.

def create_pulumi_program(content: str):
    # Create a bucket and expose a website index document
    site_bucket = s3.Bucket(
        "s3-website-bucket", website=s3.BucketWebsiteArgs(index_document="index.html")
    )
    index_content = content

    # Write our index.html into the site bucket
    s3.BucketObject(
        "index",
        bucket=site_bucket.id,
        content=index_content,
        key="index.html",
        content_type="text/html; charset=utf-8",
    )

    # Set the access policy for the bucket so all objects are readable
    s3.BucketPolicy(
        "bucket-policy",
        bucket=site_bucket.id,
        policy=site_bucket.id.apply(
            lambda id: json.dumps(
                {
                    "Version": "2012-10-17",
                    "Statement": {
                        "Effect": "Allow",
                        "Principal": "*",
                        "Action": ["s3:GetObject"],
                        # Policy refers to bucket explicitly
                        "Resource": [f"arn:aws:s3:::{id}/*"],
                    },
                }
            )
        ),
    )

    # Export the website URL
    pulumi.export("website_url", site_bucket.website_endpoint)
    pulumi.export("website_content", index_content)

The purpose of pulumi.export that you see at the end of the code is to export a named stack output. Exported values are attached to the program’s Stack resource. Later you will see how we can access this data that is being exported.

So far, nothing is calling the function we just created. At this pointe, we will create URL endpoints that can be used to call that function and create websites.

First, we'll add the code for the /new URL that will create a new site:

@bp.route("/new", methods=["GET", "POST"])
def create_site():
    """creates new sites"""
    if request.method == "POST":
        stack_name = request.form.get("site-id")
        file_url = request.form.get("file-url")
        if file_url:
            site_content = requests.get(file_url).text
        else:
            site_content = request.form.get("site-content")

        def pulumi_program():
            return create_pulumi_program(str(site_content))

        try:
            # create a new stack, generating our pulumi program on the fly from the POST body
            stack = auto.create_stack(
                stack_name=str(stack_name),
                project_name=current_app.config["PROJECT_NAME"],
                program=pulumi_program,
            )
            stack.set_config("aws:region", auto.ConfigValue("us-east-1"))
            # deploy the stack, tailing the logs to stdout
            stack.up(on_output=print)
            flash(
                f"Successfully created site '{stack_name}'", category="success")
        except auto.StackAlreadyExistsError:
            flash(
                f"Error: Site with name '{stack_name}' already exists, pick a unique name",
                category="danger",
            )

        return redirect(url_for("sites.list_sites"))

    return render_template("sites/create.html")

For a GET request, this route renders the "create.html" template. For a POST request, it creates a new site and the redirects to the list of sites at the / route.

You will notice that there is a stack created. As a reminder, every Pulumi program is deployed to a stack. A stack is an isolated, independently configurable instance of a Pulumi program. In this code, the stack name is based on the id that the user types into the form. There is also a project name, and the program, which is the code block we previously discussed.

Once the stack is created, we can execute commands against the Stack, including update, preview, refresh, destroy, import, and export. In this code we use stack.up() to update the stack. And we pass in a.callback function to stack.up() for standard output.

Next add the code for the root URL that will list the sites:

@bp.route("/", methods=["GET"])
def list_sites():
    """lists all sites"""
    sites = []
    org_name = current_app.config["PULUMI_ORG"]
    project_name = current_app.config["PROJECT_NAME"]
    try:
        ws = auto.LocalWorkspace(
            project_settings=auto.ProjectSettings(
                name=project_name, runtime="python")
        )
        all_stacks = ws.list_stacks()
        for stack in all_stacks:
            stack = auto.select_stack(
                stack_name=stack.name,
                project_name=project_name,
                # no-op program, just to get outputs
                program=lambda: None,
            )
            outs = stack.outputs()
            if 'website_url' in outs:
                sites.append(
                    {
                        "name": stack.name,
                        "url": f"http://{outs['website_url'].value}",
                        "console_url": f"https://app.pulumi.com/{org_name}/{project_name}/{stack.name}",
                    }
                )
    except Exception as exn:
        flash(str(exn), category="danger")

    return render_template("sites/index.html", sites=sites)

A Workspace is the execution context containing a single Pulumi project, a program, and multiple stacks. So we first get access ws = auto.LocalWorkspace().

Then we list_stacks(). This just gives us access to the stack name so we must use auto.select_stack() to get access to the stack. We specifically want the stack.outputs(). This will be what we exported to the stack, including the website_url.

Then we append information for each site to the sites list which is used on the frontend to show the list of sites.

Now, add the code for the /update route.

@bp.route("/<string:id>/update", methods=["GET", "POST"])
def update_site(id: str):
    stack_name = id

    if request.method == "POST":
        file_url = request.form.get("file-url")
        if file_url:
            site_content = requests.get(file_url).text
        else:
            site_content = str(request.form.get("site-content"))

        try:

            def pulumi_program():
                create_pulumi_program(str(site_content))

            stack = auto.select_stack(
                stack_name=stack_name,
                project_name=current_app.config["PROJECT_NAME"],
                program=pulumi_program,
            )
            stack.set_config("aws:region", auto.ConfigValue("us-east-1"))
            # deploy the stack, tailing the logs to stdout
            stack.up(on_output=print)
            flash(f"Site '{stack_name}' successfully updated!",
                  category="success")
        except auto.ConcurrentUpdateError:
            flash(
                f"Error: site '{stack_name}' already has an update in progress",
                category="danger",
            )
        except Exception as exn:
            flash(str(exn), category="danger")
        return redirect(url_for("sites.list_sites"))

    stack = auto.select_stack(
        stack_name=stack_name,
        project_name=current_app.config["PROJECT_NAME"],
        # noop just to get the outputs
        program=lambda: None,
    )
    outs = stack.outputs()
    content_output = outs.get("website_content")
    content = content_output.value if content_output else None
    return render_template("sites/update.html", name=stack_name, content=content)

You will notice that the /update route is very similar to the /new route. Since it uses the same stack_name, a new site is not created.

A new thing in this function is that the code gets the content of the website to return to the frontend template.

Finally, add the following code for a /delete route.

@bp.route("/<string:id>/delete", methods=["POST"])
def delete_site(id: str):
    stack_name = id
    try:
        stack = auto.select_stack(
            stack_name=stack_name,
            project_name=current_app.config["PROJECT_NAME"],
            # noop program for destroy
            program=lambda: None,
        )
        stack.destroy(on_output=print)
        stack.workspace.remove_stack(stack_name)
        flash(f"Site '{stack_name}' successfully deleted!", category="success")
    except auto.ConcurrentUpdateError:
        flash(
            f"Error: Site '{stack_name}' already has update in progress",
            category="danger",
        )
    except Exception as exn:
        flash(str(exn), category="danger")

    return redirect(url_for("sites.list_sites"))

In this delete function we first we get access to the stack then destroy and remove the stack. Just by calling the stack.destroy() function will delete the resource on AWS.

Now, in the same directory as "app.py", create "virtual_machines.py".

Add the following to "virtual_machines.py". This time, we'll add it all at once:

from flask import (Blueprint, current_app, request, flash,
                   redirect, url_for, render_template)

import pulumi
import pulumi_aws as aws
import pulumi.automation as auto
import os
from pathlib import Path

bp = Blueprint("virtual_machines", __name__, url_prefix="/vms")
instance_types = ['c5.xlarge', 'p2.xlarge', 'p3.2xlarge']


def create_pulumi_program(keydata: str, instance_type=str):
    # Choose the latest minimal amzn2 Linux AMI.
    # TODO: Make this something the user can choose.
    ami = aws.ec2.get_ami(most_recent=True,
                          owners=["amazon"],
                          filters=[aws.GetAmiFilterArgs(name="name", values=["*amzn2-ami-minimal-hvm*"])])

    group = aws.ec2.SecurityGroup('web-secgrp',
                                  description='Enable SSH access',
                                  ingress=[aws.ec2.SecurityGroupIngressArgs(
                                      protocol='tcp',
                                      from_port=22,
                                      to_port=22,
                                      cidr_blocks=['0.0.0.0/0'],
                                  )])

    public_key = keydata
    if public_key is None or public_key == "":
        home = str(Path.home())
        f = open(os.path.join(home, '.ssh/id_rsa.pub'), 'r')
        public_key = f.read()
        f.close()

    public_key = public_key.strip()

    print(f"Public Key: '{public_key}'\n")

    keypair = aws.ec2.KeyPair("dlami-keypair", public_key=public_key)

    server = aws.ec2.Instance('dlami-server',
                              instance_type=instance_type,
                              vpc_security_group_ids=[group.id],
                              key_name=keypair.id,
                              ami=ami.id)

    pulumi.export('instance_type', server.instance_type)
    pulumi.export('public_key', keypair.public_key)
    pulumi.export('public_ip', server.public_ip)
    pulumi.export('public_dns', server.public_dns)


@bp.route("/new", methods=["GET", "POST"])
def create_vm():
    """creates new VM"""
    if request.method == "POST":
        stack_name = request.form.get("vm-id")
        keydata = request.form.get("vm-keypair")
        instance_type = request.form.get("instance_type")

        def pulumi_program():
            return create_pulumi_program(keydata, instance_type)
        try:
            # create a new stack, generating our pulumi program on the fly from the POST body
            stack = auto.create_stack(
                stack_name=str(stack_name),
                project_name=current_app.config["PROJECT_NAME"],
                program=pulumi_program,
            )
            stack.set_config("aws:region", auto.ConfigValue("us-east-1"))
            # deploy the stack, tailing the logs to stdout
            stack.up(on_output=print)
            flash(
                f"Successfully created VM '{stack_name}'", category="success")
        except auto.StackAlreadyExistsError:
            flash(
                f"Error: VM with name '{stack_name}' already exists, pick a unique name",
                category="danger",
            )
        return redirect(url_for("virtual_machines.list_vms"))

    current_app.logger.info(f"Instance types: {instance_types}")
    return render_template("virtual_machines/create.html", instance_types=instance_types, curr_instance_type=None)


@bp.route("/", methods=["GET"])
def list_vms():
    """lists all vms"""
    vms = []
    org_name = current_app.config["PULUMI_ORG"]
    project_name = current_app.config["PROJECT_NAME"]
    try:
        ws = auto.LocalWorkspace(
            project_settings=auto.ProjectSettings(
                name=project_name, runtime="python")
        )
        all_stacks = ws.list_stacks()
        for stack in all_stacks:
            stack = auto.select_stack(
                stack_name=stack.name,
                project_name=project_name,
                # no-op program, just to get outputs
                program=lambda: None,
            )
            outs = stack.outputs()
            if 'public_dns' in outs:
                vms.append(
                    {
                        "name": stack.name,
                        "dns_name": f"{outs['public_dns'].value}",
                        "console_url": f"https://app.pulumi.com/{org_name}/{project_name}/{stack.name}",
                    }
                )
    except Exception as exn:
        flash(str(exn), category="danger")

    current_app.logger.info(f"VMS: {vms}")
    return render_template("virtual_machines/index.html", vms=vms)


@bp.route("/<string:id>/update", methods=["GET", "POST"])
def update_vm(id: str):
    stack_name = id
    if request.method == "POST":
        current_app.logger.info(
            f"Updating VM: {stack_name}, form data: {request.form}")
        keydata = request.form.get("vm-keypair")
        current_app.logger.info(f"updating keydata: {keydata}")
        instance_type = request.form.get("instance_type")

        def pulumi_program():
            return create_pulumi_program(keydata, instance_type)
        try:
            stack = auto.select_stack(
                stack_name=stack_name,
                project_name=current_app.config["PROJECT_NAME"],
                program=pulumi_program,
            )
            stack.set_config("aws:region", auto.ConfigValue("us-east-1"))
            # deploy the stack, tailing the logs to stdout
            stack.up(on_output=print)
            flash(f"VM '{stack_name}' successfully updated!",
                  category="success")
        except auto.ConcurrentUpdateError:
            flash(
                f"Error: VM '{stack_name}' already has an update in progress",
                category="danger",
            )
        except Exception as exn:
            flash(str(exn), category="danger")
        return redirect(url_for("virtual_machines.list_vms"))

    stack = auto.select_stack(
        stack_name=stack_name,
        project_name=current_app.config["PROJECT_NAME"],
        # noop just to get the outputs
        program=lambda: None,
    )
    outs = stack.outputs()
    public_key = outs.get("public_key")
    pk = public_key.value if public_key else None
    instance_type = outs.get("instance_type")
    return render_template("virtual_machines/update.html", name=stack_name, public_key=pk, instance_types=instance_types, curr_instance_type=instance_type.value)


@bp.route("/<string:id>/delete", methods=["POST"])
def delete_vm(id: str):
    stack_name = id
    try:
        stack = auto.select_stack(
            stack_name=stack_name,
            project_name=current_app.config["PROJECT_NAME"],
            # noop program for destroy
            program=lambda: None,
        )
        stack.destroy(on_output=print)
        stack.workspace.remove_stack(stack_name)
        flash(f"VM '{stack_name}' successfully deleted!", category="success")
    except auto.ConcurrentUpdateError:
        flash(
            f"Error: VM '{stack_name}' already has update in progress",
            category="danger",
        )
    except Exception as exn:
        flash(str(exn), category="danger")

    return redirect(url_for("virtual_machines.list_vms"))

There is a lot of similarities between this file and the 'sites.py' file. A virtual machine has quite a few more settings that must be set and Pulumi is able to create the exact type of VM we want.

We could make it so a user could customize everything from the web interface, but we just give the user the ability to choose the instance type.

One thing that is required for a VM is a public/private key pair. When creating a VM on AWS, you must give a public key. Then, in order to access the VM you have to have the private key.

You can create keys in your terminal. Run the following command:

ssh-keygen -m PEM

Later when testing out the app, you will have to open the public key file so you can copy the key and paste it into the website.

Now switch the directory with the file you just created. The directory should be the same whether you are on MacOS or Windows.

Type in on your terminal:

cd /Users/[username]/.ssh

AWS needs the file to be in .pem format and that is why we created it with "PEM" above. Now lets rename the file to have the correct extension. You need to change the name of the file called id_rsa to be id_rsa.pem.

On macOS, you can rename with this command:

mv id_rsa id_rsa.pem

On Windows, use:

rename id_rsa id_rsa.pem

When running the Flask app, you may need to enter the public key you just created. You can open the id_rsa.pub in any text editor in order to copy the text. If you have vim, you can use this command to open the file:

vim /Users/beau/.ssh/id_rsa.pub

Testing the App

Now it's time to try out the app. On the terminal, run this command:

FLASK_RUN_PORT=1337 FLASK_ENV=development FLASK_APP=__init__ PULUMI_ORG=[your-org-name] venv/bin/flask run

Now you can try out the app and create websites and VMs. Make sure to delete them after creation so AWS does not keep charging you for the resources.

Conclusion

You should now know enough to start provisioning infrastructure in your own applications using Pulumi's automation API. And if you want to see step-by-step how to do the things in this article, check out the video tutorial:

It is difficult to imagine a business that operates without at least some basic IT infrastructure.

Digital solutions and hardware that are linked together help boost a company's productivity. And good infrastructure improves internal processes and communication between different departments.

IT infrastructure is based on various components that aim to manage the internal business ecosystem or provide services outside the business. The better and more thoughtfully the infrastructure is organized, the more a business can profit and expand.

In this article, we will talk in more detail about what IT infrastructure consists of, what types of infrastructure are most popular, and how you can manage IT infrastructure to make it more efficient.

So, without further ado, let’s get started.

Components of IT Infrastructure

You can’t build IT infrastructure from nothing. The whole ecosystem is made up of software solutions, various hardware, and network connections that work together simultaneously and complement each other.

The whole idea is to improve communication between different devices, whether it's desktop computers, tablets, scanners, various servers, and cloud storage.

So, let's take a closer look at the main components of IT infrastructure:

Hardware

Hardware refers to the physical components and devices that help you organize the infrastructure. They are its foundation. Hardware refers to:

• Desktop computers
• Laptops
• Tablets, smartphones, and other mobile devices
• Servers and data centers
• Internet hubs and routers

Software

Software can include various programs and applications that a business uses to function, provide services, operate internal pipelines, and more. Also, various operating systems can be assigned to Software on top of which all programs and applications are installed.

So, the software parts include:

• Content Management Systems (CMS)
• Customer Relationship Management (CRM)
• Enterprise resource planning (ERP)
• Operating systems
• Web servers
• Custom software for internal work

Networks

Networks allow you to combine devices into a single network and connect them to the Internet. The connection is secured by security firewalls that protect it from malware and breaches. A network includes the following components:

• Servers
• Data centers
• Hubs
• Routers
• Switches

Types of IT Infrastructure

Everything that we have discussed above is part of the standard IT infrastructure that we are all used to. Such a structure can be found in enterprises of various sizes and is managed entirely from within the enterprise.

In simple terms, the company has its own servers or data centers, to which all other devices with their own software are connected.

This approach allows a company to fully control the state of its infrastructure, manage it, improve it, and much more. And it is not surprising that, to many people, this type of infrastructure seems to be the only correct one.

However, there is also a second type of infrastructure – based in the cloud – which is gaining popularity because of the many advantages if offers over a classic infrastructure setup. Let's talk in more detail about both types so you can have a complete picture.

Traditional Infrastructure

The main components of a traditional infrastructure are hardware and software. For example, this can be various servers and desktop PCs. All equipment is located under one roof, making it easy to access the necessary information and computing power.

This infrastructure requires a lot of free space at the deployment site, as well as the power of the equipment itself. As a result, it's more expensive to develop and install this type of infrastructure.

In addition, such a system requires constant monitoring, maintenance, and updating in order to meet the ever-changing market realities.

This means that in addition to spending on equipment, you will also spend money on an infrastructure management team that will monitor its condition, troubleshoot problems, and make timely diagnostics and upgrades.

This option is the safest solution for data storage, as it allows you to completely manage all data and software within the enterprise.

Advantages of traditional infrastructure

• Together with the system, you get a dedicated team that is fully involved in infrastructure management.
• Along with updating the infrastructure, you also need to update the software, which allows you to always be on the cutting edge of new technological solutions.
• You have the ability to work with various types of software.

Cloud-Based Infrastructure

Cloud infrastructures are used by more and more businesses these days due to their flexibility. So what is the main difference from standard infrastructure?

The main difference is that you do not have the equipment to support the infrastructure right in your office. All servers, software solutions, and data storage are located in the cloud.

As a result, you don’t have to spend tons of money on hardware and data centers that can handle the internal IT infrastructure for your facility. Instead, you can find a cloud service provider and rent the cloud storage space you need to transfer all your internal pipeline into it (including valuable data, applications, and much more).

Besides that, you can find your IT infrastructure on one cloud-based software. As a result, your business will receive these advantages from cloud app development:

• All data will be stored in the cloud. But if there's a need, users can store it on their devices to have offline access.
• Staff can use this kind of application on any device with access anywhere.

Now, let’s talk about the main strengths of cloud-based IT infrastructures.

Advantages of Cloud-based IT Infrastructure

Great flexibility and scalability

Flexibility and scalability are important if your company aims for rapid growth. And сloud structures have no limits on storage space and much more horsepower for computing needs.

Besides that, it is much easier to scale the whole structure up and down and raise or lower the number of servers if you need to.

Automation capabilities

By introducing cloud technologies to your business, you are relieved of the headaches associated with equipment management and security issues.

All of those operations are outsourced to a vendor who provides you with a cloud-based structure while you can focus on other important aspects of the business.

It's cost-effective

You might be surprised how cost-effective cloud IT infrastructure is. This is mainly because you pay only for the specific service you use. You do not need to spend money on equipment, management, upgrades, and other aspects that the vendor is now dealing with.

What is IT Infrastructure Management?

IT infrastructure is a complex mechanism that requires a special approach to management. And in order to make it easier, there are three areas of management that you should know about.

IT Operational Management

IT Operational Management is all about tools and processes that help keep infrastructure live. The main idea is to maximize the reliability and efficiency of the infrastructure through a wide range of functions such as network asset discovery, operational analysis, data collection, and much more.

Quite popular in this area is the solution developed by ServiceNow. It includes a developed complex set of modules, namely:

• Module for collecting information about the Discovery structure
• Event management module
• Operational Intelligence module
• Automation module for routine IT processes
• Service topology building module
• Cloud resource management module

ITOM tools track the health of your IT infrastructure in real-time. For example, Time Warner took advantage of this opportunity to simplify incident management.

IT Service Management

This approach is somewhat different from the previous one since its main goal is to combine all the operations responsible for the design, creation, delivery, support and lifecycle management of various IT services. At their core, these operations help to optimally deploy a specific IT solution for a specific user in a large business.

FreshService is an excellent purchase for ITSM as it is an ITIL-ready cloud service support platform. The tool can offer a wide range of ITSM support at any scale for businesses of all types and sizes.

Its main features are the presence of mobile applications, the features of the analysis of the root causes of incidents, as well as the management of problems, projects, and configurations.

IT Asset Management

The final type is responsible for the operations that allow you to manage the life cycle of the infrastructure, from hardware to software solutions. The main objectives of the approach are to optimize support costs, control the infrastructure itself, as well as license agreements for software used in the infrastructure.

Two ServiceNow applications, Discovery and Mapping Service, can help with these tasks.

The former automatically finds and identifies new assets (for example, servers connected to the corporate network) and enters information about them into a special database. In the meantime, the second solution defines the relationships between services and the infrastructure elements on which these services are built. As a result, all IT departments and the company processes become more transparent.

What Is Infrastructure as a Service and What Are Its Benefits for the Business?

Infrastructure as a Service is the ultimate way to create infrastructure for an enterprise today. The main idea is that the vendor that provides cloud capabilities is also involved in the management of equipment and networks, while the business itself gets access through a separate API.

The main advantages of organizing your infrastructure this way are:

• You pay for what you use. You have the right to choose the operations and services necessary for your enterprise, and the final check will contain only the cost of these services.
• Global spending is minimal as payments are made in small installments once a month.
• High-tier security measures. Vendors invest a lot of money and technical means to ensure maximum security of their customers' infrastructures.
• Access to the infrastructure is extremely simple and possible from any device with an Internet connection.

Wrapping Things Up

IT infrastructures today are the mainstay of any modern business, regardless of its size or industry. Technologies combined into one system allow businesses to automate their processes, improve internal and external pipelines, and achieve new performance records.

But without a well-tuned IT infrastructure, it's impossible to gain a competitive advantage, and as a result, you can fall into the shadow of your rivals.

That is why you should think about organizing your own IT infrastructure or work on upgrading your existing one to more advanced cloud solutions that will boost your business and improve all operational aspects.

In this article you will learn all about Infrastructure as Code. I will start with an overview of the general concepts, and then I will show you how to implement Infrastructure as Code with three different labs. The labs use Python and AWS but the concepts will apply to other programming languages and cloud providers.

There is also a video version of this article. You can watch the video on the freeCodeCamp.org YouTube channel (1-hour watch).

Let’s start by talking about what infrastructure as code is. To put it simply, it is setting up your infrastructure as code.

By infrastructure, I mean all the different things needed to deploy your software into a cloud environment. That can mean things like virtual machines, containers, or serverless functions.

Infrastructure also means all the other pieces of infrastructure you need to set up to make that successful. That can be security like IAM and KMS, or networking, or some of the monitoring and logging capabilities.

You can also use code to configure and set up data stores. These are the things that your application needs to store and manage data.

The last piece of the infrastructure landscape is the applications themselves and getting the applications that we're building into the infrastructure.

All of these different pieces of infrastructure can be set up using code.

It is becoming more and more important to automate infrastructure because applications can be deployed to production up to a hundred times per day. You don't want to have to do that manually.

Also, it is helpful to automate infrastructure to be provisioned or deprovisioned in response to load.

Infrastructures as code is is all about finding a way to describe using code what pieces of our infrastructure need to do.

Over the years there has been a transition with how people are using cloud infrastructure within their organizations.

In the first wave, it was relatively simple. The infrastructures were fairly static. It was often a single virtual machine that you just accessed through SSH.

It got a little more complex in the second wave. There were more containers and people started using provisioning tools to specify the application behaviors. People used Docker and DataDog.

Modern cloud infrastructure has added way more complexity. It uses containers, serverless, and more managed services as part of the applications. There are now way more different pieces involved in how people build the infrastructure.

Infrastructure as code is becoming a more important part of how people build and deliver applications because infrastructure as code is what what describes the glue between all the different edges on these diagrams.

Modern Infrastructure

The diagram for the modern infrastructure may look more complicated than the previous ones but it can actually be easier to maintain. A key benefit is that the dark grey squares in the diagram is the only part which is the code that you own. And that part is smaller that the previous ways of doing things. So a lot of the operational burden has been decreased compared to how it used to be done.

With infrastructure as code, you hand off some of the operational burden to AWS, Azure, Kubernetes, or some other system. Now the focus is on the glue between the different services that are being managed by a cloud provider.

Infrastructure code takes on an increasingly more important role in how you manage everything in the modern cloud.

So here are the three main ways that can be used to manage all the resources:

• manual: point and click to create/modify resources in the console.
• ad-hoc automation: CLI commands or scripts to create/modify resources.
• infrastructure as code:
  ○ provisioning: declaratively create/modify resources.
  ○ configuration: change state of an existing resource post-provisioning.

Infrastructure as code gives us the ability to write down what we want the desired state of our infrastructure to be. I'll be showing you exactly how to do these things later.

Writing Infrastructure as Code

One approach to infrastructure as code is to use JSON. Here is an example:

{
  "AWSTemplateFormatVersion": "2010-09-09",
  "Resources": {
    "EC2Instance": {
      "Type": "AWS::EC2::Instance",
      "Properties": {
        "InstanceType": "t2.micro",
        "SecurityGroups": [
          {
            "Ref": "InstanceSecurityGroup"
          }
        ],
        "ImageId": "ami-0080e4c5bc078760e"
      }
    },
    "InstanceSecurityGroup": {
      "Type": "AWS::EC2::SecurityGroup",
      "Properties": {
        "GroupDescription": "Enable HTTP over port 80",
        "SecurityGroupIngress": [
          {
            "IpProtocol": "tcp",
            "FromPort": "80",
            "ToPort": "80",
            "CidrIp": "0.0.0.0/0"
          }
        ]
      }
    }
  }
}

The above is a way to tell AWS what resources you want to have.

Another method is to use a domain specific language (DSL). This is a custom method specific to the tool or cloud provider you are using. Here is an example:

provider aws {
    region = "eu-central-1"
}
resource "aws_security_group"
"web_sg" {
    description = "Enable HTTP over port 80"
    ingress {
        protocol = "tcp"
        from_port = 80
        to_port = 80
        cidr_blocks = ["0.0.0.0/0"]
    }
}
resource "aws_instance"
"web" {
    ami = "ami-0080e4c5bc078760e"
    instance_type = "t2.micro"
    security_groups = ["$(aws_security_group.web_sg.id)"]
}

Yet another way of defining infrastructure using code is to use a well-known programming language. For instance, Pulumi can be used to write infrastructure as code using TypeScript, JavaScript, Python, Go, and .NET.

Here is an example using TypeScript (and later we'll be using Python).

import * as aws from "@pulumi/aws";
let group = new aws.ec2.SecurityGroup("web-sg", {
    description: "Enable HTTP over port 80",
    ingress: [{
        protocol: "tcp",
        fromPort: 80,
        toPort: 80,
        cidrBlocks: ["0.0.0.0/0"]
    }, ],
});
for (let az in aws.getAvailabilityZones().names) {
    let server = new aws.ec2.Instance(`web-${az}`, {
        instanceType: "t2.micro",
        securityGroups: [group.id],
        ami: "ami-0080e4c5bc078760e",
        availabilityZone: az,
    });
}

Using code gives the ability to do things that are not possible in some of the other methods. In the example above, there is a for loop that creates in Instance for each Availability Zone. Code gives the ability to use loops, conditionals, classes, packages, and more. Using popular programming languages also allows the use common IDEs, linters, and test frameworks.

Labs

There are a few different services that allow you to use popular languages to create infrastructure as code. In this article, we'll be using Pulumi.

This article (and the video course) was made possible through a grant from Pulumi.

Pulumi is an open source infrastructure as code tool for creating, deploying, and managing cloud infrastructure. Pulumi works with traditional infrastructure like VMs, networks, and databases, in addition to modern architectures, including containers, Kubernetes clusters, and serverless functions. Pulumi supports dozens of public, private, and hybrid cloud service providers.

We'll be using Python and deploying on AWS, though it can be done with other programming languages and cloud providers.

Lab 1: Provisioning Infrastructure with an S3 Bucket

We'll start out with a simple example. This will show an end-to-end experience of working with Pulumi using very simple resources.

In this first example we will do the following:

• Create a New Project
• Configure AWS
• Provision Infrastructure
• Update Infrastructure
• Make Your Stack Configurable
• Create a Second Stack
• Destroy Your Infrastructure

We will use an S3 bucket and then work through the life cycle with a simple set of resources. Then in future examples you will learn how to implement more complex things.

With Pulumi, infrastructure is organized into projects. Each project is a single program that, when run, declares the desired infrastructure for Pulumi to manage.

Before we start the first lab, make sure Pulumi is installed. The way to install is different depending on your operating system.

If you have MacsOS and Homebrew, you can use the command brew install pulumi.

If you have Windows and Chocolatey, you can use the command choco install pulumi.

This page will give you additional ways of installing Pulumi.

AWS

For this example, we are using AWS. You'll have to make sure you have an AWS account and have the CLI set up and authenticated. (There is also a method that does not use the CLI.)

You can sign up for a free AWS account here: https://aws.amazon.com/free/

Learn how to install the AWS CLI for your operating system here: https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html

For MacOS, you can use these commands:

curl "https://awscli.amazonaws.com/AWSCLIV2.pkg" -o "AWSCLIV2.pkg"
sudo installer -pkg AWSCLIV2.pkg -target /

For Windows there are a few extra steps and you should just follow the instructions here.

Next, you need to get an Access key ID and secret access key from AWS. Follow the instructions from Amazon to get these.

Now run the following in the command line:

aws configure

Enter your Access Key ID and Secret Access Key when prompted. You can keep the "Default region name" and "Default output format" as None.

Step 1: Create a Directory

Each Pulumi project lives in its own directory. Create one now and change into it:

mkdir iac-lab1 cd iac-lab1

Step 2: Initialize Your Project

A Pulumi project is just a directory with some files in it. It's possible for you to create a new one by hand. The pulumi new command, however, automates the process:

pulumi new python -y

If this is the first time you have used Pulumi, you will be directed to enter an access code or login. To get an access code, go to https://app.pulumi.com/account/tokens

This command has created all the files we need, initialized a new stack named dev (an instance of our project). We now need to install our dependencies as part of our virtualenv.

Step 3: Setup Virtual Environment

We now need to create a virtual environment and install the required Python packages. The Python module used to create and manage virtual environments is called venv. Run the following commands:

python3 -m venv venv
source venv/bin/activate
pip3 install -r requirements.txt

Step 4: Inspect Your New Project

Our project is comprised of multiple files:

• __main__.py: your program's main entry point file
• requirements.txt: your project's pip dependency information
• Pulumi.yaml: your project's metadata, containing its name and language

If you look in the __main__.py file you will notice a single line of code:

import pulumi

Configuring AWS

Now that you have a basic project, let's configure AWS support for it.

Step 5: Install the AWS Package

Run the following command to install the AWS package:

pip3 install pulumi-aws

Step 6: Import the AWS Package

Now that the AWS package is installed, add the following line to __main__.py to import it:

import pulumi_aws as aws

Step 7: Configure an AWS Region

Configure the AWS region you would like to deploy to by running the following on the command line.

pulumi config set aws:region us-east-1

You can choose a different AWS region if you like. (See this table for a list of available regions.)

Optional Step: Configure an AWS Profile

If you're using an alternative AWS profile, you can tell Pulumi which to use in one of two ways:

• Using an environment variable: export AWS_PROFILE=<profile name>
• Using configuration: pulumi config set aws:profile <profile name>

Provisioning Infrastructure

Now that you have a project configured to use AWS, you'll create some basic infrastructure in it. We will start with a simple S3 bucket.

Step 8: Declare a New Bucket

Add the following to your __main__.py file:

bucket = aws.s3.Bucket("my-bucket")

So the full file should look like this:

import pulumi
import pulumi_aws as aws

bucket = aws.s3.Bucket("my-bucket")

Step 9: Preview Your Changes

Now preview your changes:

pulumi up

This command evaluates your program, determines the resource updates to make, and shows you an outline of these changes:

Previewing update (dev)

View Live: https://app.pulumi.com/beaucarnes/iac-lab1/dev/previews/827f9488-cc23-441f-946a-9852090ab0e3

     Type                 Name          Plan
 +   pulumi:pulumi:Stack  iac-lab1-dev  create
 +   └─ aws:s3:Bucket     my-bucket     create

Resources:
    + 2 to create

Do you want to perform this update?  [Use arrows to move, enter to select, type to filter]
  yes
> no
  details

This is a summary view. Select details to view the full set of properties:

+ pulumi:pulumi:Stack: (create)
    [urn=urn:pulumi:dev::iac-lab1::pulumi:pulumi:Stack::iac-lab1-dev]
    + aws:s3/bucket:Bucket: (create)
        [urn=urn:pulumi:dev::iac-lab1::aws:s3/bucket:Bucket::my-bucket]
        [provider=urn:pulumi:dev::iac-lab1::pulumi:providers:aws::default_4_22_1::04da6b54-80e4-46f7-96ec-b56ff0331ba9]
        acl         : "private"
        bucket      : "my-bucket-185b4e1"
        forceDestroy: false

Do you want to perform this update?  [Use arrows to move, enter to select, type to filter]
  yes
> no
  details

The stack resource is a synthetic resource that all resources your program creates are parented to.

Step 10: Deploy Your Changes

Now that we've seen the full set of changes, let's deploy them. Select yes:

Updating (dev)

View Live: https://app.pulumi.com/beaucarnes/iac-lab1/dev/updates/1

     Type                 Name          Status
 +   pulumi:pulumi:Stack  iac-lab1-dev  created
 +   └─ aws:s3:Bucket     my-bucket     created

Resources:
    + 2 created

Duration: 5s

Now our S3 bucket has been created in our AWS account. If you view your buckets on the AWS website, you will see the new bucket. If you go to the URL shown in the output you will be taken to the Pulumi Console, which records your deployment history.

Step 11: Export Your New Bucket Name

To inspect your new bucket, you will need its physical AWS name. Pulumi records a logical name, my-bucket, however the resulting AWS name will be different.

Programs can export variables which will be shown in the CLI and recorded for each deployment. Export your bucket's name by adding this line to the end of __main__.py:

pulumi.export('bucket_name', bucket.bucket)

Now deploy the changes with:

pulumi up

You will now notice a new Outputs section in the output displaying the bucket's name:

Outputs:
  + bucket_name: "my-bucket-25cb812"

You can select "Yes" to perform the update.

Step 12: Inspect Your New Bucket

You can see all of the output by running pulumi stack output.

The result should look something like this:

Current stack outputs (1):
    OUTPUT       VALUE
    bucket_name  my-bucket-25cb812

Then you can run the aws CLI to list the objects in this new bucket (getting the bucket name using the command from above):

aws s3 ls $(pulumi stack output bucket_name)

There currently will not be any result from that command since the bucket is still empty.

Updating Your Infrastructure

We just saw how to create new infrastructure from scratch. Next, let's make a few updates:

1. Add an object to your bucket.
2. Serve content from your bucket as a website.
3. Programmatically create infrastructure.

This demonstrates how declarative infrastructure as code tools can be used not just for initial provisioning, but also subsequent changes to existing resources.

Step 13: Add an Object to Your Bucket

Create a directory site/ and add a new index.html file with the following contents:

<html>
    <body>
        <h1>Hello everybody!</h1>
    </body>
</html>

Now update your __main__.py file so that it looks like this:

import pulumi
import pulumi_aws as aws
import os

bucket = aws.s3.Bucket("my-bucket")

filepath = os.path.join("site", "index.html")
obj = aws.s3.BucketObject("index.html",
    bucket=bucket.bucket,
    source=pulumi.FileAsset(filepath)
)

pulumi.export('bucket_name', bucket.bucket)

Deploy the changes:

pulumi up

This will give you a preview and selecting yes will apply the changes:

Previewing update (dev)

View Live: https://app.pulumi.com/beaucarnes/iac-lab1/dev/previews/35a07205-2889-403c-b01d-0342eb01ed85

     Type                    Name          Plan
     pulumi:pulumi:Stack     iac-lab1-dev
 +   └─ aws:s3:BucketObject  index.html    create

Resources:
    + 1 to create
    2 unchanged

Do you want to perform this update?  [Use arrows to move, enter to select, type to filter]
  yes
> no
  details

A single resource is added and the 2 existing resources are left unchanged. This is a key attribute of infrastructure as code — such tools determine the minimal set of changes necessary to update your infrastructure from one change to the next.

Now, list again the contents of your bucket:

aws s3 ls $(pulumi stack output bucket_name)

You will notice that the index.html file has been added:

2019-10-22 16:50:54        68 index.html

Step 14: Serve Content From Your Bucket as a Website

To serve content from your bucket as a website, you'll need to update a few properties.

First, your bucket needs a website property that sets the default index document to index.html. That can be achieved by updating __main__.py file with the following:

bucket = aws.s3.Bucket("my-bucket",
    website={
        "index_document": "index.html"
})

Next, your index.html object will need two changes: an ACL (Access Control List) of public-read so that it can be accessed anonymously over the Internet, and a content type so that it is served as HTML. You will also need to export the resulting bucket's endpoint URL so we can easily access it.

Combining everything, the file should now look like this:

import pulumi
import pulumi_aws as aws
import os
import mimetypes

bucket = aws.s3.Bucket("my-bucket",
    website={
        "index_document": "index.html"
})

filepath = os.path.join("site", "index.html")
mime_type, _ = mimetypes.guess_type(filepath)
obj = aws.s3.BucketObject("index.html",
        bucket=bucket.bucket,
        source=pulumi.FileAsset(filepath),
        acl="public-read",
        content_type=mime_type
)

pulumi.export('bucket_name', bucket.bucket)
pulumi.export('bucket_endpoint', pulumi.Output.concat("http://", bucket.website_endpoint))

Now deploy the changes:

pulumi up

The preview will look something like this:

Previewing update (dev)

View Live: https://app.pulumi.com/beaucarnes/iac-lab1/dev/previews/50730def-0493-4667-83b8-fcedc4408f3a

     Type                    Name          Plan       Info
     pulumi:pulumi:Stack     iac-lab1-dev
 ~   ├─ aws:s3:Bucket        my-bucket     update     [diff: +website]
 ~   └─ aws:s3:BucketObject  index.html    update     [diff: ~acl,contentType]

Outputs:
  + bucket_endpoint: output<string>

Resources:
    ~ 2 to update
    1 unchanged

Do you want to perform this update?  [Use arrows to move, enter to select, type to filter]
  yes
> no
  details

Selecting details during the preview has a lot more information this time:

  pulumi:pulumi:Stack: (same)
    [urn=urn:pulumi:dev::iac-lab1::pulumi:pulumi:Stack::iac-lab1-dev]
    ~ aws:s3/bucket:Bucket: (update)
        [id=my-bucket-25cb812]
        [urn=urn:pulumi:dev::iac-lab1::aws:s3/bucket:Bucket::my-bucket]
        [provider=urn:pulumi:dev::iac-lab1::pulumi:providers:aws::default_4_22_1::032fffef-7448-4be3-97d2-3198f149eb39]
      + website: {
          + indexDocument: "index.html"
        }
    --outputs:--
  + bucket_endpoint: output<string>
    ~ aws:s3/bucketObject:BucketObject: (update)
        [id=index.html]
        [urn=urn:pulumi:dev::iac-lab1::aws:s3/bucketObject:BucketObject::index.html]
        [provider=urn:pulumi:dev::iac-lab1::pulumi:providers:aws::default_4_22_1::032fffef-7448-4be3-97d2-3198f149eb39]
      ~ acl        : "private" => "public-read"
      ~ contentType: "binary/octet-stream" => "text/html"

Do you want to perform this update?  [Use arrows to move, enter to select, type to filter]
  yes
> no
  details

Now just select yes to deploy all of the updates. The result will show the URL that can be used to access your index.html file.

Updating (dev)

View Live: https://app.pulumi.com/beaucarnes/iac-lab1/dev/updates/4

     Type                    Name          Status      Info
     pulumi:pulumi:Stack     iac-lab1-dev
 ~   ├─ aws:s3:Bucket        my-bucket     updated     [diff: +website]
 ~   └─ aws:s3:BucketObject  index.html    updated     [diff: ~acl,contentType]

Outputs:
  + bucket_endpoint: "http://my-bucket-25cb812.s3-website-us-east-1.amazonaws.com"
    bucket_name    : "my-bucket-25cb812"

Resources:
    ~ 2 updated
    1 unchanged

Duration: 6s

Step 15: Access Your Website

Besides going to the URL in a web browser (from the results shown above), you can also access the website with this command in the terminal:

curl $(pulumi stack output bucket_endpoint)

This will fetch and print our index.html file:

<html>
    <body>
        <h1>Hello Pulumi</h1>
    </body>
</html>

Making Your Stack Configurable

Right now, the bucket's contents are hard-coded. Next, you'll make the location of the contents configurable, and add support for populating the bucket with an entire directory's worth of contents.

Step 16: Adding a Config Variable

Instead of hard-coding the "site" directory, we will use configuration to make it easy to change the location without editing the program.

Add this to your __main__.py file right below the imports:

config = pulumi.Config()
site_dir = config.require("siteDir")

Step 17: Populating the Bucket Based on Config

And replace the hard-coded "site" parameter with this imported siteDir variable:

filepath = os.path.join(site_dir, "index.html")
mime_type, _ = mimetypes.guess_type(filepath)

obj = aws.s3.BucketObject("index.html",
    bucket=bucket.bucket,
    source=pulumi.FileAsset(filepath),
    acl="public-read",
    content_type=mime_type
)

Now let's see how this can be helpful. Using the terminal rename the site directory to www:

mv site www

Step 18: Deploying the Changes

Now, deploy your changes. To do so, first configure your stack. If you don't, you'll get an error:

pulumi up

This results in an error like the following:

    error: Missing required configuration variable 'iac-lab1:siteDir'
        please set a value using the command `pulumi config set iac-lab1:siteDir <value>`
    error: an unhandled error occurred: Program exited with non-zero exit code: 1

Configure the iac-workshop:siteDir variable similar to how the aws:region variable was configured:

pulumi config set iac-lab1:siteDir www

Then run:

pulumi up

This detects that the path has changed and will perform a simple update.

Step 19: Add More Files

Instead of hard-coding the set of files, you will now change the program to read the entire contents of the www directory.

Add a new file named about.html to the www directory and add some HTML. Here is one option:

<html>
    <p>I am not a cat.</p>
</html>

Now replace the object allocation code with the code below. You will notice that there is now a for loop to loop through every file in the site directory:

for file in os.listdir(site_dir):
    filepath = os.path.join(site_dir, file)
    mime_type, _ = mimetypes.guess_type(filepath)
    obj = aws.s3.BucketObject(file,
          bucket=bucket.bucket,
          source=pulumi.FileAsset(filepath),
          acl="public-read",
          content_type=mime_type
    )

Deploy:

pulumi up

You will see a single new object created for the about.html file:

Updating (dev)

View Live: https://app.pulumi.com/beaucarnes/iac-lab1/dev/updates/5

     Type                    Name          Status
     pulumi:pulumi:Stack     iac-lab1-dev
 +   └─ aws:s3:BucketObject  about.html    created

You can now go to the URL from before and add "/about.html" to the end.

You can also get the file in the terminal with the command:

curl $(pulumi stack output bucket_endpoint)/about.html

Creating a Second Stack

It is easy to create multiple instances of the same project. In Pulumi, each instance is called a stack. This is useful if you have multiple development or test environments, staging versus production, and scaling a given infrastructure across many regions.

Step 20: Create and Configure a New Stack

Create a new stack named 'prod':

pulumi stack init prod

Next, configure its two required variables:

pulumi config set aws:region eu-west-1
pulumi config set iac-lab1:siteDir wwwprod

You can see the list of stacks for your current project with the command:

pulumi stack ls

It will print all stacks for this project that are available to you:

NAME   LAST UPDATE     RESOURCE COUNT  URL
dev    12 minutes ago  5               https://app.pulumi.com/beaucarnes/iac-lab1/dev
prod*  n/a             n/a             https://app.pulumi.com/beaucarnes/iac-lab1/prod

Step 21: Populate the New Site Directory

We could have used the existing www directory for the siteDir. But for this example you will use a different wwwprod directory to demonstrate how it can be configured.

So create this new directory:

mkdir wwwprod

Add a new index.html file to it. It can contain any HTML but here is a suggestion:

<html>
    <body>
        <h1>Hello universe.</h1>
        <p>(in production)</p>
    </body>
</html>

Step 22: Deploy the New Stack

Now deploy all of the changes:

pulumi up

This will create an entirely new set of resources from scratch, unrelated to the existing dev stack's resources.

Updating (prod)

View Live: https://app.pulumi.com/beaucarnes/iac-lab1/prod/updates/1

     Type                    Name           Status
 +   pulumi:pulumi:Stack     iac-lab1-prod  created
 +   ├─ aws:s3:Bucket        my-bucket      created
 +   └─ aws:s3:BucketObject  index.html     created

Outputs:
    bucket_endpoint: "http://my-bucket-0d2f29a.s3-website-eu-west-1.amazonaws.com"
    bucket_name    : "my-bucket-0d2f29a"

Resources:
    + 3 created

Duration: 14s

You will notice a new URL in the output. You can access that URL in a browser to see the new website.

Also you can get it using this command in the terminal:

curl $(pulumi stack output bucket_endpoint)

Destroying Your Infrastructure

The final thing we'll cover in this first lab is to destroy all of the resources from the two stacks created.

Step 23: Destroy Resources

First, destroy the resources in your current stack:

pulumi destroy

This will show you a preview, much like the pulumi up command does:

     Type                    Name           Plan
 -   pulumi:pulumi:Stack     iac-lab1-prod  delete
 -   ├─ aws:s3:BucketObject  index.html     delete
 -   └─ aws:s3:Bucket        my-bucket      delete

Outputs:
  - bucket_endpoint: "http://my-bucket-0d2f29a.s3-website-eu-west-1.amazonaws.com"
  - bucket_name    : "my-bucket-0d2f29a"

Resources:
    - 3 to delete

Do you want to perform this destroy?  [Use arrows to move, enter to select, type to filter]
  yes
> no
  details

To proceed, select yes.

Destroying (prod)

View Live: https://app.pulumi.com/beaucarnes/iac-lab1/prod/updates/2

     Type                    Name           Status
 -   pulumi:pulumi:Stack     iac-lab1-prod  deleted
 -   ├─ aws:s3:BucketObject  index.html     deleted
 -   └─ aws:s3:Bucket        my-bucket      deleted

Outputs:
  - bucket_endpoint: "http://my-bucket-0d2f29a.s3-website-eu-west-1.amazonaws.com"
  - bucket_name    : "my-bucket-0d2f29a"

Resources:
    - 3 deleted

Duration: 3s

The resources in the stack have been deleted, but the history and configuration associated with the stack are still maintained.
If you want to remove the stack completely, run 'pulumi stack rm prod'.

Step 24: Remove the Stack

The AWS resources for this stack have been destroyed. You may have noticed the message printed at the end that the history and configuration associated with the stack are still maintained. This means all past history is still available and you can perform subsequent updates on this stack.

Now, fully remove the stack and all history:

pulumi stack rm

This is irreversible and so asks to confirm that this is your intent:

This will permanently remove the 'prod' stack!
Please confirm that this is what you'd like to do by typing ("prod"):

Type the name of the stack and hit enter. The stack is now gone.

Step 24: Select Another Stack, Rinse and Repeat

After destroying prod, you still have the dev stack. To destroy it too, first select it:

pulumi stack select dev

Now, go back and repeat the previous two steps to destroy it.

Step 25: Verify That Stacks are Gone

Verify that all of this project's stacks are now gone:

pulumi stack ls

Conclusion of Lab 1

Congratulations! You have completed the first lab. The next few labs are a little shorter and demonstrate some more advanced tasks.

Lab 2: Provisioning EC2 Virtual Machines

Amazon Elastic Compute Cloud (Amazon EC2) is a web service that provides secure, resizable compute capacity in the cloud. It is designed to make web-scale cloud computing easier for developers. Amazon EC2’s simple web service interface allows you to obtain and configure capacity with minimal friction.

In this second lab, you'll first create a single EC2 virtual machine (VM). Afterwards, you'll scale that out to a VM per availability zone in your region, and then add a load balancer to spread load across the entire fleet.

Step 1: Create a Directory

We could use the same project and directory as before. But we'll create a new one to get more practice.

Create a new directory that is not inside the directory of used in the last lab.

cd ..
mkdir iac-lab2
cd iac-lab2

Step 2: Initialize Your Project

Remember, a Pulumi project is just a directory with some files in it. It’s possible for you to create a new one by hand. The pulumi new command, however, automates the process:

pulumi new aws-python -y

This command has created all the files we need, initialized a new stack named dev (an instance of our project), and installed the needed package dependencies from PyPi. The "aws" part of the command made sure our __main__.py file started with samle code to create an AWS bucket.

Step 3: Configure an AWS Region

Configure the AWS region you would like to deploy to:

pulumi config set aws:region us-west-2

Step 4: Declare the EC2 Instance

Remove any existing code here from the bootstrapping of your project. Then, import the AWS package in an empty __main__.py file:

from pulumi import export
import pulumi_aws as aws

Now dynamically query the Amazon Linux machine image. Doing this in code avoids needing to hard-code the machine image (a.k.a., its AMI - An Amazon Machine Image provides the information required to launch an instance):

ami = aws.ec2.get_ami(
    most_recent="true",
    owners=["137112412989"],
    filters=[{"name":"name","values":["amzn-ami-hvm-*-x86_64-ebs"]}])

We also need to grab the default Virtual Private Cloud is a service that lets you launch AWS resources in a logically isolated virtual network that you define) that is available in our AWS account:

vpc = aws.ec2.get_vpc(default=True)

Next, create an AWS security group. This enables ping over ICMP and HTTP traffic on port 80:

group = aws.ec2.SecurityGroup(
    "web-secgrp",
    description='Enable HTTP access',
    vpc_id=vpc.id,
    ingress=[
        { 'protocol': 'icmp', 'from_port': 8, 'to_port': 0, 'cidr_blocks': ['0.0.0.0/0'] },
        { 'protocol': 'tcp', 'from_port': 80, 'to_port': 80, 'cidr_blocks': ['0.0.0.0/0'] }
])

Create the server. Notice it has a startup script that spins up a simple Python webserver:

server = aws.ec2.Instance(
    'web-server',
    instance_type="t2.micro",
    vpc_security_group_ids=[group.id],
    ami=ami.id,
    user_data="""
#!/bin/bash
echo "Hello, World!" > index.html
nohup python -m SimpleHTTPServer 80 &
    """,
    tags={
        "Name": "web-server",
    },
)

For most real-world applications, you would want to create a dedicated image for your application, rather than embedding the script in your code like this.

Finally export the EC2 instances’s resulting IP address and hostname:

pulumi.export('ip', server.public_ip)
pulumi.export('hostname', server.public_dns)

After this change, your __main__.py should look like this:

import pulumi
import pulumi_aws as aws


ami = aws.ec2.get_ami(
    most_recent="true",
    owners=["137112412989"],
    filters=[{"name": "name", "values": ["amzn-ami-hvm-*-x86_64-ebs"]}],
)

vpc = aws.ec2.get_vpc(default=True)

group = aws.ec2.SecurityGroup(
    "web-secgrp",
    description="Enable HTTP Access",
    ingress=[
        {
            "protocol": "icmp",
            "from_port": 8,
            "to_port": 0,
            "cidr_blocks": ["0.0.0.0/0"],
        },
        {
            "protocol": "tcp",
            "from_port": 80,
            "to_port": 80,
            "cidr_blocks": ["0.0.0.0/0"],
        },
    ],
)

server = aws.ec2.Instance(
    "web=server",
    instance_type="t2.micro",
    vpc_security_group_ids=[group.name],
    ami=ami.id,
    user_data="""
#!/bin/bash
echo "Hello, World!" > index.html
nohup python -m SimpleHTTPServer 80 &
    """,
    tags={
        "Name": "web-server",
    },
)

pulumi.export("ip", server.public_ip)
pulumi.export("hostname", server.public_dns)

Step 5: Provision the EC2 Instance and Access It

To provision the VM, run:

pulumi up

After confirming, you will see output like the following:

Updating (dev)

View Live: https://app.pulumi.com/beaucarnes/iac-lab2/dev/updates/3

     Type                 Name          Status
     pulumi:pulumi:Stack  iac-lab2-dev
 ~   └─ aws:ec2:Instance  web=server    updated

Outputs:
    hostname: "ec2-52-13-25-152.us-west-2.compute.amazonaws.com"
    ip      : "52.13.25.152"

Resources:
    ~ 1 updated
    2 unchanged

Duration: 8s

If you get an error, try running pulumi up again and confirming. The first time I ran it there was a race condition which required re-running the command.

You can now try accessing the URL from the output in a web browser.

Alternatively, you can use this command:

curl $(pulumi stack output hostname)

Either way you should see a response from the Python webserver:

Hello, World!

Step 6: Add more EC2 instances

Now you will create multiple EC2 instances, each running the same Python webserver, across all AWS availability zones in your region. Replace the part of your code that creates the webserver and exports the resulting IP address and hostname with the following:

...
ips = []
hostnames = []
for az in aws.get_availability_zones().names:
    server = aws.ec2.Instance(f'web-server-{az}',
      instance_type="t2.micro",
      vpc_security_group_ids=[group.id],
      ami=ami.id,
      availability_zone=az,
      user_data="""#!/bin/bash
echo \"Hello, World -- from {}!\" > index.html
nohup python -m SimpleHTTPServer 80 &
""".format(az),
      tags={
          "Name": "web-server",
      },
    )
    ips.append(server.public_ip)
    hostnames.append(server.public_dns)

pulumi.export("ips", ips)
pulumi.export("hostnames", hostnames)

After this change, your __main__.py should look like this:

"""An AWS Python Pulumi program"""

import pulumi
import pulumi_aws as aws


ami = aws.ec2.get_ami(
    most_recent="true",
    owners=["137112412989"],
    filters=[{"name": "name", "values": ["amzn-ami-hvm-*-x86_64-ebs"]}],
)

vpc = aws.ec2.get_vpc(default=True)

group = aws.ec2.SecurityGroup(
    "web-secgrp",
    description="Enable HTTP Access",
    vpc_id=vpc.id,
    ingress=[
        {
            "protocol": "icmp",
            "from_port": 8,
            "to_port": 0,
            "cidr_blocks": ["0.0.0.0/0"],
        },
        {
            "protocol": "tcp",
            "from_port": 80,
            "to_port": 80,
            "cidr_blocks": ["0.0.0.0/0"],
        },
    ],
)


ips = []
hostnames = []

for az in aws.get_availability_zones().names:
    server = aws.ec2.Instance(
        f"web-server-{az}",
        instance_type="t2.micro",
        vpc_security_group_ids=[group.id],
        ami=ami.id,
        user_data="""#!/bin/bash
echo \"Hello, World -- from {}!\" > index.html
nohup python -m SimpleHTTPServer 80 &
""".format(
            az
        ),
        tags={
            "Name": "web-server",
        },
    )
    ips.append(server.public_ip)
    hostnames.append(server.public_dns)

pulumi.export("ips", ips)
pulumi.export("hostnames", hostnames)

Now run a command to update your stack with the new resource definitions:

pulumi up

You will see output like the following:

Updating (dev)

View Live: https://app.pulumi.com/beaucarnes/iac-lab2/dev/updates/4

     Type                 Name                   Status
     pulumi:pulumi:Stack  iac-lab2-dev
 +   ├─ aws:ec2:Instance  web-server-us-west-2a  created
 +   ├─ aws:ec2:Instance  web-server-us-west-2b  created
 +   ├─ aws:ec2:Instance  web-server-us-west-2c  created
 +   ├─ aws:ec2:Instance  web-server-us-west-2d  created
 -   └─ aws:ec2:Instance  web=server             deleted

Outputs:
  - hostname : "ec2-52-13-25-152.us-west-2.compute.amazonaws.com"
  + hostnames: [
  +     [0]: "ec2-34-222-148-4.us-west-2.compute.amazonaws.com"
  +     [1]: "ec2-54-69-196-240.us-west-2.compute.amazonaws.com"
  +     [2]: "ec2-54-186-64-129.us-west-2.compute.amazonaws.com"
  +     [3]: "ec2-52-25-211-116.us-west-2.compute.amazonaws.com"
    ]
  - ip       : "52.13.25.152"
  + ips      : [
  +     [0]: "34.222.148.4"
  +     [1]: "54.69.196.240"
  +     [2]: "54.186.64.129"
  +     [3]: "52.25.211.116"
    ]

Resources:
    + 4 created
    - 1 deleted
    5 changes. 2 unchanged

Duration: 1m55s

Notice that your original server was deleted and new ones created in its place, because its name changed.

To test the changes, curl any of the resulting IP addresses or hostnames (or access the URLs in a web browser):

for i in {0..2}; do curl $(pulumi stack output hostnames | jq -r ".[${i}]"); done

The count of servers depends on the number of AZs in your region. You may need to adjust the {0..2} .

The pulumi stack output command emits JSON serialized data — hence the use of the jq tool to extract a specific index. If you don’t have jq, don’t worry; simply copy-and-paste the hostname or IP address from the console output and curl that.

Note that the webserver number is included in its response:

Hello, World -- from us-west-2a!
Hello, World -- from us-west-2b!
Hello, World -- from us-west-2c!

Add a loadbalancer

Needing to loop over the webservers isn’t very realistic. You will now create a load balancer over them to distribute load evenly.

Step 7: Update our Security Group

We need to add an egress rule to our security group. Whenever you add a listener to your load balancer or update the health check port for a target group used by the load balancer to route requests, you must verify that the security groups associated with the load balancer allow traffic on the new port in both directions.

...
group = aws.ec2.SecurityGroup(
    "web-secgrp",
    ingress=[
        { 'protocol': 'icmp', 'from_port': 8, 'to_port': 0, 'cidr_blocks': ['0.0.0.0/0'] },
        { 'protocol': 'tcp', 'from_port': 80, 'to_port': 80, 'cidr_blocks': ['0.0.0.0/0'] },
    ],
    egress=[
        { 'protocol': 'tcp', 'from_port': 80, 'to_port': 80, 'cidr_blocks': ['0.0.0.0/0'] },
    ]
)
...

This is required to ensure the security group ingress rules don’t conflict with the load balancer’s.

Step 8: Define the ALB

Now right after the security group creation, and before the EC2 creation block, add the load balancer creation steps:

...
vpc = aws.ec2.get_vpc(default=True)
vpc_subnets = aws.ec2.get_subnet_ids(vpc_id=vpc.id)

lb = aws.lb.LoadBalancer(
    "loadbalancer",
    internal=False,
    security_groups=[group.id],
    subnets=vpc_subnets.ids,
    load_balancer_type="application",
)

target_group = aws.lb.TargetGroup(
    "target-group", port=80, protocol="HTTP", target_type="ip", vpc_id=vpc.id
)

listener = aws.lb.Listener(
    "listener",
    load_balancer_arn=lb.arn,
    port=80,
    default_actions=[{"type": "forward", "target_group_arn": target_group.arn}],
)
...

Here, we’ve defined the ALB, its TargetGroup and some Listeners, but we haven’t actually added the EC2 instances to the ALB.

Step 9: Add the Instances to the ALB

Replace the EC2 creation block with the following:

...
ips = []
hostnames = []
for az in aws.get_availability_zones().names:
    server = aws.ec2.Instance(f'web-server-{az}',
      instance_type="t2.micro",
      security_groups=[group.name],
      ami=ami.id,
      user_data="""#!/bin/bash
echo \"Hello, World -- from {}!\" > index.html
nohup python -m SimpleHTTPServer 80 &
""".format(az),
      availability_zone=az,
      tags={
          "Name": "web-server",
      },
    )
    ips.append(server.public_ip)
    hostnames.append(server.public_dns)

    attachment = aws.lb.TargetGroupAttachment(f'web-server-{az}',
        target_group_arn=target_group.arn,
        target_id=server.private_ip,
        port=80,
    )

export('ips', ips)
export('hostnames', hostnames)
export("url", lb.dns_name)

After this change, your main.py should look like this:

"""An AWS Python Pulumi program"""

import pulumi
import pulumi_aws as aws


ami = aws.ec2.get_ami(
    most_recent="true",
    owners=["137112412989"],
    filters=[{"name": "name", "values": ["amzn-ami-hvm-*-x86_64-ebs"]}],
)

vpc = aws.ec2.get_vpc(default=True)
vpc_subnets = aws.ec2.get_subnet_ids(vpc_id=vpc.id)

group = aws.ec2.SecurityGroup(
    "web-secgrp",
    description="Enable HTTP Access",
    vpc_id=vpc.id,
    ingress=[
        {
            "protocol": "icmp",
            "from_port": 8,
            "to_port": 0,
            "cidr_blocks": ["0.0.0.0/0"],
        },
        {
            "protocol": "tcp",
            "from_port": 80,
            "to_port": 80,
            "cidr_blocks": ["0.0.0.0/0"],
        },
    ],
    egress=[
        {
            "protocol": "tcp",
            "from_port": 80,
            "to_port": 80,
            "cidr_blocks": ["0.0.0.0/0"],
        }
    ],
)

lb = aws.lb.LoadBalancer(
    "loadbalancer",
    internal=False,
    security_groups=[group.id],
    subnets=vpc_subnets.ids,
    load_balancer_type="application",
)

target_group = aws.lb.TargetGroup(
    "target-group", port=80, protocol="HTTP", target_type="ip", vpc_id=vpc.id
)

listener = aws.lb.Listener(
    "listener",
    load_balancer_arn=lb.arn,
    port=80,
    default_actions=[{"type": "forward", "target_group_arn": target_group.arn}],
)


ips = []
hostnames = []

for az in aws.get_availability_zones().names:
    server = aws.ec2.Instance(
        f"web-server-{az}",
        instance_type="t2.micro",
        vpc_security_group_ids=[group.id],
        ami=ami.id,
        user_data="""#!/bin/bash
echo \"Hello, World -- from {}!\" > index.html
nohup python -m SimpleHTTPServer 80 &
""".format(
            az
        ),
        tags={
            "Name": "web-server",
        },
    )
    ips.append(server.public_ip)
    hostnames.append(server.public_dns)

    attachment = aws.lb.TargetGroupAttachment(
        f"web-server-{az}",
        target_group_arn=target_group.arn,
        target_id=server.private_ip,
        port=80,
    )


pulumi.export("ips", ips)
pulumi.export("hostnames", hostnames)
pulumi.export("url", lb.dns_name)

This is all the infrastructure we need for our load balanced webserver. Let’s apply it.

Step 10: Deploy your Changes

Deploy these updates:

pulumi up

This should result in a fairly large update and, if all goes well, the load balancer’s resulting endpoint URL:

Updating (dev)

View Live: https://app.pulumi.com/beaucarnes/iac-lab2/dev/updates/5

     Type                             Name                   Status      Info
     pulumi:pulumi:Stack              iac-lab2-dev
 +   ├─ aws:lb:TargetGroup            target-group           created
 ~   ├─ aws:ec2:SecurityGroup         web-secgrp             updated     [diff: ~egress]
 +   ├─ aws:lb:LoadBalancer           loadbalancer           created
 +   ├─ aws:lb:TargetGroupAttachment  web-server-us-west-2a  created
 +   ├─ aws:lb:TargetGroupAttachment  web-server-us-west-2b  created
 +   ├─ aws:lb:TargetGroupAttachment  web-server-us-west-2c  created
 +   ├─ aws:lb:TargetGroupAttachment  web-server-us-west-2d  created
 +   └─ aws:lb:Listener               listener               created

Outputs:
    hostnames: [
        [0]: "ec2-34-222-148-4.us-west-2.compute.amazonaws.com"
        [1]: "ec2-54-69-196-240.us-west-2.compute.amazonaws.com"
        [2]: "ec2-54-186-64-129.us-west-2.compute.amazonaws.com"
        [3]: "ec2-52-25-211-116.us-west-2.compute.amazonaws.com"
    ]
    ips      : [
        [0]: "34.222.148.4"
        [1]: "54.69.196.240"
        [2]: "54.186.64.129"
        [3]: "52.25.211.116"
    ]
  + url      : "loadbalancer-c9a1e24-965935136.us-west-2.elb.amazonaws.com"

Resources:
    + 7 created
    ~ 1 updated
    8 changes. 5 unchanged

Duration: 3m28s

Step 11: Verify

Now we can curl the load balancer:

for i in {0..10}; do curl $(pulumi stack output url); done

Observe that the resulting text changes based on where the request is routed:

Hello, World -- from us-west-2a!
Hello, World -- from us-west-2a!
Hello, World -- from us-west-2d!
Hello, World -- from us-west-2b!
Hello, World -- from us-west-2b!
Hello, World -- from us-west-2c!
Hello, World -- from us-west-2b!
Hello, World -- from us-west-2a!
Hello, World -- from us-west-2c!
Hello, World -- from us-west-2d!
Hello, World -- from us-west-2a!

Step 12: Destroy Everything

Finally, destroy the resources and the stack itself:

pulumi destroy
pulumi stack rm

Lab 3: Deploying Docker images to ECS & Fargate

In this lab, you will use Pulumi to deploy a Docker Image to ECS with Fargate.

Let's start from the beginning again to get more practice.

Step 1: Create a Directory

Create a new directory and change into it:

cd ..
mkdir iac-lab3
cd iac-lab3

Pulumi will use the directory name as your project name by default. To create an independent project, simply name the directory differently.

Step 2: Initialize Your Project

A Pulumi project is just a directory with some files in it. It’s possible for you to create a new one by hand. The pulumi new command, however, automates the process:

pulumi new aws-python -y

This command has created all the files we need, initialized a new stack named dev (an instance of our project), and installed the needed package dependencies from PyPi.

Step 3: Inspect Your New Project

Our project is comprised of multiple files (these are the same as before so this is a review):

• __main__.py: your program’s main entrypoint file
• requirements.txt: your project’s Python dependency information
• Pulumi.yaml: your project’s metadata, containing its name and language
• venv: a virtualenv for your project

Run cat __main__.py to see the contents of your project’s empty program:

"""An AWS Python Pulumi program"""

import pulumi
from pulumi_aws import s3

# Create an AWS resource (S3 Bucket)
bucket = s3.Bucket('my-bucket')

# Export the name of the bucket
pulumi.export('bucket_name', bucket.id)

Feel free to explore the other files, although we won’t be editing any of them by hand.

Step 4: Configure an AWS Region

Configure the AWS region you would like to deploy to:

pulumi config set aws:region us-west-2

Step 5: Create an ECS Cluster

Remove all the boilerplate from the project bootstrap.

Import the AWS and Pulumi packages in an empty __main__.py file:

import pulumi
import pulumi_aws as aws

And now create a new ECS cluster. You will use the default values, so doing so is very concise:

After this change, your __main__.py should look like this:

import pulumi
import pulumi_aws as aws

cluster = aws.ecs.Cluster("cluster")

Step 6: Create a Load-Balanced Container Service

Next, allocate the application load balancer (ALB) and listen for HTTP traffic port 80. In order to do this, we need to find the default VPC and the subnet groups for it:

...
vpc = aws.ec2.get_vpc(default=True)
vpc_subnets = aws.ec2.get_subnet_ids(vpc_id=vpc.id)

group = aws.ec2.SecurityGroup(
    "web-secgrp",
    vpc_id=vpc.id,
    description='Enable HTTP access',
    ingress=[
        { 'protocol': 'icmp', 'from_port': 8, 'to_port': 0, 'cidr_blocks': ['0.0.0.0/0'] },
        { 'protocol': 'tcp', 'from_port': 80, 'to_port': 80, 'cidr_blocks': ['0.0.0.0/0'] }
    ],
    egress=[
        { 'protocol': "-1", 'from_port': 0, 'to_port': 0, 'cidr_blocks': ['0.0.0.0/0'] }
    ])

alb = aws.lb.LoadBalancer(
    "app-lb",
    internal="false",
    security_groups=[group.id],
    subnets=vpc_subnets.ids,
    load_balancer_type="application",
)

atg = aws.lb.TargetGroup(
    "app-tg",
    port=80,
    deregistration_delay=0,
    protocol="HTTP",
    target_type="ip",
    vpc_id=vpc.id,
)

wl = aws.lb.Listener(
    "web",
    load_balancer_arn=alb.arn,
    port=80,
    default_actions=[{"type": "forward", "target_group_arn": atg.arn}],
)

After this change, your __main__.py should look like this:

import pulumi
import pulumi_aws as aws

cluster = aws.ecs.Cluster("cluster")

vpc = aws.ec2.get_vpc(default=True)
vpc_subnets = aws.ec2.get_subnet_ids(vpc_id=vpc.id)

group = aws.ec2.SecurityGroup(
    "web-secgrp",
    vpc_id=vpc.id,
    description="Enable HTTP access",
    ingress=[
        {
            "protocol": "icmp",
            "from_port": 8,
            "to_port": 0,
            "cidr_blocks": ["0.0.0.0/0"],
        },
        {
            "protocol": "tcp",
            "from_port": 80,
            "to_port": 80,
            "cidr_blocks": ["0.0.0.0/0"],
        },
    ],
    egress=[
        {
            "protocol": "-1",
            "from_port": 0,
            "to_port": 0,
            "cidr_blocks": ["0.0.0.0/0"],
        }
    ],
)

alb = aws.lb.LoadBalancer(
    "app-lb",
    internal="false",
    security_groups=[group.id],
    subnets=vpc_subnets.ids,
    load_balancer_type="application",
)

atg = aws.lb.TargetGroup(
    "app-tg",
    port=80,
    deregistration_delay=0,
    protocol="HTTP",
    target_type="ip",
    vpc_id=vpc.id,
)

wl = aws.lb.Listener(
    "web",
    load_balancer_arn=alb.arn,
    port=80,
    default_actions=[{"type": "forward", "target_group_arn": atg.arn}],
)

Run your program with pulumi up:

Updating (dev)

View Live: https://app.pulumi.com/beaucarnes/iac-lab3/dev/updates/1

     Type                      Name          Status
 +   pulumi:pulumi:Stack       iac-lab3-dev  created
 +   ├─ aws:ecs:Cluster        cluster       created
 +   ├─ aws:ec2:SecurityGroup  web-secgrp    created
 +   ├─ aws:lb:TargetGroup     app-tg        created
 +   ├─ aws:lb:LoadBalancer    app-lb        created
 +   └─ aws:lb:Listener        web           created

Resources:
    + 6 created

Duration: 3m36s

We’ve fleshed out our infrastructure and added a LoadBalancer we can add infrastructure to, in the next steps we’ll run a container.

Step 7: Create ECS FargateService

In order to create a Fargate service, we need to add an IAM Role and a Task Definition and Service. The ECS Cluster will run the "nginx" image from the Docker Hub.

Firstly, we need to add a new import at the top of our file

import json

Now let’s define our IAM Role and attach a policy. You should define this at the end of your __main__.py:

...
role = aws.iam.Role("task-exec-role",
    assume_role_policy=json.dumps({
        "Version": "2008-10-17",
        "Statement": [{
            "Sid": "",
            "Effect": "Allow",
            "Principal": {
                "Service": "ecs-tasks.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }]
    }))

rpa = aws.iam.RolePolicyAttachment("task-exec-policy",
    role=role.name,
    policy_arn="arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"
)
...

Then we can define a task definition for our ECS service:

task_definition = aws.ecs.TaskDefinition("app-task",
    family="fargate-task-definition",
    cpu="256",
    memory="512",
    network_mode="awsvpc",
    requires_compatibilities=["FARGATE"],
    execution_role_arn=role.arn,
    container_definitions=json.dumps([{
        "name": "my-app",
        "image": "nginx",
        "portMappings": [{
            "containerPort": 80,
            "hostPort": 80,
            "protocol": "tcp"
        }]
    }])
)

service = aws.ecs.Service("app-svc",
    cluster=cluster.arn,
    desired_count=1,
    launch_type="FARGATE",
    task_definition=task_definition.arn,
    network_configuration={
        "assign_public_ip": "true",
        "subnets": vpc_subnets.ids,
        "security_groups": [group.id]
    },
    load_balancers=[{
        "target_group_arn": atg.arn,
        "container_name": "my-app",
        "container_port": 80
    }],
    opts=pulumi.ResourceOptions(depends_on=[wl])
)

pulumi.export("url", alb.dns_name)

After these changes, your __main__.py should look like this

import pulumi
import pulumi_aws as aws
import json

cluster = aws.ecs.Cluster("cluster")

vpc = aws.ec2.get_vpc(default=True)
vpc_subnets = aws.ec2.get_subnet_ids(vpc_id=vpc.id)

group = aws.ec2.SecurityGroup(
    "web-secgrp",
    vpc_id=vpc.id,
    description="Enable HTTP access",
    ingress=[
        {
            "protocol": "icmp",
            "from_port": 8,
            "to_port": 0,
            "cidr_blocks": ["0.0.0.0/0"],
        },
        {
            "protocol": "tcp",
            "from_port": 80,
            "to_port": 80,
            "cidr_blocks": ["0.0.0.0/0"],
        },
    ],
    egress=[
        {
            "protocol": "-1",
            "from_port": 0,
            "to_port": 0,
            "cidr_blocks": ["0.0.0.0/0"],
        }
    ],
)

alb = aws.lb.LoadBalancer(
    "app-lb",
    internal="false",
    security_groups=[group.id],
    subnets=vpc_subnets.ids,
    load_balancer_type="application",
)

atg = aws.lb.TargetGroup(
    "app-tg",
    port=80,
    deregistration_delay=0,
    protocol="HTTP",
    target_type="ip",
    vpc_id=vpc.id,
)

wl = aws.lb.Listener(
    "web",
    load_balancer_arn=alb.arn,
    port=80,
    default_actions=[{"type": "forward", "target_group_arn": atg.arn}],
)

role = aws.iam.Role("task-exec-role",
    assume_role_policy=json.dumps({
        "Version": "2008-10-17",
        "Statement": [{
            "Sid": "",
            "Effect": "Allow",
            "Principal": {
                "Service": "ecs-tasks.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }]
    }))

rpa = aws.iam.RolePolicyAttachment("task-exec-policy",
    role=role.name,
    policy_arn="arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"
)

task_definition = aws.ecs.TaskDefinition("app-task",
    family="fargate-task-definition",
    cpu="256",
    memory="512",
    network_mode="awsvpc",
    requires_compatibilities=["FARGATE"],
    execution_role_arn=role.arn,
    container_definitions=json.dumps([{
        "name": "my-app",
        "image": "nginx",
        "portMappings": [{
            "containerPort": 80,
            "hostPort": 80,
            "protocol": "tcp"
        }]
    }])
)

service = aws.ecs.Service("app-svc",
    cluster=cluster.arn,
    desired_count=1,
    launch_type="FARGATE",
    task_definition=task_definition.arn,
    network_configuration={
        "assign_public_ip": "true",
        "subnets": vpc_subnets.ids,
        "security_groups": [group.id]
    },
    load_balancers=[{
        "target_group_arn": atg.arn,
        "container_name": "my-app",
        "container_port": 80
    }],
    opts=pulumi.ResourceOptions(depends_on=[wl])
)

pulumi.export("url", alb.dns_name)

Step 8: Provision the Cluster and Service

Deploy the program to stand up your initial cluster and service:

pulumi up

This will output the status and resulting load balancer URL:

Updating (dev)

View Live: https://app.pulumi.com/beaucarnes/iac-lab3/dev/updates/2

     Type                             Name              Status      Info
     pulumi:pulumi:Stack              iac-lab3-dev
 +   ├─ aws:iam:Role                  task-exec-role    created
 ~   ├─ aws:lb:TargetGroup            app-tg            updated     [diff: ~deregistrationDelay]
 +   ├─ aws:iam:RolePolicyAttachment  task-exec-policy  created
 +   ├─ aws:ecs:TaskDefinition        app-task          created
 +   └─ aws:ecs:Service               app-svc           created

Outputs:
  + url: "app-lb-e92ae89-2123599743.us-west-2.elb.amazonaws.com"

Resources:
    + 4 created
    ~ 1 updated
    5 changes. 5 unchanged

Duration: 10s

You can access the URL from the output above in a web browser. Also, you can now curl the resulting endpoint:

curl $(pulumi stack output url)

And you’ll see the Nginx default homepage:

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
...

Step 9: Update the Service

Now, also update the desired container count from 1 to 3:

...
    desiredCount: 3,
...

Next update the stack:

pulumi up

The output should look something like this:

Updating (dev)

View Live: https://app.pulumi.com/beaucarnes/iac-lab3/dev/updates/3

     Type                 Name          Status      Info
     pulumi:pulumi:Stack  iac-lab3-dev
 ~   └─ aws:ecs:Service   app-svc       updated     [diff: ~desiredCount]

Outputs:
    url: "app-lb-e92ae89-2123599743.us-west-2.elb.amazonaws.com"

Resources:
    ~ 1 updated
    9 unchanged

Duration: 6s

Step 10: Destroy Everything

Finally, destroy the resources and the stack itself:

pulumi destroy
pulumi stack rm

In this article, I'll give you a brief overview of the configuration syntax of Terraform.

Terraform's docs provide the most comprehensive look at its syntax. But this article should serve as a condensed quick start introduction that'll give new users a simplified overview. We'll focus on the basic structures without getting too deep in the weeds.

To help you learn the syntax, we'll go through an example and I'll teach you the most important parts of the Terraform configuration language (which is called HCL - the HashiCorp Configuration Language). Then we'll build out some infrastructure as code (IaC) to see it in action.

Before we start, you should have Terraform installed on your local system. You should also have access to the AWS management console, and have set up and configured an IAM user for Terraform.

Arguments and Blocks in Terraform

The first example below creates an EC2 instance:

provider “aws” {
    region = “us-west-1”
}

resource “aws_instance” “myec2” {
    ami = “ami-12345qwert”
    instance_type = “t2.micro”
}

The code consists of two blocks wrapped in curly braces ( {} ), and each of these blocks has certain arguments defined.

Just like in most programming languages, you use arguments to assign values to variables. In Terraform, these variables are attributes associated with a particular type of block.

The provider “aws” block has one argument – region = “us-west-1”, where the region is an attribute associated with the block, and it is assigned a value “us-west-1”. The value is a string type, so it is enclosed in a pair of double quotes ( “” ).

Similarly, the resource block has two arguments that set the values of associated attributes.

Terraform uses various types of blocks. Based on the type, blocks represent and enclose a set of attributes and functions. In the given example, we have a block of type provider and another of type resource, and each block has an identifier and a set of input labels.

The provider block takes one input label – the name of the provider. In this case “aws”. It also tells Terraform to install the AWS provider plugin, during the init phase.

The resource block takes 2 inputs labels – the type of resource and the name of the resource. In this case, the type is “aws_instance” and the name is “myec2”. What follows is the block body enclosed in curly braces.

How to Create an EC2 Instance on AWS

So, how do we start expressing our infrastructure as code and make use of it? Let’s take an example of creating a simple EC2 instance on AWS.

Start by creating a directory of your choice where you would place all the configuration code required to create an EC2 instance.

By default, Terraform assumes that all the files with .tf* extensions in a directory are part of the configuration, irrespective of the file names. Create a file named main.tf in this directory.

The first thing we need to decide is which providers we are going to use. Since we are going to spin up an EC2 instance on AWS, we need to declare the required providers as you can see in the snippet below:

terraform {
 required_providers {
   aws = {
     source  = "hashicorp/aws"
     version = "~> 3.0"
   }
 }
}

provider "aws" {
 region = “us-west-1”
}

We have declared two blocks – terraform and provider. terraform is a top-most block, but it is optional as well. It is a good practice to specify this, especially when you're working with remote state management.

The terraform block has a nested block that specifies required_providers. We require the aws provider. aws within required_providers is a map, which specifies the source and version of the provider.

Next, we have a provider block for aws, which specifies the desired region.

Generally, this is how every Terraform code starts. Of course, there could be some variations, and the best way to be sure about it is to refer to the Terraform registry for specific versions of Terraform as well as the provider plugin itself.

For the sake of the current example, we are referring to AWS plugin documentation.

The Terraform registry documents usage of all the resources of various cloud providers with examples. It is a great reference when you're working with Terraform.

Terraform Providers

Installing Terraform on the system is not enough. Terraform works on a plugin-based architecture, where the Terraform binary forms the core and every cloud provider is a plugin.

To make configurations work, these plugins are installed in the initialization phase.

Provider plugins come with their own set of configurations, resource types, and data sources. The Terraform registry documents all the details for a given provider.

Terraform Resources

Every provider comes with a set of resources. resource, as the name suggests, represents the actual cloud resource to be created in the configuration language.

Providers enable resources. In the given example, aws is a provider and aws_instance is a resource provided by the AWS provider.

The resource has its attributes. These attributes are documented in the Terraform registry. Out of all the attributes, some of them are required. Resources are the exact constructs that are executed by Terraform.

Continuing with the example, let's define an AWS EC2 instance resource by appending the below code into our main.tf file.

resource "aws_instance" "demo" {
 ami = “ami-00831fc7c1e3ddc60”
 instance_type = “t2.micro”

 tags = {
   name = "Demo System"
 }
}

We start with a resource block named “aws_instance” and we pass a label and name it “demo”. The label is the name of your choice.

Next, open the block using curly braces and specify the required attributes used by the resource aws_instance. The first attribute is ami which specifies the Amazon machine image ID for the EC2 instance.

The second attribute is the instance_type which specifies the size of the machine to be created.

We are also passing tags which is an optional argument. As a tag, we pass “name” as the key and “Demo System” in the value. That's it – we have defined our resource.

We are now technically ready with the configuration. We can go ahead and initialize Terraform into this directory so that it installs the provider plugin for AWS. We can then plan and apply this configuration.

Save the file, and then go ahead and run terraform init and see if it installs the AWS provider plugin. Once that's done successfully, run terraform plan and observe the output.

Let's put everything into perspective. providers let Terraform know which plugins need to be installed to execute the configuration. resources represent the actual cloud resources to be created.

Generally, every resource has a name ( “aws_instance” ). The initial part of the name of the resource is the provider identifier ( “aws” ) which is separated by an underscore.

Terraform Variables

Terraform is a declarative language. In the example, we have declared the final state of the virtual machine on the desired cloud.

Now it is up to Terraform to take this configuration and execute it to create the virtual resource. Having said that, Terraform gives us the ability to specify input variables to its configuration.

Input variables are like parameters for a given function just like in any programming language.

They are particularly useful when you have to specify the same value at multiple places in your code. As the project grows in size, it becomes easier to change certain values that might be used in multiple places using variables.

Terraform supports primitive types of variables such as string, number, boolean, and several complex types such as list, set, map, object, and tuple.

Let's define some variables into our code as below:

variable "region" {
 default = "us-west-1"
 description = "AWS Region"
}

variable "ami" {
 default = "ami-00831fc7c1e3ddc60"
 description = "Amazon Machine Image ID for Ubuntu Server 20.04"
}

variable "type" {
 default = "t2.micro"
 description = "Size of VM"
}

As you can see, we have introduced three new variables for the region, the ami, and the type. We'll use this in our configuration so far. The values of the variables can be referred to using var.<variable name>.

Terraform configuration also gives us the ability to return values. These values are known as output values.

When Terraform completes the execution of the configuration, it makes the output values available. They can then be used as input to other interfaces.

We have defined one output variable “instance_id” in our code. The value of this output variable is set using the attribute reference of “aws_instance.demo”. Similarly, we can refer to other output variables available from any resource in the configuration.

Below is the updated code of our main.tf. We have used three variables here:

terraform {
 required_providers {
   aws = {
     source  = "hashicorp/aws"
     version = "~> 3.0"
   }
 }
}

provider "aws" {
 region = var.region
}

variable "region" {
 default = "us-west-1"
 description = "AWS Region"
}

variable "ami" {
 default = "ami-00831fc7c1e3ddc60"
 description = "Amazon Machine Image ID for Ubuntu Server 20.04"
}

variable "type" {
 default = "t2.micro"
 description = "Size of VM"
}

resource "aws_instance" "demo" {
 ami = var.ami
 instance_type = var.type

 tags = {
   name = "Demo System"
 }
}

output "instance_id" {
 instance = aws_instance.demo.id
}

Save the file and run terraform plan. Notice that Terraform takes note of the output variable this time. It states that the output is known after apply:

Plan: 1 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + instance_id = (known after apply)

Go ahead and do terraform apply, and check out the output. Don't forget to run terraform destroy after every successful apply.

Lastly, Terraform also supports local variables, which are temporary values used locally by functions and blocks.

Terraform Provisioners

Provisioning means to install, update, and maintain the required software once the hardware or virtual machine is ready to go.

Terraform can trigger software provisioning processes once a virtual machine is ready, but that doesn't mean it is a full-time provisioning tool.

You can use this tool to make the infrastructure ready for management by installing initial and essential software components.

There are tools like Salt Stack, Ansible, Chef, and others, and most of them are agent-based. Terraform can run initial scripts to install some patch updates, agent software, or even set some user access policies to make sure machines are ready to go.

Terraform comes bundled with generic provisioners and also supports vendor-specific provisioners.

How to Manage Code in Terraform

Before we proceed, let's first organize our code into multiple files. As a general practice, the Terraform codebase is divided into multiple files based on the providers, resources, and variables. Let's create three files as below:

1. variables.tf – This file contains all the declared input variables. In our example, we have input variables defined for region, ami, and type and an output variable instance_id.
2. provider.tf – This file contains declarations for the providers you're using. In our case, we have terraform, and the provider aws blocks.
3. main.tf – This file contains the declarations for actual resources to be created.

By default, Terraform assumes that all the code placed in a particular directory is part of the same configuration.

So technically it doesn't make much of a difference if you put the code in a single file or divide it into multiple files and sub-directories. From a maintainability point of view, it makes a lot of sense to do so.

Meta-Arguments in Terraform

Before we get into this, make sure that when you're working through the examples, always run terraform destroy after every terraform apply run.

Meta-arguments are special constructs provided for resources. We have seen that resource blocks are the actual cloud resources that Terraform creates.

Often, it's tricky to declare resources in a way that satisfies certain requirements.

Meta-arguments come in handy in situations like creating resources in the same cloud provider but in different regions. They're also useful when we are creating multiple identical resources with different names, or when we have to declare implicit dependencies at places where Terraform is not able to identify the dependency itself.

The sections below describe some meta-arguments in action.

The Provider meta-argument

You'll use the provider meta-argument when you have multiple provider configurations in a given Terraform config. Terraform automatically maps the given resource to the default provider identified by the resource’s identifier.

For example, the default provider for aws_instance is aws. This aws provider is currently configured to deploy a resource in a particular region.

However, if we want to have another aws provider for another region, or with a different configuration setting, we can write another provider block.

Even though it's possible to write multiple provider configs, Terraform by default picks the same provider for aws for creating resources.

This is where aliases come into the picture. Every provider configuration can be tagged with an alias, and the value of this alias is used in our provider meta-argument in the resource block to specify different provider configurations for identical resources.

In our example, let's duplicate the aws provider and give them appropriate aliases. Modified providers with an alias should look like the below in the provider.tf file:

provider "aws" {
  alias  = "aws_west"
  region = var.region_west
}

provider "aws" {
  alias  = "aws_east"
  region = var.region_east
}

Notice that we have also modified variables for the region to represent two different regions, west and east. Make the corresponding changes to the variables.tf file as below:

variable "region_west" {
  default     = "us-west-1"
  description = "AWS West Region"
}

variable "region_east" {
  default     = "us-east-1"
  description = "AWS East Region"
}

One final change that we need to make is in the main.tf file. There, we can now use provider meta-argument to specify a specific provider's alias.

We can mention the provider config we want by specifying <provider>.<alias> in the meta-argument. Refer to the modified main.tf file below:

resource "aws_instance" "demo" {
  provider      = aws.aws_west
  ami           = var.ami
  instance_type = var.type

  tags = {
    name = "Demo System"
  }
}

Validate the final configuration by running terraform validate, and it should say “Success!”

The Lifecycle meta-argument

The lifecycle meta-argument specifies the settings related to the lifecycle of resources managed by Terraform. By default, whenever a configuration is changed and applied, Terraform operates in this sequence:

1. Create new resources.
2. Destroy those resources which do not exist in config anymore.
3. Update those resources which can be updated without destruction.
4. Destroy and re-create changed resources that cannot be changed on the fly.

You can use a lifecycle meta-argument if you want to alter this default behavior. These meta-arguments are used in resource blocks similar to provider meta-arguments.

There are three lifecycle meta-argument settings:

1. create_before_destroy: Used when you want to avoid accidental loss of infrastructure when a changed config is applied. When this is set to true, Terraform will first create the new resource before destroying the older resource.
2. prevent_destroy: When set to true, any attempt to destroy this in the config will result in an error. This is often useful in the case of those resources where reproduction can prove to be expensive.
3. ignore_changes: This is a list type meta-argument which specifies the attributes of a certain resource in the form of a list. During update operations, often there is a situation where you want to prevent changes caused by external factors. In those cases, you need to declare the list of attributes that should not be changed without being reviewed.

lifecycle meta-arguments come in handy when we are in the process of setting up complex infrastructure.

By altering the default behavior of Terraform, we can put some protection in the form of lifecycle meta-arguments for confirmed and finalized resource blocks.

In our example, we would not use any lifecycle meta-arguments.

The depends_on meta-argument

Generally, Terraform is aware of dependencies while creating or modifying resources and takes care of the sequence by itself.

However, in certain cases Terraform can't detect the implicit dependencies and just moves on with creating resources in parallel if it doesn’t see any dependency.

Take, for example, a Terraform configuration for two EC2 instances enclosed in a VPC.

When you have this configuration, Terraform automatically knows that it shoudl create the VPC before spinning up the EC2 instances.

This is general knowledge and Terraform knows it very well. In situations where dependencies are not so obvious, the depends_on meta-argument comes to the rescue.

It is a list type of argument that takes in the list of resource identifiers declared in the configuration.

The count meta-argument

Imagine a situation where you would like to create multiple similar resources. By default, Terraform creates one real resource for a single resource block.

But in the case of multiple resources, Terraform provides a meta-argument named count. As the name suggests, the count can be assigned with a whole number to represent multiple resources.

In our example, let's create three similar EC2 instances. In your main.tf file, add an attribute count to the resource aws_instance.demo, and assign it a value of 3. It should look something like the below:

resource "aws_instance" "demo" {
  count         = 3
  provider      = aws.aws_west
  ami           = var.ami
  instance_type = var.type

  tags = {
    name = "Demo System"
  }
}

By doing this, we let Terraform know that we need to create three EC2 instances with the same configuration.

Save the file and execute terraform validate. It throws an error saying “Missing resource instance key”.

Remember in our variables.tf file we mentioned an output variable to output the id of the created resource. Since we have asked Terraform to create three instances, it is not very clear which ID it should print.

To get around this problem, we use a special expression called a splat expression.

The ideal case here is to run a for loop over the instance and print out the ID property. The splat expression is a better way to do the same task with fewer lines of code.

All you need to do is in the variables.tf file, replace the output value code with the below:

output "instance_id" {
  value = aws_instance.demo[*].id
}

Save this file and run terraform validate to see if everything is okay.

Once successful, go ahead and run terraform plan and apply and check your AWS management console in us-west-1 region, that is aws_west. Let me know the IDs too.

Splat is one of a kind and we will take a better look at expressions in upcoming sections.

The for_each meta-argument

for_each, as the name suggests, is essentially a “for each” loop. for_each meta-argument is used to create/loop through multiple similar cloud resources. Yes, it does sound similar to the count meta-argument but there is a difference.

Firstly, for_each and count cannot be used together.

Secondly, you can say this is an enhanced version of the count. Count meta-argument is a number type. Terraform simply creates this many resources.

However, if you would like to create these resources with some customizations in the output, or if you already have an object of type map or list based on which you want to create resources, then the for_each meta-argument is the way to go.

As mentioned earlier, you can assign for_each map and list type of values. A map is a collection of key-value pairs, whereas a list is a collection of values (in this case string values).

for_each comes with a special object each. This is the iterator in the loop which you can use to refer to the key or value, or only the key in case of list.

Let's take a look at our example. We would like to create EC2 instances for the given map. The map is assigned to the for_each meta-argument and Terraform creates an EC2 instance for each key-value pair in the map.

Lastly, we use the key and value information using each object to set the name attribute in the tag.

The resource block in main.tf now looks something like this:

resource "aws_instance" "demo" {
  for_each = {
    fruit = "apple"
    vehicle = "car"
    continent = "Europe"
  }
  provider      = aws.aws_west
  ami           = var.ami
  instance_type = var.type

  tags = {
    name = "${each.key}: ${each.value}"
  }
}

Execute terraform validate and observe the output. It throws an error for the output variable – “This object does not have an attribute named id”.

A quick note here: splat expressions work for the list type of variables. Since we used map while setting our for_each meta-argument, we need to change the return value expression to for each, as below:

output "instance_id" {
  //value = aws_instance.demo[*].id
  value = [for b in aws_instance.demo : b.id]
}

Execute terraform validate again. If it's successful, go ahead and apply the configuration. Check the AWS management console for the machines created and the names assigned to them.

Terraform Expressions

Expressions are ways to make your Terraform code dynamic.

Expressions come in two forms – simple and complex. Up until now in our examples, we have mostly dealt with simple expressions.

A simple expression is any argument used as part of some block. Writing down an argument with a primitive value assigned to it is a form of expression.

We have used a complex expression called splat ( * ) in our example while working with meta-arguments.

However, there are even more complex expressions that you can use to make your Terraform code more dynamic, readable, and flexible. There are various types of expressions that you can take a look at in the Terraform documentation.

Terraform Functions

Terraform has built-in functions that you can use with expressions. These are utility functions that are useful in number and string manipulations.

There are functions to work with file systems, date and time, network, type conversion, and more.

Functions along with expressions make it super easy to write a really dynamic IaC. You can refer to the list of functions here.

Conclusion

In this post, we were able to play around with AWS EC2 instances using Terraform. I hope this helps you get more comfortable working with Terraform.

If you like this content, do consider subscribing, following, and sharing this blog post! Let'sDoTech, Instagram, Twitter, LinkedIn.

Terraform is a tool that helps you manage various cloud infrastructure services in the form of code. You codify your infrastructure, and so it's also known as Infrastructure as Code (IaC).

The cloud has become important to more and more companies. It not only helps reduce time and costs but also lets customers focus on their core business.

Why Infrastructure as Code?

As the number of cloud providers increases and their services become more flexible, it's becoming more important to be able to manage your cloud infrastructure resources.

Terraform works on the concept of Infrastructure as Code (IaC). In simple terms, IaC is the ability to represent your infrastructure in the form code.

Let's take as an example any computing resource on a given cloud, like EC2 on AWS. Requesting an EC2 instance from AWS is a matter of signing up with AWS, providing a bunch of values, and clicking on the “Launch” button. The "resource" will be ready in a few minutes.

As long as we can provide those values to AWS, they'll live on that cloud provider. Of course, this is the traditional way to do this.

Terraform provides a way to take these credentials and inputs in the form of configurations and process them to create a resource in the target cloud.

These configurations describe the resource in a language that Terraform understands. Configurations are how you can declare the desired state of your infrastructure – basically, "Declarative" syntax.

Terraform uses cloud provider APIs to create the resource.

Advantages of Terraform

Terraform is a product from Hashicorp, and uses Hashicorp Configuration Language (HCL) syntax to represent the configurations.

In the example below, you can see the representation of the EC2 instance in its simplest form:

provider “aws” {
    region = “us-west-1”
}

resource “aws_instance” “myec2” {
    ami = “ami-12345qwert”
    instance_type = “t2.micro”
}

This simple example is enough for us to understand the capabilities of Terraform.

The code contains two blocks – the provider and resource. The provider block lets Terraform know that we want to use aws provider in the region “us-west-1”.

The resource block lets Terraform know that out of all the infrastructure resources offered by AWS, we want to create a resource of type “instance” (EC2).

The first parameter represents this to the resource block as “aws_instance”. The second parameter is what we've named the resource – in this case, “myec2”.

The resource block has a couple of arguments that state the AWS machine image and the type of instance used to create this resource.

Here, we have managed to express our infrastructure in the form of code. Let's go through some of the advantages of IaC.

1. Since infrastructure creation is now condensed in config/code files, it is easier to maintain since we can now leverage version control systems like Git to collaborate and maintain it.
2. The time required for the planning phase of the infrastructure is reduced as we can write the configurations in a short amount of time. These configs are readily consumed by Terraform to create cloud resources in the matter for few minutes.
3. Changes to the infrastructure are easier and are comparable to application code changes.
4. The advantages for application management lifecycle in the case of software development are also applicable for infrastructure. This makes it more efficient.

Features of Terraform

Orchestration

When deploying various end-to-end services, Terraform acts as the core of the orchestration process when it comes to creating cloud resources.

Cloud agnostic

Since Terraform supports most of the clouds including AWS, MS Azure, and GCP, you don't have to worry as much about vendor lock-in issues. The Terraform registry provides the documentation for all the supported cloud providers.

Syntax patterns used to code infrastructure on various clouds are the same, so the learning curve related to provider-specific APIs is on a back burner, but not forgotten.

Declarative syntax

Infrastructure expressed in Terraform files is declarative – so as developers, we don’t need to worry about making Terraform understand the steps required to create a resource. Rather, all we need to do is let Terraform know about the desired state and Terraform takes care of the steps internally.

Modules

Terraform provides modules which help us reuse our Terraform code. A complex infrastructure is broken into multiple modules and each module is reusable in different projects.

It is very easy to convert a given Terraform configuration into modules and Terraform has its eco-system for pre-built modules.

State management

While Terraform is creating and planning the infrastructure, state is maintained. This can be shared with other team members for collaboration purposes.

Terraform lets you manage state remotely, which helps prevent confusion amongst team members in case they attempt to recreate the infrastructure.

Provisioning

Terraform is not a full-blown provisioning tool, but it helps with day one provisioning activities. Terraform’s local-exec and remote-exec blocks let you run inline scripts. Inline scripts help install software components upon the successful creation of the resource.

This is especially useful when helping configuration management tools like Chef, Ansible, and Salt Stack install their respective agents. They can just send an “UP” signal once they are installed successfully.

Open Source

Terraform is available for use as open-source software. It also has an Enterprise version.

Terraform Workflow [init - plan - apply - destroy]

There are some simple steps you need to take to execute your Terraform code. These steps are closely related to the lifecycle of resources on cloud platforms.

Again, these steps are cloud-agnostic, meaning the same steps/commands are valid to create, update, and destroy resources on any given cloud provider.

Note: This blog post doesn’t cover installation steps for Terraform, and I assume you already have the Terraform CLI installed on your system.

Run the init command

Once we have the configuration files ready, the very first command we need to run is terraform init. The Terraform binary installation does not include support for all the cloud providers at once.

Instead, based on the provider, appropriate plugins are downloaded before Terraform executes the code. In our example, running terraform init would download the aws provider plugin. This command helps initialize the backend for a given Terraform directory.

Generate an execution plan

The terraform plan command helps generate an execution plan. Based on the configuration you provide, Terraform generates an execution plan. In this phase, Terraform performs feasibility checks in terms of syntax errors, API authentication, state verification, and more.

plan highlights any fixes in the Terraform script before actual execution. If it's successful, it outputs a summary of potential changes in the infrastructure. You should run this before apply, as it makes you aware of any risks before modifying the infrastructure.

Apply the changes

terraform apply helps to execute any changes to the infrastructure. You should run the plan command before running terraform apply, as planning creates a plan file which is referred to during apply phase.

However, if in case terraform apply is executed directly, a new plan file will be created automatically.

Destroy resources

Lastly, terraform destroy helps destroy any resources which are part of the current configuration/state.

Terraform in Action

Okay, enough theory for this post. Let's try to implement what we have learned so far by actually creating an instance of EC2 on AWS.

First, install the Terraform CLI if you haven't already. Installation is pretty easy and you can find the steps here for the OS of your choice.

Create a directory/folder on your system and create the first Terraform file. Name it main.tf. Ideally, we can name it anything as long as it has the extension .tf.

Terraform CLI recognizes all the files with a .tf extension saved in a particular directory for execution.

Paste the above code in this file and save it. Please note that you need to use the correct AMI based on your preferred region.

Since this will be the first time we are executing the Terraform code, we need to initialize Terraform in this directory. Running terraform init will install the required aws plugin.

Fire up a terminal, navigate to the directory where our Terraform file resides, and run the below command.

terraform init

This should generate the output as below. The output is very clear and we can see that the aws plugin was installed with version v3.22.0.

Initializing the backend...
Initializing provider plugins...
- Reusing previous version of hashicorp/aws from the dependency lock file
- Installing hashicorp/aws v3.22.0...
- Installed hashicorp/aws v3.22.0 (signed by HashiCorp)
Terraform has been successfully initialized!
You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

Next, we will run the terraform plan command to see a tentative execution plan. This also validates against any syntactical or reference errors. The plan command checks the feasibility of the declared resources in the main.tf file. Run this in the same terminal.

terraform plan

If everything worked well, the following output is generated.

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_instance.example will be created
  + resource "aws_instance" "myec2" {
      + ami                          = "ami-12345qwert"
      + arn                          = (known after apply)
      + associate_public_ip_address  = (known after apply)
. . .
Plan: 1 to add, 0 to change, 0 to destroy.

------------------------------------------------------------------------

Note: You didn't specify an "-out" parameter to save this plan, so Terraform
can't guarantee that exactly these actions will be performed if
"terraform apply" is subsequently run.

The plan command indicates which resources will be created. In our case, it plans to create a myec2 instance with the given configuration. It shows the AMI ID being used to create the instance.

Additionally, it also indicates that other attributes like arn and associate_public_ip_address are known after apply, that is when the instance will be created.

The bottom line here indicates the final set of changes – to add 1 resource and nothing to change or destroy.

So, everything looks good as of now. Let's go ahead and apply the configuration. Run the below command in the terminal and observe the output.

terraform apply

Once confirmed, it takes a few seconds to complete the creation of the EC2 instance on AWS, and this is indicated by the output being generated by terraform apply.

As indicated by the output below, in my case it took 51 seconds to create an EC2 instance and the ID of the instance is made available as well.

Verify the same by logging into your AWS console and searching for an EC2 instance with the below ID. If everything has worked well, you should be able to find it.

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

aws_instance.myec2: Creating...
aws_instance.myec2: Still creating... [10s elapsed]
aws_instance.myec2: Still creating... [20s elapsed]
aws_instance.myec2: Still creating... [30s elapsed]
aws_instance.myec2: Still creating... [40s elapsed]
aws_instance.myec2: Still creating... [50s elapsed]
aws_instance.myec2: Creation complete after 51s [id=i-04ef3120a0006a153]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Thus, we have successfully used IaC to define/declare and create our virtual machine configuration on AWS.

If we don't need this virtual machine anymore, we can use the same configuration to destroy it.

Please note that if we make any changes to the configuration without the intention of applying those changes – and then we try to destroy the same – we may run into errors.

This is because Terraform maintains the relationship between configuration and real-world resources in state files. Changing the configuration with its application will affect the alignment and Terraform will treat this as new resources to be created.

Terraform states is a topic that deserves its own post, so we shall cover that later. For now, to destroy the EC2 instance, run the below command in the terminal.

terraform destroy

Before destroying the resource, Terraform asks for our confirmation by providing a plan output. It indicates that running the destroy command will delete 1 resource, which is exactly what we expect.

Terraform destroy
Plan: 0 to add, 0 to change, 1 to destroy.

Do you really want to destroy all resources?
  Terraform will destroy all your managed infrastructure, as shown above.
  There is no undo. Only 'yes' will be accepted to confirm.

  Enter a value: yes

aws_instance.myec2: Destroying... [id=i-04ef3120a0006a153]
aws_instance.myec2: Still destroying... [id=i-04ef3120a0006a153, 10s elapsed]
aws_instance.myec2: Still destroying... [id=i-04ef3120a0006a153, 20s elapsed]
aws_instance.myec2: Still destroying... [id=i-04ef3120a0006a153, 30s elapsed]
aws_instance.myec2: Still destroying... [id=i-04ef3120a0006a153, 40s elapsed]
aws_instance.myec2: Still destroying... [id=i-04ef3120a0006a153, 50s elapsed]
aws_instance.myec2: Still destroying... [id=i-04ef3120a0006a153, 1m0s elapsed]
aws_instance.myec2: Destruction complete after 1m5s

Destroy complete! Resources: 1 destroyed.

Again, it takes a few seconds to destroy the resource. Terraform doesn't keep you hanging as it updates the status in a 10-second interval.

Once the resource is destroyed, it confirms that it's done. Feel free to log in to the AWS console and verify if the resource is terminated.

Thanks for reading!

I hope you understand how Terraform works from this basic introduction. I will write more posts covering deeper concepts like states, syntax, the CLI, the back end, and so on in future posts.

If you like this content, do consider subscribing, following, and sharing this blog post! Let'sDoTech, Instagram, Twitter, LinkedIn.

In my previous article, I wrote about metrics and how they help you gain visibility into the operational health of your hardware and software systems.

Wavefront is a high-performance streaming analytics platform that supports 3D observability (metrics, histograms, traces/spans).

It can scale to very high data ingestion rates and query loads. You can collect data from many services and sources across your entire application stack, and can look at details for earlier data collected by Wavefront.

Terraform is an open-source “Infrastructure as Code” tool, created by HashiCorp.

It is a declarative coding tool and enables developers to use a high-level configuration language called HCL (HashiCorp Configuration Language) to describe the desired “end-state” for the infrastructure.

This infrastructure can be on the cloud or on-premises. It then generates a plan for reaching that end-state and executes the plan to create the infrastructure.

In this article, we will take a look at how we can use Terraform to write code that will automatically build dashboards and alarms in Wavefront. This is really helpful in maintaining a DevOps culture in your team, where all the monitoring infrastructure is maintained as code and checked into your version control system such as GitHub.

How to Install Terraform

Depending on your OS, the installation instructions for Terraform will vary. This article covers the instructions for installing it on macOS.

The recommended approach for installing it on macOS is to use the Homebrew package manager.

Install Terraform

Verify that you have Homebrew installed, like this:

$ brew --version

Homebrew/homebrew-core (git revision fe68a; last commit 2020-10-15)
Homebrew/homebrew-cask (git revision 4a2c25; last commit 2020-10-15)

If not, you can install Homebrew using the following command:

$ /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"

Next, install Terraform using the following commands:

$ brew tap hashicorp/tap
$ brew install hashicorp/tap/terraform

Verify Terraform Installation

To verify that Terraform is properly installed, open another terminal session, and try a Terraform command.

$ terraform --help

Usage: terraform [global options] <subcommand> [args]

The available commands for execution are listed below.The primary workflow commands are given first, followed byless common or more advanced commands.

How to Get an API Token

To allow Terraform to access your Wavefront installation, you will need to provide it an access token. This token can be found in the API tokens section of your account.

Go to Gear Icon > Account Name > API Access

Getting your Wavefront API Token

How to Setup a Terraform Project

First, create a new folder for your Terraform project:

$ mkdir wavefront-terraform

A usual Terraform project contains 3 major files:

1. versions.tf — this contains the Terraform provider declaration that specifies the plugin version to be used
2. variables.tf — this contains the variables that you can refer to in your main Terraform code
3. main.tf — as the name suggests, this contains the actual code required to build the resources

Create a versions.tf file in the project folder and add the following code:

Next, run the terraform init command to initialize the Wavefront provider:

$ terraform init

This downloads the terraform-wavefront-provider-<version> file and puts it inside a.terraform folder in the current project folder.

Next, create a main.tf file in the project folder and add the following code:

With the setup completed, we are now ready to create some dashboards and alerts.

How to Create Wavefront Dashboards

Before we jump into creating dashboards, let us first understand the anatomy of a Wavefront dashboard.

Anatomy of a Wavefront Dashboard

A dashboard in Wavefront consists of 5 types of entities:

• Dashboard — This is the main dashboard and contains all other entities.
• Section — A dashboard can contain one or more sections. A section is a logical group of charts. For example you can have one section for displaying charts related to hardware utilization, and another section for displaying charts related to API calls.
• Row — A row is a collection of charts. You can define the number of charts you want to be present in a row. My personal recommendation is to have 3 charts in a row. Anything more than that clutters the dashboard.
• Chart — This is the final chart that displays the metrics on the dashboard. There are various options for creating charts like line charts, bar charts, pie charts, and so on.
• Source — A chart can contain one or more sources. Each source has a query that works on an underlying metric to create a visual representation on the chart.

Now we are ready to write some code to create a dashboard. Add the following code to the main.tf file:

This creates a dashboard with one section for EC2 Metrics. There is one row in this section with two charts. One chart displays CPU Utilization and the other displays Memory Utilization. Both of them are line charts and show the percentage of usage.

How to Create Alerts

The dashboard we created is great to look at the CPU and Memory utilization of our EC2 instances. But if we want to be notified when the CPU or memory utilization increases beyond a certain threshold, we need to set up some alerts.

To create an alert on CPU Utilization, add the following code to the main.tf file:

This creates two resources:

1. An alert target that sends an email to the specified address whenever an alert is opened or resolved.
2. An alert on CPU Utilization that fires when the CPU utilization crosses the given threshold (a WARN alert when it goes over 60% and a SEVERE alert when it goes over 80%).

Wavefront continuously monitors the CPU utilization and sends a notification to the email address when the threshold is breached. Similarly, when the utilization becomes normal, it sends another notification indicating that things have recovered.

How to Generate Resources in Wavefront

The code for creating our resources is ready. Now we need to apply them so that the actual resources are created on Wavefront.

To view what changes will be made to Wavefront by our code, run the following command:

$ terraform plan

This will verify our code and show the difference between the current setup in Wavefront, and the changes that will happen due to your code.

Finally, to create the resources on Wavefront, run the following command:

$ terraform apply -auto-approve

This will upload the configuration to Wavefront and create the actual dashboard and alert. You can now go to Wavefront and verify these resources.

Conclusion

Congratulations! You just created a new Wavefront dashboard and alert through code.

You can now go ahead and make any modifications to your code, and run terraform apply -auto-approve to apply your changes to Wavefront.

Terraform is a great way to maintain your resources in the form of code that can be checked into your version control system. This allows multiple developers to work on your resources while also keeping track of the changes.

The full source code for this tutorial can be found here.

Thank you for staying with me so far. Hope you liked the article. You can connect with me on LinkedIn where I regularly discuss technology and life. Also take a look at some of my other articles. Happy reading. 🙂