Storing them is straightforward. But handling rotation, stale env vars, and the gap between what your pod reads and what AWS actually holds is where many engineers go quiet.

In this guide, you'll build a complete secrets pipeline from AWS Secrets Manager into Kubernetes pods. You'll provision the infrastructure with Terraform, sync secrets using the External Secrets Operator, and run a sample application that reads the same credentials in two different ways: via environment variables and via a volume mount.

By the end, you'll be able to:

• Explain the full architecture from vault to pod

• Run the lab locally in about 15 minutes

• Prove why environment variables go stale after rotation, while mounted secret files stay fresh

• Deploy the same pattern on Amazon Elastic Kubernetes Service with OpenID Connect-based CI/CD

• Troubleshoot the most common failures

Below is an architecture diagram showing secrets flowing from AWS Secrets Manager through the External Secrets Operator into a Kubernetes Secret, then splitting into environment variables set at pod start and a volume mount that updates within 60 seconds.

Table of Contents

• Prerequisites

• How to Understand the Secret Flow

• How to Run the Local Lab

• How to Inspect the ExternalSecret and the Application

• How to Test Secret Rotation

• How to Choose Between External Secrets Operator and the CSI Driver

• How to Deploy the Pattern on Amazon Elastic Kubernetes Service

• How to Configure GitHub Actions Without Stored AWS Credentials

• How to Troubleshoot the Most Common Failures

• Conclusion

Prerequisites

Before you begin, make sure you have the following tools installed and configured.

For the local lab:

• An AWS account with access to AWS Secrets Manager

• The AWS CLI installed and configured. Run aws configure and provide your access key, secret key, default region, and output format. The credentials need permission to read and write secrets in AWS Secrets Manager.

• kubectl installed. For Microk8s, run microk8s kubectl config view --raw > ~/.kube/config after installation to connect kubectl to your local cluster.

• Terraform installed

• Helm installed

• Docker installed

• A local Kubernetes cluster: the lab supports Microk8s and kind. If you do not have either installed, follow the Microk8s install guide before continuing.

For the Amazon Elastic Kubernetes Service sections:

• An Amazon Elastic Kubernetes Service cluster you can create or manage

• A GitHub repository you can configure for workflows and secrets

The lab repository includes two deployment paths: a local path for fast learning and an Amazon Elastic Kubernetes Service path for a production-like setup. All the exact commands for each path live in the repo's docs/DEPLOY-LOCAL.md and docs/DEPLOY-EKS.md.

How to Understand the Secret Flow

Before you run any command, you need to understand how the pieces connect.

The flow has four stages:

1. A developer or automated system updates a secret in AWS Secrets Manager.

2. The External Secrets Operator polls AWS Secrets Manager on a schedule and creates or updates a Kubernetes Secret.

3. Your pod reads that Kubernetes Secret.

4. During rotation, the Kubernetes Secret updates, but your two consumption modes behave differently.

How the External Secrets Operator Sync Works

The External Secrets Operator reads a custom Kubernetes resource called ExternalSecret. That resource tells the operator three things:

• Which secret store to connect to

• Which Kubernetes Secret name to create or update

• How often to refresh

In this lab, the ExternalSecret creates a Kubernetes Secret named myapp-database-creds. The operator also adds a template annotation that can trigger a pod restart when the secret rotates.

How the App Consumes Secrets

The sample application exposes three endpoints so you can validate behavior at any time.

• /secrets/env shows what environment variables the pod sees

• /secrets/volume shows what files in the mounted secret directory look like

• /secrets/compare compares both and reports whether rotation has been detected

The app checks four keys: DB_USERNAME, DB_PASSWORD, DB_HOST, and DB_PORT.

How to Run the Local Lab

The local lab gives you a fast learning loop. You can see the full pipeline working and test rotation without waiting for a cloud deployment.

Step 1: Clone the Repo

git clone https://github.com/Osomudeya/k8s-secret-lab
cd k8s-secret-lab

Step 2: Run the Spin-Up Script

bash spinup.sh

The script will ask you to choose a local cluster type. Pick Microk8s or kind, depending on what you have installed. The script installs the External Secrets Operator via Helm, applies the Terraform configuration, and deploys the sample application.

If the script fails at any point, check docs/TROUBLESHOOTING.md before retrying. The most common causes are missing AWS credentials, a misconfigured kubeconfig, or a Microk8s storage add-on that is not enabled.

Important: Run the Lab UI

The lab ships with a separate guided tutorial interface that runs on your laptop. This is not the in-cluster application, it's a React-based checklist at lab-ui/ that walks you through each concept and checkpoint as you work through the lab.

To start it, open a second terminal and run:

cd lab-ui && npm install && npm run dev

Then open http://localhost:5173. You'll see a module-by-module guide covering the full flow from external secrets to rotation to CI/CD.

Keep this terminal running alongside your lab. The Lab UI and the in-cluster app (localhost:3000) are two separate things, the UI guides you through the steps, the app shows you the live secrets.

Step 3: Access the Application

Once the lab finishes, port-forward the service.

kubectl port-forward svc/myapp 3000:80 -n default

Open http://localhost:3000. You should see a table showing each secret key and whether the environment variable value matches the volume mount value.

Step 4: Validate That Secrets Match

Run the compare endpoint directly from the terminal.

curl -s http://localhost:3000/secrets/compare | python3 -m json.tool

When everything is working, the response will include "all_match": true.

How to Inspect the ExternalSecret and the Application

At this point the lab is running. Now you'll want to inspect the manifests so you understand what each part does.

Step 1: Read the ExternalSecret Manifest

Open k8s/aws/external-secret.yaml. Focus on these four fields:

• refreshInterval: how often the operator polls AWS Secrets Manager

• secretStoreRef: which store the operator authenticates against

• target: the name of the Kubernetes Secret to create

• data: the mapping from AWS Secrets Manager JSON keys to Kubernetes Secret keys

Here is what that mapping looks like in this lab:

spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: myapp-database-creds
    creationPolicy: Owner
  data:
    - secretKey: DB_USERNAME
      remoteRef:
        key: prod/myapp/database
        property: username

The property field tells the operator which JSON key inside the AWS secret to extract. If your secret in AWS Secrets Manager is a JSON object, each field gets its own entry here.

Two fields here are worth understanding before you move on. creationPolicy: Owner means the operator owns the Kubernetes Secret it creates. If you delete the ExternalSecret, the Secret is deleted too. ClusterSecretStore is a cluster-scoped store, meaning any namespace in the cluster can use it. A plain SecretStore is namespace-scoped. For this lab, cluster-scoped is the right choice because it keeps the setup simple.

Step 2: Read the Deployment Manifest

Open k8s/aws/deployment.yaml. You are looking for two sections: envFrom and volumeMounts.

envFrom:
  - secretRef:
      name: myapp-database-creds

volumeMounts:
  - name: db-secret-vol
    mountPath: /etc/secrets
    readOnly: true

Both paths read from the same Kubernetes Secret, myapp-database-creds. The envFrom block injects all keys as environment variables at pod start.
The volumeMounts block mounts the same secret as files under /etc/secrets.

This is the core of the rotation lesson. Both paths read the same source. But they behave differently after that source changes.

Step 3: Read the App Comparison Logic

Open app/server.js. The comparison logic reads environment variables from process.env and reads mounted secret files from /etc/secrets/<key>. Then it computes a per-key match and a global all_match value.

The /secrets/compare endpoint sets rotation_detected: true when any key differs between env and volume.

How to Test Secret Rotation

Secret rotation is where real teams feel pain. This lab makes that pain visible so you can explain it clearly and fix it confidently.

How the Rotation Gap Works

When a pod starts, Kubernetes gives it two ways to read a secret.

The first way is environment variables. Think of these like sticky notes written on the wall of the container the moment it boots up. The value gets written once, at startup, and never changes. Even if the secret in AWS gets updated ten minutes later, the sticky note still says the old value. The container cannot see the update because nobody rewrote the note.

The second way is a volume mount. Think of this like a shared folder that someone else can update remotely. Kubernetes creates a small folder inside the container and puts the secret value in a file there. When the secret changes in AWS and ESO syncs it into Kubernetes, the kubelet quietly updates that file within about 60 seconds. The container reads the file fresh every time it needs the value, so it sees the new password automatically.

Same secret, two paths. One goes stale while one stays fresh.

The problem happens when your app reads the database password from the environment variable, the sticky note, and someone rotates the password in AWS. ESO updates Kubernetes. The file gets the new password. But your app is still reading the sticky note, which has the old one. Connection fails.

That difference isn't a bug. It's how the Linux process model and the kubelet work. Understanding it is the difference between knowing Kubernetes secrets and actually operating them.

Here is what you're about to observe in the lab:

• The rotation script updates the secret in AWS

• ESO syncs the new value into Kubernetes within seconds

• The volume file updates automatically

• The environment variable stays stale until the pod restarts

• The /secrets/compare endpoint shows both values side by side so you can see the gap live

Step 1: Confirm the Lab Is Ready

Make sure your pod and the External Secrets Operator are both running before you start.

kubectl get pods -n external-secrets
kubectl get pods -n default

Both should show Running.

Step 2: Run the Rotation Test Script

bash rotation/test-rotation.sh

The script performs these actions in order:

1. Reads the current DB_PASSWORD from the volume mount at /etc/secrets/DB_PASSWORD

2. Reads the current DB_PASSWORD from the environment variable

3. Updates AWS Secrets Manager with a new password using put-secret-value

4. Forces an immediate ESO sync by annotating the ExternalSecret with force-sync

5. Reads the volume value again

6. Reads the environment variable again

After the script runs, the volume and the env var will show different values.

Step 3: Validate With the Compare Endpoint

Hit the compare endpoint and look at the output.

curl -s http://localhost:3000/secrets/compare | python3 -m json.tool

You'll see something like this:

{
  "comparison": {
    "DB_PASSWORD": {
      "env": "old-password-value",
      "volume": "new-password-value",
      "match": false
    }
  },
  "all_match": false,
  "rotation_detected": true,
  "message": "Volume has new value; env still has old value."
}

Step 4: Restart the Deployment to Sync Env Vars

Env vars don't update in place. You need a pod restart so new containers start with the updated Kubernetes Secret.

kubectl rollout restart deployment/myapp -n default
kubectl rollout status deployment/myapp -n default

Then hit /secrets/compare again. All rows should now show "all_match": true.

How to Automate Restarts With Reloader

If you don't want to restart deployments manually after every rotation, you can install Stakater Reloader. It watches an annotation on the Deployment and triggers a rolling restart automatically when the referenced Kubernetes Secret changes. New pods start with fresh env vars, while old pods drain cleanly. The repo's local deployment guide includes the install steps.

How to Choose Between External Secrets Operator and the CSI Driver

Two patterns dominate when it comes to pulling external secrets into Kubernetes: the External Secrets Operator and the Secrets Store CSI Driver.

Both get cloud secrets into pods, but they do it differently. Here's a plain comparison:

This lab uses the External Secrets Operator for two reasons. First, it produces a native Kubernetes Secret, which means your application and deployment patterns match standard Kubernetes workflows. Second, having both envFrom and a volume mount point to the same Secret makes the rotation behavior easy to observe side by side.

Use the CSI Driver when your security team prohibits storing secrets in etcd through a Kubernetes Secret. The driver mounts secret data directly into the pod file system without creating a Kubernetes Secret. The tradeoff is that you lose the native envFrom model.

How to Deploy the Pattern on Amazon Elastic Kubernetes Service

The local lab is ideal for learning. The Amazon Elastic Kubernetes Service path adds the production-like pieces: IAM role-based permissions for the operator, a load balancer for the app, and a full CI/CD workflow.

Step 1: Prepare Terraform and OpenID Connect Access

The repository includes a one-time setup guide for OpenID Connect-based access from GitHub Actions to AWS. Run these commands in the terraform/github-oidc folder.

cd terraform/github-oidc
terraform init
terraform plan -var="github_repo=YOUR_ORG/YOUR_REPO"
terraform apply -var="github_repo=YOUR_ORG/YOUR_REPO"
terraform output role_arn

Copy the role ARN from the output. You'll need it in the next step.

Step 2: Set the Required Environment Variable

The Amazon Elastic Kubernetes Service spin-up path needs your GitHub Actions role ARN so Terraform can grant the CI/CD runner access to the cluster.

To find your AWS account ID, run:

aws sts get-caller-identity --query Account --output text

Then set the variable, replacing ACCOUNT with the number that command returns.

export GITHUB_ACTIONS_ROLE_ARN=arn:aws:iam::ACCOUNT:role/your-github-oidc-role

Step 3: Run the Spin-Up Script for Amazon Elastic Kubernetes Service

bash spinup.sh --cluster eks

When the script finishes, it prints the application URL. Open that URL in a browser and confirm that you see the same secrets table you saw locally, with all keys showing Match ✓.

Step 4: Test Rotation on the Deployed App

After you confirm normal operation, run the rotation test the same way you did locally.

bash rotation/test-rotation.sh

Then use /secrets/compare on the Amazon Elastic Kubernetes Service load balancer URL to validate behavior in the cloud environment.

⚠️ Cost warning: Amazon Elastic Kubernetes Service runs at approximately $0.16 per hour. When you're done with the lab, run bash teardown.sh from the repo root to destroy all AWS resources and stop charges.

How to Configure GitHub Actions Without Stored AWS Credentials

The typical CI/CD setup stores AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in GitHub repository secrets. Those keys never rotate. Anyone with repo access can read them. When someone leaves the team, you have to revoke keys and update every workflow.

OpenID Connect eliminates that problem entirely.

How OpenID Connect Works for GitHub Actions

GitHub can issue a short-lived token for each workflow run. That token identifies the run: the repository, branch, and workflow name. You create an IAM role in AWS whose trust policy says: only accept requests that come from this specific GitHub repository and branch. The GitHub Actions runner exchanges that token for temporary AWS credentials via AssumeRoleWithWebIdentity. No long-lived keys are ever stored anywhere.

Step 1: Create the IAM Role With Terraform

The terraform/github-oidc folder creates the OpenID Connect provider and the IAM role for you. You already ran this in the Amazon Elastic Kubernetes Service setup above. The role ARN is the only value you need to store.

Step 2: Add the Role ARN to GitHub Repository Secrets

In your GitHub repository:

1. Go to Settings → Secrets and variables → Actions

2. Click New repository secret

3. Name it AWS_ROLE_ARN

4. Paste the role ARN from the Terraform output

That is the only secret you store. The role ARN isn't sensitive. It's an identifier, not a credential.

Step 3: Configure Terraform State

For CI/CD to work consistently across runs, Terraform needs a shared state backend. The lab stores Terraform state in an Amazon S3 bucket and uses an Amazon DynamoDB table for state locking. The Amazon Elastic Kubernetes Service deployment guide in the repo covers the backend setup in full.

Step 4: Push to Main and Let Workflows Run

After your first spin-up, every push to the main branch drives the CI/CD pipeline. The repo includes separate workflow files for Terraform infrastructure changes and application deployment changes. Once your application is reachable, use /secrets/compare to validate rotation behavior on the live environment.

How to Troubleshoot the Most Common Failures

Here's a shortlist of the most common symptoms and their fixes.

How to Inspect the Operator and the ExternalSecret

When something isn't syncing, start with these two commands.

# Check the ExternalSecret status
kubectl describe externalsecret app-db-secret -n default

# Check the ESO operator logs
kubectl logs -n external-secrets -l app.kubernetes.io/name=external-secrets

The status conditions on the ExternalSecret resource will usually tell you exactly what failed.

How to Validate Rotation From the App Side

When you are debugging rotation, don't rely only on Kubernetes resource state. Use the /secrets/compare endpoint to see what the running application actually observes. The endpoint tells you whether env and volume match and whether rotation has been detected. That is the ground truth for your application's behavior.

Conclusion

You now have a complete secrets pipeline from AWS Secrets Manager into Kubernetes pods using Terraform and the External Secrets Operator. You ran the local lab, inspected the ExternalSecret and Deployment manifests, and validated that the application sees the right credentials.

You also tested secret rotation and observed the key behavior firsthand: mounted secret files update within the kubelet sync period, while environment variables stay stale until the pod restarts. That single observation explains a large class of production incidents.

Finally, you saw how the same design extends to Amazon Elastic Kubernetes Service with OpenID Connect-based CI/CD, and you have a troubleshooting checklist for the failures most teams hit.

The lab repository is at github.com/Osomudeya/k8s-secret-lab. If you ran the local lab, the natural next step is phases 4 and 5 from the repo's staged learning path: try the CSI driver path on Microk8s, then follow the EKS setup to see the same pipeline with a real CI/CD workflow and no credentials stored in GitHub. Both are documented in the repo and take less than 30 minutes each.

If this helped you, star the repo and share it with someone who is learning Kubernetes.

I send weekly breakdowns of real production incidents and how engineers actually fix them, not tutorials but real failures
→ Join the newsletter

Gaps in APIs can occur before requests reach the APIs, within the code housing the APIs, and even along the path of the APIs’ communication with downstream services, dependencies, or other microservices.

Attackers leverage flaws in APIs to gain access to confidential data, harvest or manipulate data, or even make your service unavailable through distributed denial of service attacks.

In this article, you’ll learn to deploy your APIs in Lambda and apply some security measures pre-function, within the function, and post-function.

Table of Contents

• What is an API?

• Requirements/Prerequisites

• Project Goal

• Project Overall Architecture

  • AWS Set Up

  • Clone Project

  • Set Up Simple Notification Service

  • Set Up Secrets Manager

  • Set Up Internal Lambda

  • Set Up External Lambda

  • Configure Web Application Firewall

  • Configure Cognito User Pools

  • Configure API Gateway

  • Test Setup End-to-End

  • Clean Up

• Improvements

• Conclusion

What is an API?

The focus of this article is the security of Application Programming Interfaces (APIs). An API is an interface that connects two programms or applications, allowing them to exchange data and communicate.

An API can be internal to an organization or it can belong to a third-party that allows other users to consume their data through the API.

Requirements/Prerequisites

While this tutorial is beginner-friendly, you’ll need the following prerequisites to follow along seamlessly:

• A basic knowledge of the AWS Cloud.

• An AWS account with administrator access.

• AWS CLI. You can find the installation guide here. Follow the instructions for your operating system.

• Python. You can visit Python’s official documentation site for a guide on how to download and install Python for your specific operating system.

• Pipenv or any Python virtual environment creation tool. You can find the Pipenv installation guide here.

• A basic knowledge of Git.

• An API client, like Postman or Thunderclient.

Project Goal

By the end of this project, you should be able to deploy APIs in Lambda securely, leveraging AWS cloud-native security services.

Project Overall Architecture

Below is the architecture of the project workflow:

As shown in the architectural diagram, when a user sends a request (a JSON object consisting of the user’s name) to an API hosted in Lambda, the user first gets authenticated by an authentication service called Amazon Cognito.

The request passes through a Web Application Firewall, then an API Gateway. API Gateway will perform a check to see if the user is authorized to access the API using the token that the user sends with the request after authentication. API Gateway then allows the traffic to pass through to the API if the user is authorized.

The user’s request will first get to an External Lambda function, which will then save the user’s name as a message to a Simple Notification Service (SNS) topic. This will then invoke an Internal Lambda to run and log the output in Amazon CloudWatch logs. The SNS topic will be accessed by External Lambda using the SNS’s unique identifier stored in Amazon Secrets Manager.

AWS Set Up

You’ll need to set up an AWS environment to get started. This requires creating an account if you don’t already have one.

Following account creation, a root user is automatically created, with all privileges attached to the user. Security best practice is to create another user with administrator privileges and use this user for subsequent tasks.

Then, create an access key for this user, which usually consists of two parts (Access Key ID and Secret Access Key) by navigating to the following:

IAM —→ Users —→ Create Access key

Follow the prompts and choose the Command Line Interface option. Check the Confirmation box, and go on to create the key. Download the CSV file provided, or manually copy the Access Key ID and Secret Access Key. Save them securely.

Open up your terminal and run the following commands using the AWS CLI:

aws configure

The above command will give some prompts to provide the components of the Access Key created earlier and your default region (the AWS region hosting the service you intend to interact with).

Clone Project

In the next step, you’ll clone the GitHub repository containing the assets and resources used in the project implementation.

Visit the project URL and clone the repository locally.

git clone <repository_clone_url>

Set Up Simple Notification Service

Amazon Simple Notification Service (SNS) connects system components, enabling asynchronous communication and messaging among them.

Find SNS on the console, click on it, and create a topic that your APIs will send messages to. After successfully creating a topic, navigate to the topic, and in the topic details, you’ll find the topic’s ARN. An ARN is an Amazon Resource Name, and it’s a unique string attached to a resource you’ve created on AWS to help identify the resource. Copy the ARN of the topic.

Set Up Secrets Manager

Amazon Secrets Manager is used to store, manage, and retrieve sensitive information such as keys, credentials, tokens, and so on. You’ll store the Topic ARN created earlier. With this approach, you’ll demonstrate how your API can securely access the data and information it needs for its performance.

Go to Secrets Manager on the AWS console and create a secret. Provide the secret’s details, and add a new secret named TOPIC_ARN as the key and the actual SNS Topic ARN as the value.

Next, you’ll create some Lambda functions to serve your APIs and consume the output of the APIs. There’re three Lambda functions to set up. Two of the functions will host APIs, each of which can only be accessed by specific users. These will be referred to as ExternalLambda. The third Lambda will consume the output of the External Lambda functions through SNS.

Set Up Internal Lambda

AWS Lambda is a serverless service on AWS that users can leverage to run application functions or code when needed. You’re billed for your Lambda function based on the number of invocations of the function, the duration each invocation lasted, and the amount of memory allocated to the function. Lambda can be provisioned to use any runtime, such as Python or NodeJS. In this demonstration, you’ll focus on the NodeJS runtime.

Now that you know what Lambda is and does, you can create one. Let’s call the first Lambda function InternalLambda. On the AWS console, search for Lambda, and on the Lambda dashboard, click Create a function and provide the details. We’ll be using Node.js – JavaScript at the backend as the runtime of choice.

For the Permissions details, let Lambda create a default IAM Role. This default role is named according to your function, and the permissions attached to the role allow your Lambda function to send logs to CloudWatch, another AWS service used for monitoring and observability.

As you can see in the last image above, the Lambda function you’ve created needs a trigger and sometimes, a destination. For your InternalLambda, the trigger is the SNS topic we configured earlier. This Lambda will read the messages that’ve been published to it, and then you can access the message from your client or even CloudWatch logs.

To achieve this, click the Add trigger button and provide the details.

Next, you’ll provide the code you want to invoke through Lambda. Find the code in the GitHub repository that you cloned earlier. Paste the code in the Lambda function code space and click on Deploy to deploy the function.

secure-lambda/InternalLambda/index.js

export const handler = async (event) => {
    try {
        console.log('Request successfully received from SNS');                            

        let name = event['Records'][0]['Sns']['Message'];
        let response = {
            statusCode: 200,
            body: JSON.stringify(`Hello ${name}. Greetings from InternalLambda!`),
        };       
        console.log('Response: ', response);                                                
        return response;
    } catch (err) {                            
        let response = {
            statusCode: 500,
            body: JSON.stringify('An error occurred while processing your request.'),
        };

        console.error('Error processing event', err);
        return response;
    }   
};

The function defined in the index.js file above is simply taking the event object sent to it from SNS and extracting the Message attribute within it. We’re using console.log here to view outputs from the function and ensure it behaves as expected. Just don’t use this in a production-ready application.

Set Up External Lambda

You’ll be creating two external Lambda functions: 1 and 2. These two functions will receive user requests, process them, and publish messages to your SNS topic.

On the Lambda console, create another function and name it ExternalLambda1. Allow Lambda to create a default IAM Role, as previously.

Paste the code snippet below in the ExternalLambda1 code space:

secure-lambda/ExternalLambda1/insex.js

import {
  GetSecretValueCommand,
  SecretsManagerClient,
} from "@aws-sdk/client-secrets-manager";

import { SNSClient, 
    PublishCommand 
} from "@aws-sdk/client-sns";

const secretsManagerClient = new SecretsManagerClient();

const snsClient = new SNSClient({});

// Fetch topicArn from AWS Secrets Manager
async function getSecretValue(secretName) {
    try {
        const data = await secretsManagerClient.send(
                            new GetSecretValueCommand({
                            SecretId: secretName,
                            }),
                        );
        if (data.SecretString) {
            return JSON.parse(data.SecretString);
        }   else {
            let buff = Buffer.from(data.SecretBinary, 'base64');
            return JSON.parse(buff.toString("utf-8"));
        }
    } catch (err) {
        console.error('Error retrieving secret', err);                             // added for debugging
        throw err;
    }
}                                        

export const handler = async (event) => {

    let name = event['name'];
    console.log(`Request successfully received from ${name}`);    

    // Retrieve SNS Topic ARN from Secrets Manager
    let topicArn;
    let response;
    try {
        const secret = await getSecretValue('LambdaSNSTopicARN');
        topicArn = secret.TOPIC_ARN;
    } catch (err) {
        response = {
            statusCode: 500,
            body: JSON.stringify('An error occured, try again later.'),
        };
        console.error('Failed to load SNS Topic ARN from Secrets Manager', err);
        return response;        
    }

    // Publish to SNS topic
   try {
        const snsResponse = await snsClient.send(
        new PublishCommand({
            Message: name,
            TopicArn: topicArn,
        })
        );
        console.log("Message published successfully:", snsResponse.MessageId);
        response = {
            statusCode: 200,
            body: JSON.stringify(`Hello ${name}. Greetings from ExternalLambda1! Message forwarded to InternalLambda.`),
        };
        return response;
  } catch (err) {
        response = {
            statusCode: 500,
            body: JSON.stringify(`Sorry ${name}.An error occurred while processing your request.`),
        };
        console.error("Failed to publish message:", err);
        return response;
  }  
};

The code above leverages the AWS SDK to fetch the ARN of the SNS topic created earlier from Secrets Manager. It then publishes a message to the topic.

The SDK already comes installed in the Lambda function. Outside of Lambda, the SDK has to be explicitly installed. The function receives its event from the client via API Gateway, which we’ll configure later.

The SNS topic you created earlier will be the destination for this function. For Lambda to publish a topic to SNS, it needs the necessary permission attached to its IAM Role. AWS can automatically take care of that during your configuration, as shown below.

For the trigger, you’ll use another service known as API Gateway. More on that later.

Follow the same steps to provision another Lambda known as ExternalLambda2.

The outcome of the External Lambda setup is as shown below:

Paste the code below into ExternalLambda2. It performs the same function as ExternalLambda1, but their output differ. Each of the two Lambda functions will be receiving traffic to a specific user that’s authorized to access the function.

secure-lambda/ExternalLambda2/index.js

import {
  GetSecretValueCommand,
  SecretsManagerClient,
} from "@aws-sdk/client-secrets-manager";

import { SNSClient, 
    PublishCommand 
} from "@aws-sdk/client-sns";

const secretsManagerClient = new SecretsManagerClient();

const snsClient = new SNSClient({});

// Fetch topicArn from AWS Secrets Manager
async function getSecretValue(secretName) {
    try {
        const data = await secretsManagerClient.send(
                            new GetSecretValueCommand({
                            SecretId: secretName,
                            }),
                        );
        if (data.SecretString) {
            return JSON.parse(data.SecretString);
        }   else {
            let buff = Buffer.from(data.SecretBinary, 'base64');
            return JSON.parse(buff.toString("utf-8"));
        }
    } catch (err) {
        console.error('Error retrieving secret', err);  
        throw err;
    }
}                                        

export const handler = async (event) => {

    let name = event['name'];
    console.log(`Request successfully received from ${name}`);    

    // Retrieve SNS Topic ARN from Secrets Manager
    let topicArn;
    let response;
    try {
        const secret = await getSecretValue('LambdaSNSTopicARN');
        topicArn = secret.TOPIC_ARN;
    } catch (err) {
        response = {
            statusCode: 500,
            body: JSON.stringify('An error occured, try again later.'),
        };
        console.error('Failed to load SNS Topic ARN from Secrets Manager', err);
        return response;        
    }

    // Publish to SNS topic
   try {
        const snsResponse = await snsClient.send(
        new PublishCommand({
            Message: name,
            TopicArn: topicArn,
        })
        );
        console.log("Message published successfully:", snsResponse.MessageId);
        response = {
            statusCode: 200,
            body: JSON.stringify(`Hello ${name}. Greetings from ExternalLambda2! Message forwarded to InternalLambda.`),
        };
        return response;
  } catch (err) {
        response = {
            statusCode: 500,
            body: JSON.stringify(`Sorry ${name}.An error occurred while processing your request.`),
        };
        console.error("Failed to publish message:", err);
        return response;
  }              
};

Before moving on, you need to modify the External Lambda’s IAM Roles. Currently, IAM Roles only have permissions to write to CloudWatch and SNS (automatically added). External Lambda also needs permission to fetch the ARN of the SNS topic that was created earlier.

The point here is to show how to leverage a secrets manager, such as AWS Secrets Manager, to store sensitive information or data, and still access these securely. This approach is more secure than storing the ARN as an environment variable within Lambda.

Navigate to IAM, and click on Policies tab on the left. This brings you to a list of policies. Next, click on Create policy.

Search for secrets manager in the Policy editor.

Select the permissions Lambda needs to access Secrets Manager. In this case, that would be Read —> GetSecretValue.

Select Specific for Resources, and click on Add ARNs. On the next tab, add the details of the Secrets Manager Secret created earlier.

The Secret’s ARN will be populated here.

Next, give the policy a name and create it.

Next, navigate to Roles, and search for the IAM Roles assigned to the External Lambda functions. These are named according to the Lambda.

Click Add permissions to add a new permission to the IAM Role selected.

Configure Web Application Firewall

A firewall is a system placed in front of an application, workload, APIs, and so on to inspect traffic, filter it, and either allow or block the traffic based on some preconfigured rules.

For this project, you’ll use the AWS Web Application Firewall (WAF) service to inspect user requests before routing the traffic to your APIs running in Lambda.

Head over to the AWS console and search for WAF.

Click on the IP sets tab on the left. This will enable you to create a list of IP addresses that you want to allow (as in this case) or deny.

The IP addresses should include a CIDR block. For instance, if adding a single IP address, it should be X.X.X.X/32. The same applies to IP address ranges such as X.X.X.X/24.

Next, click on the Web ACLs tab, then Create web ACL.

Choose Regional resources as the Resource type, and enter your region. It’s best to keep all resources you’re creating in this project within the same region. Give your Web ACL a name, then click next.

Add rules to the Web ACL.

Choose a rule type. In this case, you’ll use IP set, and give the rule a name. Choose the IP set created earlier.

Select Source IP address, and Count as the Action. For this project, you’ll focus on counting the requests sent to your APIs. But as shown in the image below, you can perform other actions, such as allow, block, and so on.

Your final rule configuration will appear this way.

Scroll down, then click on Create web ACL.

Configure Cognito User Pools

Amazon Cognito is an identity management service used for creating and managing users. You can leverage it to authenticate and authorize users to applications, APIs, or other workloads.

You’ll create User Pools within Cognito and add a user to each pool. You’ll configure how these users can be authenticated and authorized to access the External Lambda functions already created.

Search for Cognito on AWS.

Click on Get started for free, then Create user pool.

Select Single-page application (SPA), give the User pool the name MyUserPool1, and select Email as an option for sign-in. This means the main attribute users will provide at signup and sign-in will be their email address. Leave everything else as the default. We’ll keep things as simple as possible.

After creating the User pool, you’ll find the page shown below. You can view the login and signup page for the pool you’ve just created by clicking on the View login page button.

You can add App clients to your User Pool. By default, a client named MyUserPool1 will be added to the pool. Navigate to your User pool, and click on App clients to see details of this client. Note the Client ID. You’ll also make some edits to the App client by clicking on the Edit button.

You’ll edit the Authentication flows field by ticking the Sign in with username and password… and Sign in with server-side administrative credentials… boxes. These changes will enable you to authenticate the user who will be added to this client programmatically, rather than through a UI. With this approach, we can fetch the token assigned to the user by Cognito and use this token to authorize access to Lambda.

Now, add a user to this pool. The user needs a valid email address. You’ll need the login page URL to create the user.

You need access to the email used to create the user. Fetch the code sent to the email address and submit to confirm the account.

Follow the same steps and create another User pool named MyUserPool2. Add a user with a different email to this pool.

Configure API Gateway

API Gateway is a service used to manage access and route traffic to API backend services such as APIs. It serves as a reverse proxy and provides an extra layer of security for backend services.

You’ll configure API Gateway to direct traffic to your Lambda functions.

Navigate to API Gateway and click on Create an API.

Select the REST API option —→ Build.

Select New API, provide a name, and choose Regional as the API endpoint type. IP address type can be IPv4 or Dualstack. We’ll select IPv4 here. Then create.

An important part of API Gateway configuration for this project is the Authorizer. API Gateway uses Authorizer to allow traffic from clients to backend services.

You’ll create two Authorizers. Each will be connected to one of the User pools you configured earlier. On the left-hand side of the API Gateway you configured, click on Authorizers —→ Create authorizer.

Provide the name AGAuthorizer1, and select Cognito as the Authorizer type. Add the User pool for MyUserPool1 created earlier. For the Token source, use Authorization. When you send a request from your API client, a token will be added to the request header for authorization. The token’s key will be named Authorization, while the value will be the token itself.

Create another Authorization for MyUserPool2 the same way.

Both Authorizers will appear this way.

Next, you’ll create resources and endpoints within the API Gateway that you’ve defined.

A resource in API Gateway is used to group certain endpoints within a specific path. You’ll define two resources within the API Gateway you’ve created. This will create two different paths, <BASE_URL>/<RESOURCE1> and <BASE_URL/RESOURCE2>.

On the API Gateway dashboard, navigate to your Gateway, click on Create resource, define your root path (‘/’ in your case), and provide the resource name (lambda1).

Create another resource named lambda2.

Now, click on /lambda1, then Create method to define an endpoint within this resource. You’ll use the POST method to send requests to the backend service via this endpoint.

For the backend service or Integration type, select Lambda function, and provide the ARN of ExternalLambda1.

For Authorization, select AWS IAM —→ Cognito user pool authorizers —→ AGAuthorizer1. Leave other configurations, then create the endpoint.

Repeat the same step to create a POST method for /lambda2 resource. The method should be attached to ExternalLambda2, and AGAuthorizer2.

The API Gateway you’ve created needs to be deployed to become accessible. Deployment is usually done to a Stage.

Click on Deploy API, select New stage and name the stage development. Then, deploy.

After deployment to a stage, an invoke URL will be provided. This will serve as the base URL for the endpoints you’ve defined.

The stage you’ve created needs some modifications for enhanced security. Firstly, you need to attach the WAF that you created earlier. Secondly, the default rate limit for the API deployed to this stage is 10000. Rate limit restricts excessive resource consumption and protects your API from abuse. For this project, you can reduce the limit to 50.

To test the API Gateway set up, click on the endpoint you want to test, then the Test button. This initial test doesn’t need any authorization, since the test is done directly within the Gateway.

Add JSON data as the Request body. The key will be name, and the value will be any string.

The response sent back from ExternalLambda1 shows a status code of 200, and a response body containing exactly the message expected from the Lambda function.

If you head over to CloudWatch Log groups, you should also find the Log groups that were automatically created for the Lambda functions. Click on the Log group for ExternalLambda1 and navigate to the latest Log stream. You should find the logs for the request you’ve just made from API Gateway.

Test Setup End-to-End

To test our setup properly, and from the internet, send the same request from your API client with no additional information in the request header. This should return a 401 error – Unauthorized. This is expected.

API Gateway expects an authorization token from each request it receives before routing traffic to the appropriate backend service. It validates this token through Cognito.

You’ll mimic a user login for each user added to Coginito User pools to get a token for the user. This token will then be sent alongside any request. To achieve this, you’ll use the two Python scripts I’ve provided below:

secure-lambda/auth-scripts/user1.py

import boto3

client = boto3.client("cognito-idp")

response = client.initiate_auth(
    AuthFlow="USER_PASSWORD_AUTH",  # or ADMIN_USER_PASSWORD_AUTH if using admin creds
    AuthParameters={
        "USERNAME": "",             # user1 email
        "PASSWORD": ""              # user1 password
    },
    ClientId=""                     # Cognito App Client ID
)

id_token = response["AuthenticationResult"]["IdToken"]
access_token = response["AuthenticationResult"]["AccessToken"]
refresh_token = response["AuthenticationResult"]["RefreshToken"]

print("ID Token:", id_token)

Using the Python boto3 library, you’ll initiate an authentication request to Cognito. Provide the email address and password of the user in MyUserPool1. Also, add the Client ID of the App client.

To run the script, create an isolated environment using Pipenv, uv, or a similar library. Install the dependency used in the project as defined in the Pipfile, and run the script with the Pipenv shell.

pipenv install
pipenv shell
Python secure-lambda/auth-scripts/user1.py

The Python command will return with a token assigned to the user. Next, you use this token to authorize a user to access ExternalLambda1.

Ensure that the URL for the POST request is in the format: <BASE_URL/lambda1>. You should receive a response from API Gateway indicating success.

Now try accessing ExternalLambda2 using User1 token. You should get an Unauthorized message. Note that user1 will always receive an unauthorized message when it tries accessing ExternalLambda1 without an Authorization token in the header, a wrong token, or when it tries accessing ExternalLambda2, which it is not authorized to access.

Repeat the process with User2 using the token generated for the user in MyUserPool2. First, test access to ExternalLambda2 without a token in the request header.

Then test access with the token.

Next, try accessing ExternalLambda1 using User2.

You can also view the outcome of some of the requests made by your client on CloudWatch Logs.

Also, since WAF has been configured previously to count requests (although, in a real scenario, you want to achieve much more with WAF, such as allow or block certain traffic), you can view activities captured by WAF by navigating to the service on AWS, then searching for the WAF you configured, and navigating to Traffic overview.

You can find other details, such as the client device types and where requests originated.

Clean Up

It’s important to clean up the resources created so far after the hands-on exercise. Due to the dependencies among the resources, trying to delete a resource that another resource depends on may lead to an error. So, you should delete them in this order:

• Secrets Manager

• Cognito – Users, App Client, then User Pool

• API Gateway – Endpoints/ Methods, Resources, API, Stage

• Web Application Firewall – IP Set, Web ACL

• All Lambda Functions

• Lambda IAM Roles and the policies attached to them

• CloudWatch Log Group for all the Lambda functions

• SNS Topic

Also, you can deactivate or delete the credentials created for your IAM Admin user if not in use.

Improvements

Consider the following areas to improve, apply best practices to, and enhance the security posture of your systems further.

1. Use of API keys

2. Third-party API consumption

3. API inventory management/ documentation

4. Resource provisioning using Infrastructure as Code

Conclusion

Security at every layer of an IT system is not negotiable. In this project, we’ve demonstrated how to leverage cloud-native solutions to secure APIs hosted in a serverless service, allowing only authorized users access to the APIs.

I’m Agnes Olorundare, and you can find out more about me on LinkedIn.