Web applications are built to provide various online services to end-users. Developing and hosting these services involves hard work and talent. And it all begins with an idea.

But imagine, after putting in all that hard work, users cringe about the performance of the system – “It's too slow...”, “I wish I could get the response in this lifetime...”, “The product is good, but not really worth waiting for...” and they go on.

On the other hand, if you decide to provide your users with the best performance but you've got a poorly architected system, then your infrastructure costs can soar.

In this article, will see how making the right trade-offs matter.

Think of a music concert. Everyone's there, waiting to enjoy their favorite acts live. There are so many audio parameters associated with every line of input and output that runs across the stage, and those need to be set at an optimum level.

Blasting everything to its full level would make people leave the concert. Of course, this is not the artist’s fault – but the sound engineer who's job it is to make the artist sound good.

After all, this is a production system – similar to IT production environments. In IT, managing the system's performance essentially means managing the tradeoffs well. Of course, there are clear choices, but at times, making those straightforward choices is not so obvious.

Basic Web Application Infra Design

What is Cloud-Native?

Deploying business applications and services on managed data centers, also known as the cloud, is a long-running trend in the IT industry. This is mainly because the cloud offers numerous benefits and expertise while offering data-centers as a service.

Security, regulatory, and operational challenges still exist, so organizations are still lagging as far as moving 100% of their workloads to the cloud. On the other hand, startups go pro-cloud deployment as it is much easier to get your infrastructure managed by the cloud platforms from day one.

But what do we mean by cloud-native? You may think that merely shifting your workloads to the cloud will help you reap maximum benefits. That is partly true, since there are stages of cloud adoption. Cloud providers offer many rich infrastructure services, and when leveraged properly, they can drastically reduce your IT infrastructure costs.

The term "cloud-native" indicates the degree of cloud adoption in an organization. You might have come across cloud migration projects executed by organizations where they shift their workloads from on-prem hardware servers to VMs in the cloud. This is useful, as they benefit from getting rid of the effort required to maintain data centers themselves – but the situation could still be better.

Mere lifting and shifting of workloads is not very intelligent and is not cloud-native. Cloud platforms offer many more services like container registries, cluster management solutions, DevOps services, Serverless/Function-as-a-Service, and so on. All together, these give better results – in terms of everything, like cost, performance, maintenance, flexibility, reliability, security, and so on.

The notion of going cloud-native means adopting as many services provided by cloud providers as you're able, and aligning or refactoring your workloads to be deployed on the cloud in the most beneficial manner.

With that in mind, let's see how cloud-native adoption can help you improve the performance of your systems.

How to Improve Your System's Performance

When it comes to architecting any IT system, performance is one of the key aspects. We can classify the performance topics discussed below into three broad categories – compute, storage and memory, and network. Such categorization helps us look at the system from various lenses, and isolate the issues.

1. Workload Partitioning

Category: Compute

You might be aware that monolithic architectures represent the single point of failure. The probability of the entire system going down is high, even due to trivial issues.

The emergence of microservice architecture has helped solve this problem, but it also depends on how those microservices are designed.

While refactoring a monolith into microservices, you should follow the single responsibility principle. Build one microservice for one purpose only, and deploy multiple instances of critical microservices in auto-scaling mode to avoid the drop in performance.

This kind of workload partitioning helps tackle issues in an isolated way, reducing the risk of failures.

One of the first services offered by cloud providers was perhaps the ability to spin up virtual machines as needed. We can choose the flavor of the OS, size, networking, and various other aspects. Leverage this flexibility to deploy partitioned workloads.

Going a step ahead, you can isolate the runtime requirements for the services using containerization. This could be OS resources, CPU, network bandwidth, memory, and so on, where the quota can be predefined. This allows shared but dedicated resource allocation, thus breaking the one application, one server rule.

When workloads are containerized, it provides a firm base system that ensures its performance on any system that supports running container workloads – independent of hardware or OS.

Cloud providers also provide container orchestration services (for example Kubernetes), making it easier to deploy, debug and release newer versions of applications without downtime. Using these services and a wise deployment strategy can help roll out new features without any lag or glitch in the user experience.

I want to highlight that you should break your applications into microservices and host each service on isolated infrastructure. This avoids a lot of inter-process interference and resource contentions, thus optimizing the resource consumption for performance.

2. Compute Optimization

Category: Compute

Every application has different needs. Standard VMs offer everything that is required in a general-purpose server – CPU, Memory, and Networking capabilities.

However, not every application running on these VMs follows a standard as far as resource consumption is concerned. Applications or microservice components are purpose-driven, in a way that they have different computing needs.

A frontend server may rely more on networking capabilities as compared to its compute or memory requirements. A microservice dealing with large data transformation activities may need a better memory management solution for accessing the memory and performing transactions.

Major cloud providers offer features to optimize the virtual resources – especially VMs and databases – to align them with the application’s needs. In general, these optimizations can be categorized into three groups:

1. Compute-optimized
2. Memory-optimized
3. Storage-optimized

Depending on the criticality of the applications, there are options that help you optimize costs. You just need to choose the appropriate pricing plan and then trade off the availability of critically low applications with cost benefits.

Optimized virtual servers are selected depending on their purposes:

1. Batch processing, heavy data transform workloads, and ML algorithms usually need high-performance processors given their compute-intensive tasks. Choose to provision memory optimized instances for this purpose. Compute optimized instances are provisioned with enhanced block and file storage, along with widened network bandwidth. This helps churn out the best performance to process data oriented workloads.
2. If your app needs to deliver high network performance to transfer high volumes of data and serve a high volume of requests, then it is important to choose the right networking hardware associated with a provisioned VM. In this case, you can choose to optimize the network performance by opting for a high performance network interface card.
3. SSD-based volumes are used to assist CPU performance if the processing needs multiple transactions on the memory. A storage optimized VM helps with faster IO operations, and it's tuned for high throughput. Compared to using a general purpose VM, storage optimized VMs deliver greater performance.

It is important to analyze the system requirements for a given workload. A balanced configuration offered by general purpose VMs may just result in higher costs without significant improvement in the performance. You can make a wise choice by taking a look at types of VMs which can be provisioned.

3. Scaling

Category: Compute

There are 2 types of scaling:

1. Scaling up – where the virtual compute resources scale by their size.
2. Scaling out – where they scale in numbers.

In the cloud-native world, both options are available. Given the nature of templated size selection – the scaling up of resources is not always the best case since applications may experience bottlenecks in one of the aspects. Increasing the size of everything in a VM to address a single bottleneck creates under-utilized resources. This simply adds to the costs.

Besides, scaling up also means we are scaling a single point of failure. The better option would be to scale out in numbers. That way, even if a node fails, there are still others to serve the users and avoid potential losses.

Cloud providers offer an auto-scaling feature which lets you be sure that as many instances as you need are always up and running even if some of them fail in the meanwhile. This is done automatically.

But that does not mean there is no reason to worry about node failures. Ideally, the nodes should not fail at all, and autoscaling only provides a fallback mechanism as an attempt to recover from the loss. This in itself is a virtue.

Going a step ahead, Kubernetes capabilities provided by cloud-native platforms add an additional level of customization of resource allocation. From compute’s perspective, this means we have better ways to manage bottlenecks.

4. Serverless

Category: Compute

Going beyond containers – if you don’t want to worry about the vulnerability scans in images used, cluster management, OS, and orchestration, and you only want to write the code and let it run, then serverless is a great option.

In the serverless services offered by cloud providers, you are only expected to write “functions'' that define the logic for your hosted application. All the infrastructure required to run these functions is abstracted by the cloud provider platforms.

Apart from the huge cost benefits offered by serverless (topic for another day), developing applications on the serverless framework is the closest thing to cloud-native. The more cloud-native you are, the more services you can leverage.

It should be noted that serverless does not just mean writing code. The cloud providers may only provide a place to execute the incoming request. The way that request is processed and routed to appropriate queues and APIs requires leveraging additional services.

However, refactoring existing applications requires huge efforts, which is where containerization is an easier option to begin with. Applications, especially web services to be developed from scratch, are often good candidates for serverless.

5. Database Partitioning

Category: Memory & Database

Partitioning a database offers a clear advantage on performance. Think of it as a high level classification of the data itself that is stored in large volumes.

When a query is run against these volumes, chances are that the entire database or the storage volume is scanned to fetch the requested data. Partitioning reduces the scope of this scan, thus improving the response time. A good partitioning strategy is defined based on the stored data itself.

For example, an archive of all the newspapers from the year 2000 can be partitioned based on year, and further partitioned based on month and so on. So if you know the name and date of the newspaper you would like to read, it would be easier to find it in the archives.

Cloud platforms provide various services as far as the partitioning of the data is concerned. As opposed to the traditional ways, all the partitioning needs are handled by the platform itself once configured. So consider partitioning the data.

6. Dynamic Data Caching

Category: Memory & Database

Web applications serve multiple requests in parallel. These requests may require reading the data from the database. If the data is read frequently, each request would need to connect to the database, read it and make it available for the business layer.

If the data being read is the same, then it would make sense to store it in a cache for faster access. This avoids unnecessary, expensive, and frequent round trips to the database layer.

In a multi-node environment, every node may cache its own copy of data. Although it improves performance, it also creates multiple copies of the same data on multiple nodes so it is still very inefficient.

One of the workarounds is to enforce client affinity which makes sure that a particular request is always served by the same node in the cluster. If the node is busy, it introduces additional latency and having multiple nodes in the cluster does not serve the purpose.

This is where shared cache databases come into the picture. Redis and memcached are the most used shared cache databases, which are used in backend architecture of many web applications.

Shared cache databases are not the replacement for databases – they help in storing data for fast retrieval of temporary data. They are usually hosted between the business logic nodes and database.

Shared cache databases help consolidate database transactions and keep the database state consistent. Once installed and configured, you can use them to set and get values. So instead of maintaining the node-specific copy of data, using shared cache database helps keep the data, once stored, available for all the nodes.

Cloud providers offer support, solutions, and services to host Redis and memcached to enhance performance of the systems.

7. Consider Eventual Consistency

Category: Memory & Database

In terms of databases, consistency defines if the immediate reads are up-to-date with the latest written information. Applications that perform many IO tasks to the databases have the tendency to use locking mechanisms to make sure multiple intended writes happen in a consistent way.

However, when such frequently changing data is being read, especially where multiple reads are allowed, the data can either be consistent with the latest write or not.

When multiple readers are enabled, in some cases the database is also replicated to have a real-time or near real-time copy of the primary database (the purpose of which is to serve all the read requests).

In such cases if you need consistency, then that adds to the performance lag as all the read operations have to be suspended until the write operations are replicated on all the copies of the database.

When you're developing business critical services, distributed relational databases that have ACID compliance/guarantees can result in a rather slower system.

Eventual consistency is a feature mostly offered by NoSQL and Document databases. Eventual consistency is a guarantee that the data that is being read and displayed to the user is up-to-date with latest write and will be updated in the database.

Cloud providers offer NoSQL and Document database services which also offer eventual consistency. Improving system performance by enabling eventual consistency and simultaneously protecting data integrity is a challenge that needs an overarching solution.

Identify the cases where you can take advantage of eventual consistency and develop routines and transactions to support end-to-end data integrity. This will help improve performance. Cloud native solutions provide this ability, and eventual consistency also helps with cost optimizations.

8. Leverage The Backbone Network

Category: Network

As far as networking is concerned, well established Cloud-Native platforms have data centers and server farms in various locations spread across the globe. It doesn’t matter where the traffic is coming from – it's likely that these cloud providers have a datacenter available in their vicinity.

These data centers are connected with a backbone network which is also developed by the cloud providers that are dedicated for their inter-regional communication. This is a great advantage. If the application users live on multiple continents, deploying applications on the cloud inherently accelerates the delivery of your services thanks to these backbone networks.

Since this is a global network supporting all the services, hosting cloud-native applications can be served closer to the users regardless of their continent or country.

Direct and dedicated connections and VPNs are also used to connect on-prem or private data centers securely to the cloud infrastructure in the form of access points. In a way, establishing these connections helps onboard the organizations network on this backbone network, helping them reach their employees across the globe.

9. API Gateway Caching

Category: Network

Like other services, cloud providers provide services for hosting and configuring API gateways for your applications. Today, APIs are the first choice of interface for any web application.

API Gateways not only help develop these API paths, they are also seamlessly integrated with other services related to compute, storage, databases, functions, containers, orchestrations, queues, and so on. The rich Web UI makes it easy to configure API Gateways and maintain versions of the same.

Since APIs deal with sending and receiving payload data, there are several pre-built features which assist in formatting, encoding, and security of payloads. Since these advanced features are pre-built in these API Gateway services, the efforts required to configure these for optimum delivery are low.

You can leverage these features for automatic compression, decompression, classify and route requests to target service, enable cache to improve latency of application response.

10. CDNs

Category: Network

Considering the fact that cloud providers offer their own backbone network that carries dedicated traffic between their data centers, it becomes very convenient for them to provide CDN services as well.

Major cloud platforms offer CDN services that give really good results as far as content delivery of static data is concerned. Web applications being served to users via Web Browser typically use HTML, CSS, JS files to build the interactive user interface. These files are fairly static and do not change very frequently.

While applications are being deployed on cloud platforms, it makes sense to take advantage of CDN services to improve the network performance of these applications.

Conclusion

When we talk about optimizing the performance of the cloud-native system, there are many more factors involved. It is important to think about the system using various lenses – compute, storage, databases, and networking – and identify the bottlenecks by performing various load and stress tests.

Monitoring is an important feature which many major cloud companies provide, and is well-integrated with other services. Leverage these monitoring capabilities to analyse the performance bottlenecks and identify effective action items for change.

I have covered challenges and approaches with deeper insights in my FREE eBook published on my website Let's Do Tech. If you are interested in following more of the architectural stuff, do keep an eye on me. Social: Instagram, Twitter, LinkedIn, FB.

IAM, or Identity and Access Management, is one of the most common terms you'll hear in cloud-native environments.

But what does it do? And if you're already familiar with IAM, how long did it take you to fully understand it?

I will explain the main concepts behind this massive family of software, with you, the busy engineer, in mind.

The fundamentals described here are vendor agnostic, though most of my experience is with AWS's implementation.

What Is IAM?

IAM is a complex system of entities (humans, applications, and so on) that request access to a system. It is also a hierarchical set of rules to grant or deny requested access.

Before we go any further, here are the main terms you'll encounter:

• Resource: Anything worth protecting. A storage service, virtual machine, etc.
• Policy: A set of rules that dictate who can and can't do something on a single resource or group of resources.
• Action: Anything someone can do inside the cloud environment. For example, creating a virtual machine.
• User: Well... A user :)
• Group: A group of users with the same permissions applied.
• Principal: A user or an application requesting access.
• Role: A set of powers assigned to a principal, usually for a limited amount of time.

Why IAM Is Useful

IAM is mainly used for authentication, authorization, granular access, and governance.

Let's see what those all mean:

• Authentication: The act over verifying who you are.
• Authorization: The act of identifiying if someone can perform the action they are requesting. This is usually combined with authentication, but not always.
• Granular access: Permissions that control each action that can happen on a resource. For example, a user might have permission to see firewall rules, but doesn't have permission to change them. This is implemented with Role-Based Access Control.
• Governance: The actions you take to know what is happening in your environment, mostly for reasons of budget, compliance, and proper access scope.

If you're a company of 1-3 people, then setting up a full-blown IAM solution is probably overkill. Buf if your team is larger than that, or you're planning to scale up, then you should start considering it.

Common Problems You Don't Use IAM

I believe you can see the benefits of an IAM solution.

Now let's take a look at some common problems organizations face in the absence of it.

It's Hard to Audit and Administer Access

Have you heard of cases where an employee had more access than they should? And additionally, no one knew?

This can be prevented with a properly set-up IAM solution.

Setting Up Accounts for New Hires Is a Pain

With an IAM solution in place, this would just be a matter of a few clicks. Namely, set up the users and add them to the IAM groups their teams use. That's it.

But without an IAM solution? You would need to set all the permissions for each account manually.

You might have a reference user to copy from, but does each new account need all the permissions the reference user has? Do you have special handling for user accounts that are less than 6 months old? Does the reference user have superuser permissions that should not be accidentally assigned to a new hire's account?

Offboarding People Is Time Consuming

Here you'll have similar problems to the new hire case above. But when a collegue is leaving, you'll need to change the password to all the accounts they potentially used.

This can turn ugly very fast, not to mention the side effects this has on other team memebers.

And you would have to do this for every script, application, and other resource whenever there's an offboarding. What if you have a team change 2-3 times per month? You and your team would have a hard time being productive.

Simple Things Require Human Intervention

Without an IAM solution, tasks like resetting a password or re-enabling an account that was locked need to be done manually.

Top-tier IAM solutions have a way to resolve such issues fast without much hassle.

Best Practices

If you've decided to set up IAM, here are some best practices. This is far from a full list, and is based on my personal experience. But I've seen these practices on more than one team, so they should work for you as well.

Never Grant Full Access... EVER

In a real-world scenario, you wouldn't want every user to have unlimited access to an account. Ideally, no one should full access to anything (apart from the account owner).

For example, if an employee's responsibility is to monitor logs, they should have read access only to that tool. They should not be able to restart a service, or view billing information.

Prefer Groups to Multiple Users

It's better to use groups instead of multiple users when you have a choice. Groups make administration exponentially easier.

For example, if a new person joins your organization as a developer, they can be added to an IAM group for developers. That new person will then inherit all the powers of that IAM group.

The alternative, creating a user for each group (reader_susan, admin_susan) is considered obsolete.

Prefer Roles on Existing Users to Creating a New User

When given the option, prefer assigning a role to an existing user rather than creating a new user.

For example, don't create an admin user and share the password between 10 people. Create an admin role and assign it to whoever needs it for a limited amount of time.

Audit Permissions Frequently

It is easy to make mistakes or perform malicious actions. At the very least, a company should audit permissions regularly, and ensure that only the proper people have the minimum level of access necessary for their roles.

You could also send an email to a certain team when a suspicious action happens. For example, assigning an admin role to a new hire.

Set Up Boundaries Beforehand

If an IAM solution allows for it, add boundaries to your ecosystem.

According to Amazon's documentation:

A permissions boundary is an advanced feature for using a managed policy to set the maximum permissions that an identity-based policy can grant to an IAM entity. An entity's permissions boundary allows it to perform only the actions that are allowed by both its identity-based policies and its permissions boundaries.

(I know, I know — I promised to be vendor-agnostic 🙂)

In layman's terms, you can define the "maximum" permissions that can be assigned to anyone.

For example, a user will at most be able to view the logs from the relevant tool and restart a service. If someone attempts to get a role to create a new virtual machine they will be disallowed.

Conclusion

Thank you for reading this far. I hoped you enjoyed this introduction to IAM.

If you have any questions, please reach out to me on Twitter.

• A little about AWS
• What are the benefits of serving from S3 and CloudFront?
• Before we start, you’ll need an AWS account
• Storing your website on S3
• Serving your website on S3
• Distributing your website on CloudFront
• Custom domain names
• Advanced AWS Usage
• Resources

A little about AWS

If you’re not familiar, AWS (Amazon Web Services) is a cloud service provider that gives developers opportunities to build pretty much anything they can imagine in the cloud.

Though their services extend beyond the likes of machine learning and artificial intelligence, we’re going to stick with the entry level services for the purpose of this guide that will allow us to easily host an HTML website.

Types of AWS services available

Building a site with S3 and CloudFront is a common recipe that small and high scale companies across the web use, but let’s break down what each service actually does.

Object storage with S3

S3 (Simple Storage Service) acts as your hosting for your static website. Think of it like a hard drive in the cloud which we’re not able to use it for processing purposes, but rather for simple file storage and access.

List of files from a static site in an AWS S3 bucket

When an app or website is compiled in static form, this is all we need to serve it to the people visiting our site. The HTML is sent in the initial request “as is” (unless there’s processing with your provider) and any additional work occurs after the page loads in the browser usually by JavaScript. This allows us to take this simple (and cheap) approach by serving these files from S3.

Content Delivery Network with CloudFront

CloudFront works as a CDN (Content Delivery Network) that sits in front of your website, caching the files, and serving them directly to the people visiting your site.

CDN Diagram

Where you host and serve your website from, typically called the origin, is the main source of your files and can serve the website itself. But putting a CDN in front of it provides the people accessing your content a shorter and faster way to make their request.

What are the benefits of serving from S3 and CloudFront?

Given the rise in the JAMstack era, many services are popping up that provide similar services for static sites that make it really easy to deploy. Some even come with a generous free tier like Netlify and Zeit!

But sometimes developers need a little bit more control over their services or they need to integrate into a larger cloud pipeline that’s already 99% percent in AWS, which is exactly where S3 shines. Also, chances are, during your first year you might still qualify for AWS’s free tier.

Fitting in to the AWS Well-Architected Framework

As a lead provider in cloud services, AWS has published many guides to help developers and teams strive for excellence in their solutions in terms of performance, cost, and security.

One particular guideline is their 5 pillars of what they describe as a “well-architected" infrastructure.

AWS Well-Architected Framework

By default, we check all of these boxes with our hosting solution by using S3 and CloudFront. Out of the box, the HTML and assets you serve will be fast, cheap, secure, and reliable.

The beauty of static and JAMstack sites

Building on top of the pillars, what you’re actually serving is a static HTML file and group of assets that won’t require any type of rendering resources on the initial request. Before this, a common problem was having to worry about a site crashing due to heavy load. But with S3 and CloudFront, your website is infinitely scalable.

On a similar note, when that server scales up as it's trying to serve millions of hits on your post that went viral, so will your costs. Serving a static site is cheap and can greatly reduce the cost associated with running a web server.

Before we start, you’ll need an AWS account

To work through this guide, you’ll need an AWS account. Luckily, it's free to create an account – you’ll only pay for the services used.

On top of that, AWS provides a generous free tier for some of its services. Some services provide only 12 months of a free tier (like S3) where others are always eligible for the free tier (like Lambda), so make sure to do your homework so you don’t rack up an unexpectedly high bill.

To create your account, head over to the AWS website and then continue on to get started: https://aws.amazon.com/.

Storing your website on S3

To get started, we’re going to begin with a simple HTML file that will serve as our website. This will allow us to focus more on the process of hosting rather than the intricacies of the website itself.

Creating our website file

Begin by creating a new folder called my-static-site. Inside that folder, let's create a new file called index.html and add the following to the file:

<!DOCTYPE html>
<html lang=“en”>
<head>
  <meta charset=“UTF-8”>
  <meta name=“viewport” content=“width=device-width, initial-scale=1.0”>
  <title>My Static Website</title>
</head>
<body>
  <h1>Hello World!</h1>
  <p>This is my static website. ?</p>
</body>
</html>

If you open this file from your computer in your favorite browser, you should now be seeing this.

Hello World! Opening a local webpage

Creating a new bucket

Head on over to your AWS account, log in, and navigate to your S3 console.

Once there, let’s create our bucket by clicking on the blue Create bucket button:

Creating a bucket in AWS S3

The first thing AWS wants us to do is enter a Bucket name. The bucket name must be globally unique, meaning, the name you use can be the only one in the world, so let’s try something like [yourname]-static-website, where I’ll use colbyfayock-static-website.

Naming a bucket in AWS S3

Next, let’s set the Region. This is the geographic location where AWS will host the bucket and your website. You’re probably fine with the default, but if you’d like, you can select the location closest to you if it’s permitted. Since I’m in Virginia, I’m going to stick with my default of US East (N. Virginia).

Setting the region of a bucket in AWS S3

Finally, hit the Create button on the bottom left of the page.

Note: even if you use the [yourname]-static-website pattern, there’s a chance the name will be taken. If it’s taken, AWS will show an error stating “Bucket name already exists,” at which point you’ll want to try a new name of your choosing.

Alternatively, you can hit Next for advanced usage, but for this guide, we’re okay with all of the defaults S3 provides.

If successful, you should now see your bucket in the list on the S3 console dashboard.

New bucket in AWS S3

Uploading your website to the bucket

Let’s navigate to our new bucket by clicking the row of our bucket. You’ll be greeted with a message stating “This bucket is empty. Upload new objects to get started,” so that’s what we’ll do.

Click the Upload button to get started.

Uploading files to AWS S3

You’ll then see a popup that will ask you to upload a file. Click on the Add files button and select your index.html file we created earlier.

Once selected, click the Upload button on the bottom left.

Selecting files to upload in AWS S3

And now your file is uploaded to S3!

Serving your website on S3

If you try to navigate to your index.html file and open it, you’ll notice a big ugly "Access Denied" message.

Access Denied to bucket file

This is because your file doesn’t currently have the permissions and settings necessary to serve the file to the public, so let’s fix that.

Setting up your bucket as a website

Navigate to the Properties tab inside of your bucket, then click Static website hosting.

Setting up an AWS S3 bucket for statice website hosting

Once there, we want to do a few things:

• Note down the Endpoint at the top of the block. We’ll use this to access our site later (you can always find this here again)
• Select the “Use this bucket to host a website” option
• Enter index.html in the Index document field
• Finally hit Save

Configuring an AWS S3 bucket for static website hosting

Setting up your bucket policy and permissions

Next, navigate to the Permissions tab. Here we’ll want to do 2 things: unblock all public access and add a Bucket Policy.

First, on the main page, let’s click Edit to unblock all access.

Configuring an AWS S3 bucket permissions

Then, uncheck the “Block all public access” checkbox and hit Save.

Allowing public access to an AWS S3 bucket

AWS will ask you to confirm these settings, as this may not always be what you want to do with your bucket. But for the purposes of hosting a website, we want the whole world to see, so type in the word “confirm” and hit the Confirm button.

After confirming, click the Bucket policy button and you’ll be taken to a text editor.

In this text box, we’ll want to paste the following snippet. Within this snippet, make sure to replace [your-bucket-name] with the name of your bucket, otherwise you will not be able to save this file.

{
  "Version":"2012-10-17",
  "Statement":[{
    "Sid":"PublicReadGetObject",
        "Effect":"Allow",
      "Principal": "*",
      "Action":["s3:GetObject"],
      "Resource":["arn:aws:s3:::[your-bucket-name]/*”
      ]
    }
  ]
}

This policy states that it’s allowing the public to perform a GetObject request on the S3 resource, which is your S3 bucket.

Setting up a public policy for an AWS S3 bucket

After you add the policy, click the Save button. Your should now see a message stating "This bucket has public access.”

Previewing your new bucket website

If you noted down the Endpoint from your Properties page, you can now visit that address to see your website. The endpoint should look like this:

http://[your-bucket-name].s3-website-[region-id].amazonaws.com

If you didn’t, jump back up a few steps to remind yourself how to find it or look under the Properties tab.

Hello World! Opening an AWS S3 website

Congrats, you're halfway there! ?

Distributing your website on CloudFront

Now that we have our static website being served from a bucket on S3, let’s take it up another level and serve it across the world using CloudFront.

Creating a CloudFront distribution

Navigate to your CloudFront dashboard and click the Create Distribution button.

Creating a new distribution in AWS CloudFront

Next, select Get Started under the Web delivery method.

Getting started with an AWS CloudFront distribution with Web delivery

Here, we’ll enter a few custom parameters to get our distribution set up.

Click into the Origin Domain Name field. Once selected, a dropdown list should appear where you can select the S3 bucket you just created. Go ahead and select your S3 bucket.

Setting the origin domain name in AWS CloudFront to your bucket

While you can customize most of the settings to your liking, for our purposes, we’re going to leave all as their default values except for one.

Scroll down to the Default Root Object field and type index.html.

Setting the Default Root Object for a distribution in AWS CloudFront

After, scroll down to the bottom and click Create Distribution in the bottom right.

Creating an AWS CloudFront distribution

Previewing your new CloudFront distribution

After hitting the Create button, it will take some time for your distribution to be created and set up. You’ll notice on the CloudFront Distributions list page that the Status of your new distribution is In Progress.

AWS CloudFront distribution deployment is In Progress

Once this completes, it will say Deployed. Then you can find your Domain Name in the same row.

AWS CloudFront distribution is Deployed

Using the value in the Domain Name column, open your distribution in your browser and success! You now are viewing your S3 bucket through CloudFront’s distribution network.

Hello World! Opening an AWS CloudFront website

Custom domain names

While most of us will probably want to use a custom domain name with our website, we’re not going to dive too deep into that this guide, as there are many ways to set that up depending on where you purchase your domain name.

However, here are a few things to consider.

HTTPS / SSL Certificate

If you’re creating your CloudFront distribution to use with a custom domain name, you'll most likely want to configure your distribution with an SSL certificate using AWS’s Certificate Manager. Alternatively you can provide your own certificate with tools like Let's Encrypt, but by using ACM, AWS makes it easy to pull in the records for use with your distribution.

Once in ACM, you’ll want to configure the certificate, map what domains and subdomains should match (typically *.domain.com), and then create your certificate to use with your distribution.

To get started, you can check out the AWS guide for requesting a public certificate.

CNAMEs and Aliases

A common approach to setting up a custom domain is to use a CNAME. CloudFront makes this pretty painless, as you’ll add it as a configuration option when you’re configuring your distribution.

To get started with setting up a CNAME in CloudFront, see the AWS guide.

If you’re using Route53 to manage your DNS, you can then set up an A record (alias) to point to your distribution. You can learn more using this guide.

Advanced AWS Usage

For this guide, we walked you through setting up a new static website and app using the AWS console. But whether you want to learn more, improve your deploy efficiency, or want to automate this process, you’ll want to take a it a step further with the AWS CLI or CloudFormation.

While we won’t walk you through how to use these tools here, we’ll get you started with a little bit of an idea of what you’re up against.

AWS CLI

The AWS CLI allows someone to perform AWS operations from the command line. This can be incredibly powerful when you want to script out your resource creation or if you simply prefer to do all of your work from the terminal.

Once set up locally, you’ll be able to perform actions like creating a bucket using the following command:

aws s3api create-bucket —-bucket [your-bucket-name] —-region [bucket-region]

To get started, check out the AWS CLI Github page or the AWS CLI User Guide .

AWS CloudFormation

AWS preaches “infrastructure as code.” It’s the idea that you can spin up your infrastructure using something that’s written in a file, where in this particular case, it would be a CloudFormation template. This allows you to have a repeatable process that will be the same each time you perform the deploy.

CloudFormation allows you to set up a configuration file that will deploy the services and resources of your choosing by pointing to that file with the CLI or by uploading it in the console.

Here’s an example from AWS of what that looks like for a static S3 bucket that could serve as a website.

AWS CloudFront template example

To get started, check out AWS’s CloudFormation example templates or their Get Started guide.

Resources

If you’re interested in getting deeper into the AWS ecosystem, here are a few resources to get started:

• AWS Certified Cloud Practitioner Training 2019 - A Free 4-hour Video Course (freeCodeCamp.org)
• Introducing The #AWSCertified Challenge: A Path to Your First AWS Certifications (freeCodeCamp.org)
• 10-Minute Tutorials (AWS)
• A Cloud Guru (Paid courses)
• AWS Case Studies (AWS)

My company’s Internet of Things (IoT) side project began when we couldn’t reset the door lock that we inherited from a previous tenant. It was one of those minor details we learned about after moving in to our new last-minute office.

Normally, people just pay for a new one. But our team was too cheap to replace the lock and no one ever wanted to get the door bell. Plus, we’re engineers and we wanted to fiddle with some hardware.

Our goal was to open the door with a phone or wearable technology. We had several options for how to approach the problem. In theory, we could use an app, an integration into another platform, or anything that could send a signal to trigger the door lock.

Chima Open Door on Pebble and iOS

So far in our door lock experiment, we’ve developed solutions for a Slack integration, native iOS and Android apps, the Apple Watch, and Pebble. I’ll focus on the architecture of the mobile apps. I admit the final product is a bit over-engineered, but we just love it!

iOS and Android architecture

Our IoT door lock project’s architecture

What exactly happens when you press the button in our iOS / Android app? An HTTP request is sent to the cloud server, which then triggers a message to the door lock daemon via the client server, which then tells a relay board to open the door lock.

Traditionally, the door lock is opened with a button beside the door. But with modern technology, the possibilities extend beyond a direct, physical button. In addition to the physical button that triggers the Doorlock Daemon in the diagram, we added two other triggers: a cloud-based trigger, and a Bluetooth Low Energy (BLE) trigger, thanks to our choice of hardware.

This article focuses on the cloud-based trigger, which is what our door lock app depends on.

Starting from pressing the button to a record saved on Skygear server.

When a user presses the open door button on the mobile app, the app accesses the cloud server.

Two things happen in the cloud server. The first is that a record is saved to our choice of server, Skygear Cloud Database, which allows you to synchronize your data to the cloud. The server will log when the door access is being requested.

Once a record is saved, it would trigger an after_save function provided by Skygear Cloud Functions, which runs our code in the cloud without bothering server deployment.

The after_save function is triggered after a record is saved. def after_open_door_save(record, original_record, db): is triggered asynchronously when a record of type 'OpenDoor' is saved. The function publishes a message to the channel 'xxx-channel'.

The Node Client and Clojure Server on Raspberry Pi

The next step is to create a listener for the request. This is where the Node client and a Clojure server on Raspberry Pi come in. The Node client listens to the message in the specified channel on the Skygear server. The Clojure server is the only one with the right to access the Raspberry Pi 3 circuit. Then the Node client issues a request to the Clojure server once it hears any message.

Here is the script for the Node client, which includes code related to our specific configuration for Skygear. The endpoint and the API Key are for accessing the main server on Skygear. skygear.on('xxx-channel', onReceiveOpenDoor) means to subscribe the function callback (onReceiveOpenDoor) on receiving a message on the 'xxx-channel' channel.

The Clojure server directly controls General Purpose Input/Output (GPIO) on a Raspberry Pi. GPIO are the pins on the Raspberry Pi 3. The GPIO connects to the external circuit that is connected with the door magnet.

Here is the Clojure code showing how the Raspberry Pi opens the door. Once the Clojure server receives the request from Node client, it will open the door and set it open for 3 seconds. However, if there is a new request coming in during that 3 seconds, the door will reset the timer to another 3 seconds. When the count down time is up, the door will lock again.

A random side note: Skygear is using AWS in America, while the door and the Raspberry Pi is in Hong Kong. Effectively, our ‘芝麻開門’ (Chima Open Door) request travels around the world before it reaches the door.

Why Raspberry Pi?

Now, you may be wondering why we specifically chose Raspberry Pi. We considered using Arduino boards because we had them in the office. The reason we couldn’t use our specific Arduino model was because we wanted to synchronize data via Skygear JS SDK and this specific Arduino can’t set up the Node server.

What’s more, Raspberry Pi is Bluetooth Low Energy ready (which means we could access the door lock using a third method, Bluetooth).

Linus-based Raspberry Pi is compatible with Oursky’s open-source serverless platform, Skygear

Additional integrations

Considering the app is internal-use only, we started a Slack customized command /chima-open-door to open the door since every Ourskyer has access to Slack.

Later some other Oursky colleagues got involved in this project and helped write the WatchOS app and Android app published on the internal platform. Apart from pressing the button inside the app, we also provide alternatives such as iOS 3D touch, Today extension, Android widget and even a Pebble integration because some of our developers use it.

That’s it! Before you dive in, there are two other main factors to consider: the reverse electricity flow (in this case for the Raspberry Pi) and the security of each of your integrations. For example, we also integrated Bluetooth app access with Bluetooth Low Energy (BLE), which has a self-implemented 2FA-like authentication. Other integrations you can include are notifications when the door is open (bell, LED).

If you want to learn about any of the above, feel free to get in touch!

Link to Repo / files
Doorlock: https://github.com/oursky/doorlock

I would like to credit my colleagues David Ng, Boris (akiroz), Brian (b壹貳參肆零零), and May Yeung for working on the Android application, the circuit implementation & Clojure, Pebble application, and this blog piece respectively. Here’s to teamwork!

At Oursky we’re all about helping brands and entrepreneurs make their ideas happen, as well as fellow developers — our latest project Skygear (https://skygear.io), an open source (https://github.com/skygeario) serverless platform for mobile, web & IoT apps — helps you build better apps faster. ?