Build a real-world serverless application on AWS with Serverless framework and ready-to-use functions from StackStorm Exchange open-source catalog.

Episode One | Episode Two | Episode Three | Episode Four

It took us three exciting episodes to create a non-trivial serverless app. The back-end now works, but two things are still missing: a Web UI, and a final discussion to sum up the lessons. Let’s continue… after a quick recap of the previous episodes:

• In Episode One, I described the application we are building, walked you through setting up the development environment and creating a Serverless project, and showed how to build your first Lambda function from a StackStorm Exchange action with Serverless Framework.
• In Episode Two, we added more actions: one native Lambda to record user info to DynamoDB, and another one from StackStorm Exchange to make a call to ActiveCampaign CRM system. You learned more of serverless.ymlsyntax and practiced the development workflow with Lambda functions.
• In Episode Three, we wired the actions with StepFunctions, learned tricks on passing the data between the steps, struggled debugging StepFunction executions, and finally got the backend working end-to-end.

You can get the final code for this episode from GitHub.

Add Web UI

We need a web form that takes user names and emails, and posts to our StepFunction API Gateway endpoint. I just copied the one I used while exploring Serverless with Python, StepFunctions, and Web Front-end Serverless. Which, I confess, I grabbed from the old Serverless Slack Invite project.

Those of you who are Web developers can surely create something more elegant with ReactJS — PRs welcome! Whether you build your own or grab mine, do this: create a directory web and place the static content there.

I use [http-server](https://www.npmjs.com/package/http-server) for a quick look that at my static form:

cd webhttp-serverStarting up http-server, serving ./Available on:  http://127.0.0.1:8080

Open a browser, check the form at http://127.0.0.1:8080 , see that it comes out right, and let’s move on to the next step.

How is a serverless web front end deployed to AWS? Typically, you put the static content on S3, configure it to serve the web, make your web app call your backend endpoint, and remember to enable CORS on API Gateway endpoint.

Alternatively, you add a path to your API Gateway resource to serve your static web content from the S3 bucket.

Pros: 1) no mess with CORS, 2) no adjusting your web app to point to the correct backend endpoint. Extra bonus: easily done with serverless-apig-s3 plugin.

Cons: 1) paying API Gateway charges for web requests 2) the plugin is a bit shaky: fine for examples and small apps, but I wouldn’t use it for anything resembling production.

PRO TIP: For serious web front-ends with high loads: deploy them to CloudFront. Or use something like Netlify — check out the recent post “How to build a static Serverless site with Netlify” from serverless.com.

I’ll use the API Gateway approach with serverless-apig-s3 plugin here: we don’t expect massive load. And it is simple. Watch:

Install the plugin (been there, done that):

npm install --save-dev serverless-apig-s3

Change serverless.yml. Add the plugin (been there, done that). Add the plugin’s configurations:

...custom:  private: ${file(env.yml):private}  stage: ${opt:stage, self:provider.stage}  region: ${opt:region, self:provider.region}  apigs3:    dist: web    topFiles: true

...

plugins:  - serverless-plugin-stackstorm  - serverless-step-functions  - serverless-apig-s3

We are telling the plugin to pick our web directory and put it to S3. The topFiles flag tells API Gateway to expose our index.html and formsubmit.js at our endpoint, under /web/

Now deploy the service, and, separately, the client side, with two commands:

sls deploy

sls client deploy

Deploying the service will update the API Gateway with the paths to access our web content. Deploying the client puts the web content up to an S3 bucket. Now you can reason that if you add or remove web files and client deploy is not enough, a full sls deploy is required. But if you only change the web files, quick sls client deploy will put them up on AWS.

That’s it!
Go to https://YOUR-ENDPOINT.amazonaws.com/dev/web/index.html.The form is right there. The button is orange. You type in the user info and email, hit the orange button. The green message tells you everything is OK. Soon you receive an invitation from Slack. You check the StepFunction executions in AWS console and see everything is green.

No?! Most likely, you hit this apig-s3 bug, just like I did. And just like I did, you got to do a little final step manually.

Open AWS Console ->API Gateway->Resources->Actions-Deploy API.
Or use AWS CLI:

# Find out your rest-api-id firstaws apigateway get-rest-apis

# Deploy the changes to the stage# Don't COPY-PASTE, it needs YOUR ID!aws apigateway create-deployment --rest-api-id  YOUR_API_ID  --stage-name dev

Now it works for me.

PRO TIP: The plugin works when everything is deployed from scratch. But if you’re tempted to sls remove and redeploy again, you hit at least one other plugin bug on removing the stack. See Tips and Tricks at the end of the post on how to properly clean-up.

Congratulations! You made it. You might have followed along and built your own app. You might have dropped it at some point out of frustration or fatigue. You might have just skimmed the text and examples. But if you made it to the end, you learned enough of serverless to be dangerous.

Reflection

After wrestling through all the 4 episodes, you likely sensed it: serverless is not simple. Many things can go wrong as you build a serverless app.

You can blame some on me: I surely made a few mistakes and omissions (BTW reports and comments much appreciated!)

But serverless complexity is not all my fault. We wire together many different services — AWS and 3rd party — with many different wires — frameworks and tools. The services are of different quality and [in]convenience (don’t get me started on AWS StepFunction CLI). The frameworks and tools are still maturing (which serverless plugin made you struggle the most?).

Why get into serverless? Because for some classes of apps, it is substantially cheaper. This community signup is a case in point: we used to run StackStorm’s with StackStorm. It was simple to build and solid like a rock, but it took a m3.large instance at $100/month, without even getting to multi-zone reliability. With serverless it runs at less than $1/month, thanks to the occasional load pattern. $100 would buy us ~5,000,000 sign-ups — that’s more that we’ll ever need!

On the other extreme, high volume apps with high reliability requirements might be better as serverless too, leveraging “infinite” elasticity which may be expensive to build and operate yourself. I carefully use “might” and “would”, as the equation varies greatly from case to case. Do your math upfront, watch your load patterns, explore your bill.

As for taming complexity, Serverless framework helps. To fully appreciate it, try and redo the same app without it. Make it reliable, repeatable, reviewable infrastructure-as-code. While Serverless framework is not the only game in town, I specifically like it for it’s pluggable architecture and the plugin ecosystem.

Plugins help, picking up in areas not covered by the core Serverless framework. Using the plugins, we enjoyed the simplicity of building StepFuncions and adding simple web front-ends (and if you haven’t, try and redo it without the plugins). There are more: skim and bookmark the list of official plugins to keep them in mind when for your next project.

StackStorm Exchange — the recent addition to the serverless ecosystem — brings a catalog of reusable integrations. While it is not hard to scrape APIs for ActiveCampaign or find and use undocumented Slack API calls, it is low-value work that takes away from building the application logic.

Explore what is already there: while DevOps integrations dominate due to StackStorm roots in IT automation, I expect that variety will grow now that Serverless community is joining to contribute and co-pilot the catalog.

With that, we are officially done. Enjoy!

A few more Tips and Tricks

• Removing everything will not remove the DynamoDB table. A reasonable default, but when you try and re-deploy the service, it’ll cry that the table can’t be created as it already exists. Delete it: aws dynamodb delete-table --table-name signup-stormless-dev
• Due to an apig-s3 bug, sls remove will fail complaining that the bucket is not empty. Remove the web s3 bucket manually when removing a stack. aws s3 rb s3://bucketname --force
• Removing the stack does not remove the DynamoDB table. If you really want to start from scratch, delete it manually (export the data first):
  aws dynamodb delete-table --table-name slack-signup-dev
• Sometimes sls delete fails to clean up. Reasons might be various, but the place to look is the same. Did I tell you to master CloudFormation? Go there, find a stack that failed to delete, find the reason, fix, and delete the stack.

Hope this helped you learn something new, find something interesting, or provoked some good thoughts. Please share your thoughts in the comments here, or tweet me @dzimine.

Build a real-world serverless application on AWS with Serverless framework and ready-to-use functions from StackStorm Exchange open-source catalog.

Episode One | Episode Two | Episode Three | Episode Four

We are at Episode Three. Quick recap:

• In Episode One, I described the application we are building, walked you through setting up the development environment and creating a Serverless project, and showed how to build your first Lambda function from a StackStorm Exchange action with Serverless Framework.
• In Episode Two, we added more actions: one native Lambda to record user info to DynamoDB, and another one from StackStorm Exchange to make a call to ActiveCampaign CRM system. You learned more of serverless.yml syntax and practiced the development workflow with Lambda functions.

In this third episode, I’ll show how to use AWS StepFunction to wire the actions into a workflow.

You can get the final code for this episode from GitHub.

Wiring functions together with StepFunction

Now that our building blocks — Lambda functions — are all lined up, it’s time to string them together. An AWS StepFunction will define the sequence of calls, maintain the state of the sign-up workflow, and carry the data between the steps. I’ll use the[serverless-step-functions](https://github.com/horike37/serverless-step-functions) plugin from Serverless Champion @horike37, give him a heart:

Let’s get busy. Install the plugin:

npm install --save-dev serverless-step-functions

Add the plugin to the serverless.yml file:

plugins:  - serverless-plugin-stackstorm  - serverless-step-functions

The Step Function definition will require my accountID. As it is something I want to keep to myself, I add it to env.yml, which now looks like this:

# ./env.yml# Don't commit to Github :)

slack:  admin_token: "xoxs-111111111111-..."  organization: "your-team"active_campaign:  url: "https://YOUR-COMPANY.api-us1.com"  api_key: "1234567a9012a12345z12aa2aaa..."private:  accountId: "000000000000"

Go back to serverless.yml and add the following two blocks:

# ./serverless.yml......custom:  private: ${file(env.yml):private}  stage: ${opt:stage, self:provider.stage}  region: ${opt:region, self:provider.region}

stepFunctions:  stateMachines:    signup:      events:        - http:            path: signup            method: POST            cors: true      definition: ${file(stepfunction.yml)}

In the custom block, I assigned the private object from the private key in env.yml. I also defined variables for stage and region so that the values are picked from CLI options, if provided, or default to the current AWS settings.

The stepFunctions block is here to define - you have already guessed - StepFunctions. Mine is called "signup".

The events section here is doing exactly what events sections do in function definitions: it configures an API Gateway endpoint for invoking StepFunction from outside of AWS. We'll use it later to call the back-end from a Web form.

The definition can be written as YAML right here in serverless.yml, but I prefer to include it from a separate file, keeping the logic separate from the configuration. Here it is:

StepFunction definitions are written in Amazon States Language. The spec is short, well written and worth a read. Using YAML instead of JSON is a nice perk from the plugin — it reads better and allows comments. But if you want JSON — no problem, help yourself.

• Resource refers to Lambda functions by ARN. I used the variables we defined earlier to construct the ARNs matching account ID, region, and stage with the function name: arn:aws:lambda:${self:custom.region}:${self:custom.private.accountId}:function:${self:service}-${self:custom.stage}-RecordDB
• ResultPath is used to pass data between steps. By default, StepFunctions work on a "need-to-know" basis: the step downstream receives only the output from the step directly upstream. If you think it logical, think again: if only RecordDB receives the workflow input, how will RecordAC and InviteSlack get it? RecordDB may just return "200 OK", not email. Changing the code of functions to return their input would make them inappropriately intimate. The trick is to use ResultPath to write the function output under a function-specific key, like ResultPath: $results.RecordDB. This preserves initial workflow input in the step output for downstream Lambda steps, while appending the output of each Lambda. Like this:

{  "body": {    "name": "Vasili Terkin",    "email": "dmitri.zimine+terkin@gmail.com",    "first_name": "Vasili",    "last_name": "Terkin"  },  "results": {    "RecordDB": {      "statusCode": 200    },    "RecordAC": ...    ...  }}

To fully grasp it, read the “Input and Output” section in the spec. Ansd entertain youself with a video from “AWS Step Functions Tutorial” by Marcia Villalba.

PRO TIP: I made the workflow sequential to demonstrate the data passing trick. It is more proper to run all three steps in parallel: it is faster and more resilient: the failure of one step will not prevent the other function invocations. Go ahead change the StepFunctions to parallel.

That is it. Time to try. Deploy, invoke, check the logs.

You CURL fans know what to do with the new API Gateway endpoint for our StepFunction. If you forgot the endpoint, sls info to the rescue. I’ll show off again with httpie:

# DON'T COPY! Use YOUR ENDPOINT!

http POST https://YOUR.ENDPOINT.amazonaws.com/dev/signup \body:='{"email":"santa@mad.russian.xmas.com", "first_name":"Santa", "last_name": "Claus"}'

Ok, http or curl, either way it returns the StepFunction execution ARN so that we can check on it to see how our StepFunction is being executed. How do we check on it? I’m afraid you gotta open a browser and login to your AWS Console. If you want to use AWS CLI first, fine, don’t say I didn’t show you how:

aws stepfunctions describe-execution --execution-arn arn:aws:states:us-east-1:00000000000:execution:SignupStepFunctionsStateMac-seo9CrijATLU:cbeda709-e530-11e7-86d3-49cbe4261318 --output json{    "status": "FAILED",     "startDate": 1513738340.18,     "name": "cbeda709-e530-11e7-86d3-49cbe4261318",     "executionArn": "arn:aws:states:us-east-1:00000000000:execution:SignupStepFunctionsStateMac-seo9CrijATLU:cbeda709-e530-11e7-86d3-49cbe4261318",     "stateMachineArn": "arn:aws:states:us-east-1:00000000000:stateMachine:SignupStepFunctionsStateMac-seo9CrijATLU",     "stopDate": 1513738370.481,     "input": "{\"body\":{\"email\":\"santa@mad.russian.xmas.com\",\"first_name\":\"Santa\",\"last_name\":\"Claus\"}}"}

This is the output for an execution that failed because the RecordAC function timed out. Can you infer this from the output? The only valuable info here is FAILED. No kidding! I must say AWS don't give StepFunction the love it deserves. Not in CLI. If you think I missed something, check the CLI docs, find it and tell me.

The most irritating part is that the CLI doesn’t tell me which step failed. They make me call the logs on every Lambda, one by one. Luckily I only have 3 functions, what if there were more?

Or, open a browser and hop on AWS Console.

Even there, debugging some Lambda failures, like timeouts, is tricky. StepFunction execution “Exception” report says "The cause could not be determined because Lambda did not return an error type."Go to the lambda logs to see what happened there.

There I find the line which I wish I saw in the StepFunction exception:

2017-12-20T04:21:44.298Z 4230a73b-e53d-11e7-be6b-bff82b9b3572 Task timed out after 6.00 seconds

PRO TIP: For debugging, invoke the StepFunction with sls invoke stepf: it creates the execution, waits for completion, and prints the output to the terminal. Three AWS CLI commands in one.

sls invoke stepf --name signup \--data  '{"body": {"email":"santa@mad.russian.xmas.com", "first_name":"Santa", "last_name": "Clause"}}'

Your StepFunction executions may work just fine — we already adjusted the timeouts. I took you on this debugging detour for a taste of StepFunction troubleshooting, admittedly a bit bitter. On the sweet side, once debugged, StepFunctions run reliably like Toyota cars.

As a back-end developer, I am tempted to call it done here. But to make it a
“complete example” we need one more thing. The Web front-end.

Let’s call it a day and save the web part and conclusions for the next and final episode.

Episode 4: Adding Web Front-end, Reflection and Summary

Hope this helped you learn something new, find something interesting, or provoked some good thoughts. Please share your thoughts in the comments here, or tweet me @dzimine.