I’m recognized as an Amazon Alexa Champion and have published more than twenty custom skills on the platform. I continue to look for new ways to stretch this technology, create more robust skills, and share with the community.

Last September, Amazon released a new product for Alexa called Echo Buttons. These hardware devices expand the capabilities for the millions of customers that already have an Alexa. They’re relatively inexpensive, right now going for $20/2 pack.

In April of this year, Amazon opened up the platform for developers like me to begin incorporating them into custom skills. This is my experience so far programming with them.

Echo Button in inactive state.

Getting Started with Echo Buttons

Network connectivity of Echo Buttons

Buttons are paired with Alexa devices through a bluetooth connection, becoming a physical extension of the speaker. All network traffic from the button to use Amazon services goes through the paired device. Don’t buy buttons if you don’t already have an Alexa, as they don’t do anything on their own.

If you have multiple speakers associated with your account, the button will only work with the device it’s currently paired with. The pairing can be undone with just a few steps on your phone. To unpair a device, just use the Alexa companion app and go into the settings section. Select the button that has already been paired, and it will release it. Then go back through the steps to repair to another device.

Screenshot of Alexa App on a Mobile Device

This makes buttons an easy add on for Alexa enthusiasts, and makes them flexible for those with multiple devices.

How Buttons Interact with Custom Skills

Alexa is an event driven architecture. Events are created by the sounds going into the speaker that are translated through Cloud based service. Normally these events are initiated after the array of microphones picks up a command from a user. These commands are then translated by the ASR (Automated Speech Recognition) models depending on which custom skill is being used.

With buttons, a new type of event is created that follows a similar pattern. When the button is pressed, an event is created and sent to the skill that runs in the Cloud. When the button is released, a separate event is created. These events are created independently of anything being picked up by the microphones on the Echo Speaker and flow through a Cloud based service called the Game Engine.

Alexa Event Architecture including Buttons

The Lambda function that contains the logic for the custom skill needs to process these events along with the existing voice triggered ones. Translating request objects is facilitated by the Alexa SDK installed in the Lambda function. Examples of how this works are provided in the Alexa repo.

This opens up new possibilities for gameplay, as a user can leverage both their voice and hands to interact with the skill. Engaging more senses broadens the experience, and enables more complex gaming. For example, in the Seventh Inning Stretch skill, a user can play a baseball game listening on their speaker while pressing the button to swing the bat.

How Skills use the Gadgets API

Using buttons within a custom skill requires the Gadgets API. Documentation is currently on this website, and note that it is still in beta. Buttons are just a type of gadget, and provide a glimpse into what’s possible with enabled hardware.

Connectivity between systems is facilitated by interfaces across the internet. The API that buttons need is invoked by adding directive attributes to a standard Alexa response object. This enables the SDK to handle the explicit details for the HTTPS call (i.e. encoding the header, setting the attributes, error handling, etc.)

Here is an example of adding a directive to set the lights on a button at the same time that the Echo speaker reads back an introduction to the user.

"response": {               "shouldEndSession": false,               "outputSpeech": {                     "type": "SSML",                     "ssml": "<speak> Welcome back to Seventh Inning Stretch.<break time=\"1s\"/>We found an prior game in progress. Would you like to resume? </speak>"                 },               "reprompt": {                     "outputSpeech": {                           "type": "SSML",                           "ssml": "<speak> Say yes to resume the in-progress game, or no to delete it.  </speak>"                     }               },               "directives": [    {                           "type": "GadgetController.SetLight",                            "version": 1,                           "targetGadgets": [],                           "parameters": {                      "animations": [               {                         "repeat": 1,                            "targetLights": ["1"],              "sequence": [                          {                    "durationMs": 30000,                                                               "color": "FFFF00",                  "blend": false                                                     }              ]                      }           ],      "triggerEvent": "buttonDown",                                "triggerEventTimeMs": 0    }}

The Game Engine creates events just like the Echo speaker. The same taxonomy is used, and the attributes within the request identifies event details. Below is an example of a request indicating that a button was pressed.

“request”: {   “type”: “GameEngine.InputHandlerEvent”,   “requestId”: “amzn1.echo-api.request.xxx”,   “timestamp”: “2018–07–21T21:33:25Z”,   “locale”: “en-US”,   “originatingRequestId”: “amzn1.echo-api.request.xxx”,   “events”: [     {       “name”: “button_down_event”,       “inputEvents”: [ {         “gadgetId”: “amzn1.ask.gadget.xxxx”,         “timestamp”: “2018–07–21T21:33:25.374Z”,         “color”: “000DD6”,         “feature”: “press”,         “action”: “down”       } ]     }   ] }

Save Battery Life through Roll Call

Echo Buttons are battery powered, making energy management important. When a custom skill requires a button, it must initiate a connection and wake the button. This is done through a process called ‘roll call’ within the custom skill.

To initiate a roll call in a custom skill, add a directive to the response object providing the parameters to perform the task. In parallel, audio instructions need to be included in the response object. These will encourage a user to do something with the buttons. For example, ask the user to press each button to get started.

For specifics on what a directive looks like, here is the roll call directive I use for my Seventh Inning Stretch skill. It’s a series of attributes in a large JSON object. This sets the timeout parameter for when the buttons can go back to sleep if not used (the 300,000 value is in milliseconds — this translates to five minutes), and looks for just the button down event.

"directives": [  {     “type”: “GameEngine.StartInputHandler”,     “timeout”: 300000,     “recognizers”: {       “button_down_recognizer”: {         “type”: “match”,         “fuzzy”: false,         “anchor”: “end”,         “pattern”: [{ “action”: “down” }]       }    },     “events”: {       “button_down_event”: {         “meets”: [“button_down_recognizer”],         “reports”: “matches”,         “shouldEndInputHandler”: false       },       “timeout”: {         “meets”: [“timed out”],         “reports”: “history”,         “shouldEndInputHandler”: true       }     }  }]

The buttons handle the timer for when to turn off. This minimizes the risk of battery drain when a user ends their session. Skills that use buttons should also make a request to the shut off the device if the skill is exited.

Translating Game Engine Events

The Game Engine can create events just like the speaker after it is awake. The request uses the same taxonomy, and the attributes identify details of the event. Here is an example of a request indicating that a button was pressed.

“request”: {   “type”: “GameEngine.InputHandlerEvent”,   “requestId”: “amzn1.echo-api.request.xxx”,   “timestamp”: “2018–07–21T21:33:25Z”,   “locale”: “en-US”,   “originatingRequestId”: “amzn1.echo-api.request.xxx”,   “events”: [     {       “name”: “button_down_event”,       “inputEvents”: [ {         “gadgetId”: “amzn1.ask.gadget.xxxx”,         “timestamp”: “2018–07–21T21:33:25.374Z”,         “color”: “000DD6”,         “feature”: “press”,         “action”: “down”       } ]     }   ] }

The logic within the Lambda function for the skill will need to respond to these events, and process functionality accordingly.

Buttons Can Change Color

Inside the buttons are a series of LED’s that can be turned off and on. They are bright and the color is very rich.

The buttons can also change colors by altering how the different LED’s are illuminated.

Color changing of the buttons is also done within directives in a response object. If you start your skill using the repo for buttons, there are helper functions that make this easy to integrate into your skill.

Closing

If you’re interested in trying out a game that uses them, please test out my baseball simulation game on Alexa. It’s called “Seventh Inning Stretch” and is an attempt to recreate the fun of old handheld games from the 80’s. It’s a good example of what’s possible using these new accessories.

Example of Alexa Skill that uses Buttons

And why you would do such a thing

There are two ways to implement custom skills for Alexa.

The first is the most common and is the Amazon-recommended way. Use AWS Lambda, a server-less computer service. There is no shortage of articles and tutorials on the topic. This is not one of them.

The second way is far less common. This is hosting an endpoint using a HTTPS web service that you manage. It is a bit more difficult to find good examples of this approach. This article will attempt to do just that, and will use Spring Boot as the basis for the implementation.

Endpoint configuration in the Alexa Skills Developer Console

But before we jump into the how, let’s talk about the why:

• Why would you not use AWS Lambda?
• Why ignore an Amazon recommendation?

These questions are important. You’ll find reams of great examples and documentation on building Skills with Lambda, and not so much for the alternative. It’s also important if you buy into the premise — which I do — that the world is going server-less. This is a great article on that topic by James Beswick. Going the HTTPS route will lead you down a lonely path, but sometimes that’s OK.

Here are some reasons why you may want or need to take that lonely path.

• You can write your Alexa web services using any programming language.
• If you have existing RESTful services already deployed and you want to leverage that infrastructure/investment.
• If your CISO does not allow off-premise or cloud-based infrastructures.
• If you like lonely paths.

In my case, I decided to explore the non-Lambda path primarily because I already had an existing services tier, which I wanted to leverage. There are POJOs and methods I wanted to reuse, without having to expose new endpoints. Of course, I could have created Lambdas that simply fronted those services, and that is a valid pattern. But I didn’t want to add another layer of deployment scripts, testing, and monitoring.

You probably have your own reasons. If I were building a new greenfield application, I’d probably head down the Lambda path.

Prerequisites

You’ll use the Alexa Developer Console to register your skill, define the intents and utterances, and test it. I’ve assumed that you’ve already created a developer account, and that you can setup a new custom Skill within the console.

A Spring Boot Alexa Skill

The rest of this article will show you how easy it is to add an Alexa Skill to your existing Spring Boot applications. To demonstrate this, we’ll create a skill that will lookup fun facts about either a specific year, or a random year. You’ll be able to interact with our skill by saying things such as:

Alexa, ask my demo app to tell me trivia about a random year.Alexa, ask my demo app, what happened in the year 1984?

Okay, yes, this is a pretty useless skill. But it will demonstrate all of the important aspects of an Alexa app, including:

• custom intent handling
• using slots, session management
• built-in intent handling

To accomplish this, our example skill will call a free third-party API to lookup the trivia information, using the wonderful NumbersAPI. Many thanks to David and Mack for creating this fun service.

Getting started

First things first. Add the Alexa Skills Kit SDK to your “pom.xml” file. At the time of writing, the latest version of the SDK is 1.8.1.

<dependency>    <groupId>com.amazon.alexa</groupId>    <artifactId>alexa-skills-kit</artifactId>    <version>1.8.1</version></dependency>

The SDK includes a special servlet, named SpeechletServlet, which you will need to load as part of the app bootstrap. This is actually very easy to do.

The servlet is a workhorse. It handles all of the messy requirements for hosting a skill, such as verifying that the request was sent by Alexa, validating the signature, and checking the timestamp. Fortunately for us, we don’t have to deal with any of those headaches. We just need to load the servlet. Here’s how we do that:

Simple. You can see above that we’ve created a configuration class that loads our servlet. On line 10, you can see that SpeechletServlet is instantiated, and then on line 13 it is registered with Spring.

That’s pretty much all you need to do to load the servlet.

As I noted, the servlet takes care of all the complicated handshake communications with Alexa. And after it does that, the servlet delegates the actual interaction business logic to a Speechlet, which you must implement. You can see on line 11 that the Speechlet, which I named HandlerSpeechlet, is assigned to the servlet. This Speechlet will be invoked with every Alexa interaction.

A Speechlet is just a POJO that conforms to the SpeechletV2 interface that is defined in the Alexa Skills SDK. Here is what the interface looks like.

It’s your job to implement those four methods.

They’re all important, but most of the work happens in OnIntent(), which is invoked when the user says something meaningful. If you’re new to the vocabulary of Alexa programming, you should read Intents, Utterances, and Slots: The New Vocabulary Needed To Develop For Voice.

Defining intents

Jump into the Alexa Developer Console. Set up a new custom skill within the console — that part is very easy. I’ve named my skill “MyDemoApp”, and under the Invocation menu, I set the Skill Invocation Name to “my demo app”.

Recall earlier in this article I said that you can interact with our skill by saying things such as:

Alexa, ask my demo app to tell me trivia about a random year.Alexa, ask my demo app, what happened in the year 1984?

You can see that each of the above sentences start with the wake word (“Alexa”) and the Skill Invocation Name (“my demo app”).

Alexa, ask my demo app ...

Everything after the Skill Invocation Name is known as an utterance. It’s your job to list utterances in the developer console, and map those to an intent. A single intent normally has many utterances, representing the variations a user might say. For example, all of the following utterances are essentially the same:

tell me trivia about a random year.say something about any year.pick a random year.tell me some trivia.say anything about any year.

All of these utterances mean the same thing, and therefore can map to a single intent. In my application, I’ve called this the “RandomYearIntent”. Here’s what it looks like in the developer console.

Back to writing code

With our first intent defined, it’s now time to jump back into our Spring application and write some code. Let’s edit our “HandlerSpeechlet”. For the moment, let’s skip the onSessionStarted and onLaunch methods of the Speechlet, and jump right into the onIntent method.

Because our application will eventually handle multiple intents, we’ll first need to determine which intent is being invoked.

On line 10, we now have the name of the intent. In our example so far, this should be RandomYearIntent. At this point, you might be tempted to write a bunch of if-else statements against the intent name, but let’s try something a little smarter.

By making sure that our intents follow a specific naming convention, we can use some Spring magic to load and invoke specialized handlers for each intent. What follows is not something specific to the Alexa Skills SDK. It’s just my own way of handling multiple intents. There are many ways to implement this logic. Here’s mine.

Let’s start by defining an interface which we’ll use for all of our handlers. Let’s create IntentHandler as:

Next, for each intent, we’ll create a class that implements this interface. We’ll also be careful to name our classes the same as the intent name, with the word “Handler” appended. For example, for the intent “RandomYearIntent”, we’ll create a class named RandomYearIntentHandler.

Okay, let’s leave this unimplemented for now. We’ll go back to our “HandlerSpeechlet” to add code that will pass control to our new “RandomYearIntentHandler”. The basic strategy is to rely on our naming convention of “IntentName” + “Handler”.

I have removed some details and error handling from the code below so that we can focus on the important bits. At the end of this article, you can find a link to my GitHub repo with the complete code.

Check out lines 12 and 13 below, which construct the class name, then ask Spring to find a registered bean with that name. Then, finally, on line 17, we pass control to that handler.

RandomYearIntentHandler

We now have the plumbing in place to call our specialized intent handlers. Let’s go back to the implementation of our “RandomYearIntentHandler”. This particular intent is simple. Here’s the complete implementation.

Simple! Did you catch the annotation on line 1? That @Component annotation is a cool little trick that will tell Spring to create an instance of this class as a Bean. This is how we’re able to access the handler in our Speechlet, using beanFactory.getBean(handlerBeanName).

On line 11 we create a random number between 1900 and the current year. Then we call the Numbers API to get trivia about that year.

On lines 14 and 15 we create a Card and a Speech. The Card is what is displayed in your Alexa mobile app — or on the screen in the Echo Show. The Speech is what is spoken back to the user.

AlexaUtils is a simple class I created. I won’t go into the details here, but you can review it on GitHub.

That was easy — what about slots?

Slots are basically variables in utterances. Look back at the two intents in our application. The second intent, which I’ll name “SpecificYearIntent”, allows the user to say any year. For example:

Alexa, ask my demo app, what happened in the year 1984?

In the above utterance, the year is highly variable. We do not want to define an utterance for every possible year. Instead, we’ll define this utterance using a slot, as:

Alexa, ask my demo app, what happened in the year {Year}?

{Year} represents a number. Jumping back into our developer console, we’ll setup a new “SpecificYearIntent” intent with its associated utterances as follows:

SpecificYearIntent utterances

I’ve defined a Slot named “Year”, which is of type “AMAZON.NUMBER”. Now, in my handler, I can fetch the slot value easily by name.

Line 13 is where we resolve the year Slot. The rest is all boilerplate Java code, which follows the same pattern as the first intent — call the NumbersAPI service for the year and handle the response.

Sessions and state

I didn’t get into the OnSessionStarted or OnLaunch methods of the Speechlet in this article, but I did include an implementation for OnLaunch in the sample project in GitHub.

You can use the Session object to store variables that will be retained across invocations when in conversation mode. Conversation mode occurs when the user invokes your skill but does not speak a recognized utterance. For example:

Alex, open my demo app.

>> Hello.  Here are some things you can say: Tell me something about a random year.  Or, what happened in nineteen eighty-nine?

What happened in the year 1984?

>> 1984 is the year that the European Economic Community makes £1.8 million available to help combat the Ethiopian famine on October 25th.

>> What else can I tell you? Say "Help" for some suggestions.

Cancel.

>> OK.  Goodbye.

You’ll notice in my source code that I have a method named setConversationMode. It simply sets a variable in the Session, letting me know that we are in conversation mode.

Summary

Writing Alexa skills with Spring Boot is quite easy, and can be a powerful way to reuse your existing infrastructure. Download the complete app from GitHub for more details.

And if you’re in the South Florida area, you can use my Cutler Stew Skill (built with Spring Boot) to learn when and where my band Cutler Stew is performing next.

I’m both a sports nut and a software engineer. I’m also recognized as an Amazon Alexa Champion. I continue to look for new ways to stretch this technology.

Over the past two years, I’ve won hackathons for skills on the Alexa platform.

Skills are the part of the Alexa voice service that activates its capabilities. In Alexa, skill is a term used for capabilities that make an experience more personal. You can enable or disable them as you choose. And, with the Alexa Skills Kit, You can create and customize them.

Sports games are an enormous market

Video games are a huge market, with annual revenues projected at more than $100B worldwide. Shooter and action games are the most popular, followed by sports games.

Voice platforms are growing at a fantastic rate. The popular Amazon Alexa platform has grown by 4 times in the past year. There are now 20,000 custom skills on the Alexa platform. Yet no sports games are more complex than calling out trivia.

Here is a sample of my new football game called End Zone Football. This shows how an advanced game can work on the Alexa platform.

Begin game design with Storyboards

Designing the skill requires writing storyboards to script the action. Start with how the game will begin. Then write the narrative for basic gameplay.

Consider yourself to be a playwright or a movie director. Ask yourself: What should the native Alexa voice say? What sounds can be played that are part of the game?

Here’s what I learned from publishing a Baseball and Football game on Alexa. These tips will improve the usability of your game:

• Keep the interaction simple
  yes or no and 1/2/3/4 answers work best
• Identify sounds that can make the gameplay more exciting
  Crowd cheering, whistles, or the crack of a bat
• Simplify the game
  The Baseball game skills should include foul balls
  The Football game skills should simulate penalties
  These help keep the user engaged
• Don’t rely on visuals
  A background image for the Echo Show is nice, but the images on the companion app should be secondary
• Bound the game to 2–5 minutes
  Users can play again and again if they have time
• Create a help function that explains the game in detail
  Give examples of phrases to use to play
• Above all else
  Make it Fun!

Learn SSML to include sounds with voice

Speech Synthesis Markup Language (SSML) integrates sound with voice. It’s used to create the audio for the Alexa device.

This is how I set the main response attribute to call the Alexa API. After splicing the strings in JavaScript, the code looks like this:

var speechOutput = “Welcome to End Zone Football. “ +   “<audio src=\”” + bucketLoc + “BandMusicIntro.mp3\” />” +  “The game that lets you drive the “ +   “ball down the field using just your voice. “ +   “<break time=\”1s\”/>” +   “You are in charge of the Blackbears, and are down “ +  sessionAttributes.away + “ to “ + sessionAttributes.home + “. “ +   “<break time=\”1s\”/>” + “The ball is on the “ +   yardline + “ yardline. “ +   “When you are ready, just call out the play you want to run, and the game will begin. “ +   “For a list of plays at anytime say, Read Playbook. “;

Here are some examples of how to use SSML in this context:

• To create a pause
• To insert a soundclip
  <audio src=”https://s3.amazonaws.../file.mp3">

I recommend using an S3 bucket to store the media files. This is a low-cost way of storing data. And they can be accessible to Alexa.

State management tracks the progress of play

Alexa has the ability to store the gameplay state. Use this feature to simplify the coding and testing of your skill.

In the Node.js SDK, a session attribute is passed with the request and response attributes. You can use the session attribute for the gameplay state. Store critical game information in this field. You can include anything, like the play number or recorded number of outs.

Here is an example used in the Football game

// save the game attributes through to the next play    if (session.attributes) {          sessionAttributes = session.attributes;    }...

// gameplay rules for passingif (offensivePlaybook[i].playType === “pass” &&    offensivePlaybook[i].playNumber.toString() ===     intent.slots.playNumber.value) {       console.log(“Matched Play Number”);       // calculate pass distance based on play selected      passDistance = Math.round(Math.random() *         (offensivePlaybook[i].maxYardage            — offensivePlaybook[i].minYardage)            + offensivePlaybook[i].minYardage);       // make sure distance of play can’t be longer than       // the remaining field       if (passDistance > sessionAttributes.yardline) {         passDistance = sessionAttributes.yardline;       }       playDesc = offensivePlaybook[i].playDesc;       speechOutput = speechOutput + playDesc + “. “;       // based on the play selected, determine relative       // completion rate       passCompletion = offensivePlaybook[i].completionRate;)...// pass back the response to Alexa, and save the gamestatecallback(sessionAttributes,                 buildSpeechletResponse(cardTitle, speechOutput,       cardOutput, repromptText, device, shouldEndSession));

Storing this data in a table for analytics is helpful, but is not needed in an initial version. Let the Alexa platform do this work for you.

Advertise your skill on social media

People don’t yet realize all the things that an Alexa can do. When doing demos of these skills, I consistently get feedback that “I didn’t think Alexa could do that.”

All skills must be certified before publication on the Amazon Store. When your skill passes the certification process, take the time to record a video of it in action. It is the best way to prove what the platform is capable of.

Thank you for reading, and please enjoy these games — they’re free!

Coding for social good is a welcome movement. And it translates advancements in technology into social advancement for people everywhere.

“Technology lets you see problems in ways my friends and I never could, and it empowers you to help in ways we never could. You can start fighting inequity sooner, whether it is in your own community or in a country halfway around the world.” — Bill Gates, Letter to the Class of 2017

Organizations like freeCodeCamp are on the leading edge of this movement. The community brings together aspiring software developers to build open source projects. Their coding contributions to the projects help nonprofits carry out their missions more effectively.

Every day, coders around the world are making significant contributions toward solving social issues and supporting non-profits, human rights, global literacy, the environment, educators, or individuals in need.

Speak Up!

To help bring technologists together with social organizations that struggle with difficult challenges, A Cloud Guru launched the Alexa “Speak Up!” Challenge.

The premise of the “Speak Up!” is simple — learn to code an Alexa skill, impact the greater good, and inspire change.

Publish a skill by August 18th and enter the Alexa “Speak Up!” Challenge

Be the voice of social good

The challenge is inspired by Cameron Gallagher, the daughter of my close friend. Cameron’s dream was to create a 5k race in our community to raise awareness for teenage depression.

She planned for her race to be named the SpeakUp5k to let other teens like herself know that it was okay to speak up about their personal battles.

Cameron’s dream was to create a 5k race in her community to raise awareness of teenage depression

Before her SpeakUp5k could launch, sixteen-year-old Cameron passed away suddenly from an undiagnosed heart condition after crossing the finish line of a half-marathon.

During her final race, she looked at friend after struggling with cramps and said “Let’s finish this.” The CKG Foundation carries this message forward in her honor — and so does this challenge.

Why Alexa?

The user base of voice-assisted devices is already very large — and growing fast. More than 35 million Americans will use a voice-enabled speaker at least once per month in 2017. With over 70 percent of the U.S. market share, Amazon is currently dominating the voice-enabled speaker market.

Individuals of any age who use a voice-enabled speaker. Source: eMarketer, April 2017

By the end of 2020, Gartner’s expects that 75% of US households will have a voice-enabled smart speaker. At that point, the install base of Alexa devices is forecasted to be 128 million users.

With a massive user base and a simple Alexa Skills Kit (ASK) for coders, the Amazon Echo provides software developers a powerful platform for amplifying your message of social good into the homes of millions of users worldwide.

Let’s get started — Hello Word

Designing, developing and publishing an Alexa skill is really very easy — a basic skill can be completed within an hour.

1. A Cloud Guru provides a free Alexa course that’ll walk you through the entire development process — from creating an account through publishing the skill.
2. Explore Amazon’s Alexa Github repository of templates for simple guides that explain the basic building blocks and concepts of skill development.
3. Don’t have an Echo? No problem. EchoSim.io is a browser-based interface that allow developers to test their skills in development.

Need some inspiration … just say “Alexa, Inspire Me!”

How to measure the success of your code?

For any product in the early stages of development, it’s important to maintain focus by picking a single metric to measure progress. The Lean Startup movement advocates that startup use One Metric That Matters (OMTM) to track customer impact.

The purpose of the OMTM is to track that the changes being made to the product are having a measurable impact on the behavior of your customers.

For coders participating in the “Speak Up!” challenge, the OMTM is the number of customer lives touched by your sessions of social good. Once you’ve published an Alexa skill, the Amazon developer’s console provides a simple interface for tracking this metric.

Metrics from my youngest daughter’s Harry Potter Alexa skill — Potterhead Sorting Ceremony

With millions of households listening, each invocation of your skills offers the potential to inspire, create awareness, and advance social good on a massive scale.

And now that Amazon is offering incentives to developers of top gaming skills, creating a fun and engaging Alexa skill for social good could also result in financial rewards to contribute to your cause. At the very least, your published skill will earn you some free Alexa swag!

“You must be the change you wish to see in the world.” — Mahatma Gandhi

Consider participating in the Alexa Speak Up! Challenge — learn to code an Alexa skill, impact the greater good, and inspire change.

“Let’s Finish This!”

Thanks for reading! If you like what you read, hold the clap button below so that others may find this. You can follow me on Twitter.

I’m an Amazon Alexa Developer Champion, and have published more than a dozen skills on the Alexa platform. This includes award winning skills like Hurricane Center, Scavenger Hunt, and Robot Roxie.

The latest device added to the Amazon Echo family features a screen that complements its audio experience. It’s called the Echo Show, and here are the hardware specifications. It features a 7" touchscreen display, a 5 Mega-pixel camera, and dual 2 inch stereo speakers.

In this article, I’ll show you how to design and code a new Alexa skill that takes advantage of this new hardware.

Major Upgrades to the Software Platform

These hardware improvements required an upgrade to the underlying software running the platform. The Alexa Skills Kit (ASK) introduced a major release to power the added screen features. This is the largest set of enhancements to the ASK since the platform launched.

Background on Skill Building

If you’re not familiar with developing Alexa applications — called “Skills” — here’s a brief overview.

• A custom voice application interfaces with the natural language processing engine. This interprets audio commands provided by the user. Each application includes a customized intent model to direct the skills messages. A set of sample utterances train the voice recognition models which intent in the model to use.
• A unique API receives all messages for the custom skill. Alexa requires a RESTful interface to be specified in the custom voice application. The skill author writes the API, and chooses the hosting platform. The preferred hosting platform is the AWS service Lambda. This is due to the slick compatibility of the security model across Amazon and low hosting costs. The API call contains a standard set of attributes relating the call made by the device. This contains data around the current invocation of the skill.

Here is an example of a request to the Hurricane Center skill API:

{  "session": {    "sessionId": "SessionId.5725be2d-99f8-4afd-909f-1f1d3882067a",    "application": {      "applicationId": "amzn1.echo-sdk-ams.app.709xxx"    },    "attributes": {},    "user": {      "userId": "amzn1.ask.account.AFP3xxx"    },    "new": true  },  "request": {    "type": "LaunchRequest",    "requestId": "EdwRequestId.260fd856-668f-4dd2-af9d-60d80e4cc8e0",    "locale": "en-US",    "timestamp": "2017-07-02T00:54:57Z"  },  "version": "1.0"}

The API for the custom skill processes the request, and responds with values for the Alexa device. The message model returned is a standard set of attributes, and written in JSON. The Alexa voice on the device reads the outputSpeech attribute.

Here is an example the corresponds to the call made to Hurricane Center skill above:

{ “version”: “1.0”, “sessionAttributes”: {} “response”: {     “outputSpeech”: {         “type”: “PlainText”,         “text”: “Welcome to the Hurricane Center, the best source for information related to tropical storms, past or present. There are no active tropical storms right now, but if you would like to learn more about storms, please say something like tell me a storm fact.”     },     “card”: {         “content”: “No Current Storms in either the Atlantic or Pacific Ocean.”,         “title”: “Welcome to Hurricane Center”,         “type”: “Simple”     },     “reprompt”: {         “outputSpeech”: {             “type”: “PlainText”,             “text”: “Please tell me how I can help you by saying phrases like, list storm names or storm history for 2013.”         }     }, }, “shouldEndSession”: false }}

What Changed for the Echo Show?

To leverage the new visual features of the Show, the model for the response expanded. The model now includes new attributes to use the new hardware features. The custom written API for the skill includes these attributes in its response.

New templates supporting different use cases for the screen are central to it. These templates enable the Show to augment the voice experience. Full detail on these changes is available on the Amazon Developer website.

Six visual templates to choose from

There are six templates available with the initial launch of the new ASK. Each template provides the ability to leverage the touch screen on the Echo Show. There are two types of templates. Four of them provide basic usage of the screen, the other two templates add the ability to manage lists of data. The skill developer selects which template is most applicable for each use case.

Echo Show supports basic HTML tagging

The Alexa platform already has a visual option before these changes. This is within the mobile application on your device pairs with — named the companion app. This performed initial setup and configuration by pairing with the device. Some skills use it to complement the voice first user experience. Here is what it looks like for Hurricane Center.

The companion application has limits in how it can render information. All words on the screen are plain text. It does not support HTML, including changing the font type, font size, underlining, etc.

An optional small image displays on the screen that changes with each voice response. Usability on the app is poor when rendering a large amount of data given the clear text constraint. Usability is also poor when rendering lists of data which may be important to the narrative of the skill.

Within the templates of the new ASK, both simple text and basic HTML syntax are now supported. Pairing this with the list template provides a whole new level of user experience.

New intents to handle touch events

The screen on the Echo Show does more than act as a display, it also has touch sensors. Interacting with the screen creates events along with audible utterances. These include, back-button, right, left, etc. as the user can gesture with text and lists on the screen.

The device handles most of the new standard intents. They can be incorporated into your skill when rendering lists.

Existing Skills work on the new Device

All 15,000 existing skills on the platform work with the Echo Show. They will use information that is currently rendered on the companion application. Here is an example with the Hurricane Center skill.

The skill works, but the usability of the screen is poor as the font is too large. The screen shows the same information that shared with the companion application. Without changes, it isn’t using a template, so there is no scrolling enabled in case the user wants to read the full text.

For reference, here is the GitHub repo that contains the entire source code for this skill.

terrenjpeterson/hurricane
_Alexa Skill for providing hurricane updates_github.com

Extreme Makeover — Echo Show Edition

Here are the steps required to take advantage of the new features in the ASK. These changes will leverage the hardware capabilities for users with an Echo Show.

Step 1 — Activate template feature for the skill

Activating it for a skill requires selecting a radio button on a new or existing skill. There is a new indicator to “Render Template”. Once it is set, the skill can now leverage the extra abilities of the Echo Show.

Step 2 — Select a template, then add attributes to the API response

The ASK now contains templates outlining the visual patterns supported by the Echo Show. These map to different UI patterns, including displaying images and lists. There is good reference documentation on the Amazon developer website.

For my Hurricane Center skill, I’ve chosen to use the “BodyTemplate1” for the main display. This takes advantage of the ability to display a background image on the Show. It also renders an improved user experience for the textual response. Here are the new attributes to include to the existing JSON response.

directives: [ {   type: “Display.RenderTemplate”,   template: {       type: “BodyTemplate1”,       token: “T123”,       backButton: “HIDDEN”,       backgroundImage: {           contentDescription: “StormPhoto”,           sources: [               {                  url: “https://s3.amazonaws.com/hurricane-data/hurricaneBackground.png”               }           ]      },      title: “Hurricane Center”,      textContent: {          primaryText: {              text: cardInfo,              type: “PlainText”          }      }  }}],

The background image used matches the dimensions of the Echo Show — 1024 x 600 pixels. Hosting images follows the same pattern as with the companion application. The image object needs to be accessible from the internet without any authentication.

Step 3 — Publish a new version of the skill

Rendering templates changes the skill, requiring to go through the certification process again. It doesn’t need publishing a separate skill for the new device. This means all the prior ratings and users that have enabled it on their accounts will carry forward. Test the skill on other devices to make sure that nothing broke.

New Version of the Skill

Here is what the hurricane skill looks like after the work described above.

The photo now appears as a background image rather than the default canvas. The text intended for the companion application renders on the Echo Show screen. The font size is appropriate, in a readable format. This is a major improvement in usability, with not a significant effort in coding.

Backwards Compatibility for all Devices

We want the skill to continue to work for existing devices given that the Echo Show is new. That requires a little extra logic when formatting the response in the API. The interface contract requires the API provide only attributes readable by the device. This means extra attributes intended for the Echo Show creates errors when sent to an earlier Echo.

Stopping this error requires interpreting the context section of the native request object. This section provides information about the device, including if it contains a display. This attribute is only present with an Echo Show. If this attribute is not present, the API excludes the directives attribute. Here is a sample of a request from the Hurricane Center skill generated by an Echo Show.

“context”: {     “AudioPlayer”: {         “playerActivity”: “STOPPED”     },     “Display”: {         “token”: “T123”     },     “System”: {         “application”: {             “applicationId”: “amzn1.echo-sdk-ams.app.709xxx”         },         “user”: { “userId”: “amzn1.ask.account.AFP3xxx” },         “device”: {             “deviceId”: “amzn1.ask.device.AFAQxxx”,            “supportedInterfaces”: {                 “AudioPlayer”: {},                 “Display”: {                     “templateVersion”: “1.0”,                     “markupVersion”: “1.0”                 },             “VideoApp”: {}         }     },}

In the example above, the System.device.supportedInterfaces.Display attribute exists, indicating that the request is originated by an Echo Show. Here is the request from the same skill generated by an Echo Dot.

“context”: {     “AudioPlayer”: {        “playerActivity”: “IDLE”     },     “System”: {         “application”: {            “applicationId”: “amzn1.echo-sdk-ams.app.709xxx”         },         “user”: { “userId”: “amzn1.ask.account.AFP3xxx” },         “device”: {             “deviceId”: “amzn1.ask.device.AFAQxxx”,             “supportedInterfaces”: {                 “AudioPlayer”: {}             }         },         “apiEndpoint”: “https://api.amazonalexa.com”     } }

In the second example, there are no display attributes. That’s easy to recognize, and now reflected in my skill.

What’s Next?

It’s amazing the speed at which the platform is growing. There are now 15,000 custom skills available for users to try out. The hardware options are enabling even deeper user experiences. I will be spending the next few months porting over my skills to the new platform. Hope this helps updating yours!

Learning how to play guitar takes practice. A lot of practice. One of my self-improvement goals is to get better playing the guitar.

An area I’m working on now is finger positioning on the fretboard. This requires memorization of different patterns of where to touch the strings. Improvement takes frequent practice and coaching, but is quite rewarding.

Researching what’s available on the internet showed me many musicians lending a hand. A popular medium where they are sharing is YouTube. The video below has more than five million views, demonstrating the broad usage. The instructor steps through the basics of finger positioning and notes.

These videos are great for getting started, but what gets lost is how to focus completely on the music. Focusing my attention on finger placement and note recognition is how I move forward.

I tried different YouTube artists in an attempt to find the perfect coach. The limitation I continue to run into is not the artist or the audio quality of the device. Rather the limitation is the medium itself of a video playing on my phone or laptop. When practicing, I never get any exercise correct in the first attempt. Given the interface of YouTube, it’s difficult to playback sections given that my hands are on my guitar.

So I built the Alexa Guitar Teacher Skill

This highlights where interactive voice applications are better than traditional mobile platforms. A voice application doesn’t need use of your hands. Playback of a section requires my voice, and my hands stay on the guitar. For learners like me that need to repeat lessons to get them correct, being hands-free is a huge asset.

I tried out what was available in the skill store, but found skills that were very simplistic . Some didn’t even feature the sound of a guitar, rather a monotone voice giving instructions. So I did what any software engineer would do, and came up with one myself called “Guitar Teacher”.

The Alexa platform has the ability to interact with someone using only their voice. This frees up fingers to focus on playing the strings. Here’s a quick demonstration of the skill with some of the core features.

How does it work?

Despite what you see with most skills on the platform, Alexa can do more than speak words. More complex voice applications use a syntax known as SSML. This mixes together the machine voice with sounds that recorded to a Mp3 file.

Here’s the syntax required that uses Alexa to explain how to position your fingers to play a standard chord. This section of the code plays sound of the chord to reinforce what comes out of an actual guitar.

{“outputSpeech”: {“type”: “SSML”,“ssml”:“<speak>Okay, let’s get started on how to play the chord C Major.<audio src=\”https://s3.amazonaws.com/.../Chordcmajor.mp3\" /><break time=\”1s\”/>Here are the finger positions. Your index finger will be on string 2 pressing down on fret 1.<break time=\”2s\”/>Your middle finger will be on string 4 pressing down on fret 2.<break time=\”2s\”/>Finally, your ring finger will be on string 5 pressing down on fret 3.<break time=\”2s\”/>Now go ahead and play the chord C Major.<break time=\”1s\”/><audio src=\”https://s3.amazonaws.com/.../Chordcmajor.mp3\" />One more time.<break time=\”1s\”/><audio src=\”https://s3.amazonaws.com/.../Chordcmajor.mp3\" /><break time=\”2s\”/>If you’re ready for another chord to learn, please ask for it now.For example, say Teach me how to play A Major.</speak>”}}

Using visuals to complement the audio experience

A well written skill shares visuals by pushing cards to the companion app on the users phone or tablet. These visuals reinforce the audio delivered, and complement the content.

Below is an example of a card that displays how to position each finger on the guitar when playing a chord. The audio provides explicit details using voice instructions to the user. The visual reinforces the message using another medium. The diagram included is a standard notation used by guitar players.

There are also cards showing the location on the fretboard of different notes. Once again, this reinforces what is being described through the audio of the skill.

These cards contain similar content to graphics embedded in YouTube videos.

One advantage is that these cards display salient information without using your hands. Rewinding the skill with to play back a previous session also replays the cards. Your voice completely drives the narration. No buttons, hand gestures, or mouse clicks hit the back button or similar actions. Fingers focus on plucking and pressing down strings to create music.

What feature is Alexa missing?

It’s still early in the adoption of voice platforms. I’d expect to see more applications like mine that take advantage of the voice interface. Gaps exist where apps exclusive to a mobile device are superior.

One feature for Alexa is enabling the raw sound data from the microphones to the application. Today Alexa translates every sound directly to text. This limits the ability for the platform to listen to tones when I play the instrument. Helpful mobile apps validate the sounds played, and let the user know the correct tones.

Feedback Welcome

If you’re an aspiring musician like me and have an Alexa device, please give the skill a try. Like all Alexa skills, it’s free to enable. It would be great to have some feedback on what’s useful and other features to add. I’m still learning the instrument, so I also welcome any advice on what’s missing from the skill.