We talk about:

• How she grew up a first generation American and public school kid in Milwaukee

• How her love of robotics helped her get into Stanford

• How freeCodeCamp served as a key resource to build her developer chops

• The near future of humanoid robots, self-driving cars, and human-like AI agents in games

Support for this podcast comes from a grant from Wix Studio. Wix Studio provides developers tools to rapidly build websites with everything out-of-the-box, then extend, replace, and break boundaries with code. Learn more at https://wixstudio.com.

Support also comes from the 11,043 kind folks who support freeCodeCamp through a monthly donation. Join these kind folks and help our mission by going to https://www.freecodecamp.org/donate

You can watch the interview on YouTube.

Or you can listen to the podcast in Apple Podcasts, Spotify, or your favorite podcast app. Be sure to follow the freeCodeCamp Podcast there so you'll get new episodes each Friday.

Links we talk about during our conversation:

• Peggy's GameDev company, Ego AI: https://www.egoai.com/

• Quincy's interview with hardware engineer Bruno Haid that he mentions toward the end of this episode: https://www.freecodecamp.org/news/podcast-hardware-engineering-bruno-haid/

In this article, I'll discuss how we reached young audiences by combining robotics with e-sports.

What on Earth?

Ever heard of anything like it before? Me neither. The robot was created as part of Futurice’s project with Yle, the national broadcast company of Finland.

Yle produces content for TV, radio, and the web. It has a broad reach of older audiences, but has had trouble reaching younger ones. The goal of this project was to use new technology to reach young audiences — specifically teenagers.

Yle’s content has traditionally been non-participatory: the performers perform, and the audience watches. However, younger audiences generally view content that is more participatory, such as YouTube videos or streams.

We wanted to create participatory content — the performer should interact with their audience. Yle’s journalists specialized in teenage audiences pointed out that gaming and e-sports are popular content. We realized that gaming was the perfect context for this: the audience could play together with the performer. We wanted to explore what an entertainer, or even influencer of the future could look like. So why not create a streaming robot gamer?

Yle’s journalists and Futurice’s roboticists coming up with the idea of a robot gamer

First drafts

What Was This Robot Like?

We chose two games for the robot to play: Flappy Bird and Minecraft.

Flappy Bird was a cult game that had a brief period of extreme popularity in 2014. We chose Flappy Bird because the mechanisms of the game are simple, and allowed for the game to be played with machine learning. We wanted to try a neuroevolution algorithm, which evolved new birds into the game based on which birds did best in the previous generation. This way we could see how the audience reacted when a computer was playing the game.

We chose Minecraft for its communal features, which allow interaction between players. Players can co-operate or fight each other, trade with each other, and chat with each other. They can “grief” each other — i.e. be a nuisance. Players can also mine materials, and turn them into items, or even build cities. They can store precious things, farm the land, herd animals, and fight monsters.

Minecraft also has a material called redstone, which players can use to build logic. Effectively, players can build an entire computer inside of Minecraft. Poetic, eh?

For playing Minecraft, we decided that a human should control the robot. The game is complicated, and having authentic interactions with other players would require a human on the other end.

Flappy Bird and the robot

Our team of Yle’s journalists and Futurice roboticists defined some clear advantages for using a robot in this context:

• A robot is tireless — it can play forever, and provide content 24/7. It’s an ideal streamer.
• A robot can be gender neutral. Gamers and gaming audiences are typically male, but a gender neutral robot could potentially attract more diverse audiences.
• A robot can reflect gamers’ behaviour, stirring emotions. Gaming culture is often aggressive. The robot could be aggressive in turn, making gamers reflect on their own behaviour.
• A robot can interact within the game and chat simultaneously. Humans are limited to one output, a robot can have several.

Testing our streaming equipment and having problems with interference from the robot’s face screen.

We decided on a six hour gaming session, alternating between Minecraft and Flappy Bird. To nail down the user experience for the session, we defined guidelines for our design of the robot:

• Experimental user experience — the user should be able to explore within the interaction with the robot.
• Streamers are usually strong characters. The robot is a character with a strong personality.
• The robot can cause “WTF?” reactions in players. We wanted the experience to be memorable, rather than bland.
• The culture and space of online gaming is unique, and should be designed for.

Based on these guidelines, we created the character IQ_201. IQ_201 is based on aggressive online gamers that are convinced of their own superior intelligence (see this meme about having an IQ over 200). The robot would be rude and reactive, meant to get a reaction out of adolescents interacting with it.

Before implementation, the team also wanted to take into account some ethical considerations:

• Transparency about how the robot is operated is needed. If this robot was going to be put into production, users should be able to find information on how it works.
• The robot should treat all the players the same. This was also part of the decision to make the robot appear gender neutral. And due to the sometimes angry, hateful, even racist or sexist culture of gaming, we needed to design the robot’s personality carefully. It could be out-there, even rude, but never hateful. We didn’t want any heated gaming moments.
• The robot should be rude, but not too over-the-top.
• The chat needed to be moderated. As stated, gaming culture can be toxic. We wanted to keep a careful eye on both the Minecraft and Twitch chats, to ensure no shenanigans would go on.

To fulfill all these requirements, we selected the Furhat robot. The Furhat robot had a relatively easy-to-use teleoperation interface, which allowed for the user to input text to be turned into the robot’s speech, as well as perform gestures at the click of a button.

Flowers and Violence

We had a 6 hour stream, starting at 2pm and ending at 8pm. We alternated between games: 2-3pm was Flappy Bird, 3-5pm Minecraft, 5-6pm Flappy Bird, then 6-8pm Minecraft again. This schedule allowed the robot operators playing Minecraft to have a much-needed break in the middle.

Our robot

In the beginning, a few people joined. Gradually, we gained more and more people. We reached our peak at 4:20 — 49 simultaneous viewers on Twitch. Overall, we had 431 unique viewers. In Minecraft, there were around 30 active players. Considering how minimal our advertising was (one forum post and a few tweets), we were pleasantly surprised by the turnout.

The Minecraft sessions were directed by two robot operators (myself and another Minja). The other Minja played Minecraft, and I operated the robot’s voice and gestures. A third person was answering messages on chat.

Minecraft was overwhelming. The robot’s provocative character provoked teenagers into repeatedly killing it. After fleeing to the mountains to be with the llamas a couple of times and being killed yet again, we modified the robot’s behaviour to be more friendly. We wanted to create more constructive interactions.

Two Minjas as robot operators

Toward the end of the second Minecraft stream, teenagers were cooperating with the robot. They protected it from the few aggressive players that were left, and gave it gifts such as flowers. Some even complimented the robot directly, to make it happy. The players started following the robot, agreeing to cooperate when it initiated building a lighthouse. They also built a house, and captured and named a llama: IQ_201 Junior.

There were two clear factions in the game: some were intent to kill the robot throughout the entire game, and some protected it throughout. Some became more comfortable with the robot as the stream went on, switching sides. Either way, the robot aroused strong emotions. Teenagers sought out genuine interaction with it. No-one ignored the robot, or was bored.

Discussions about how the robot worked were had throughout the stream. No-one asked the robot itself though, maybe out of respect or fear of annoying it. Discussions focused on whether the robot was “real”, i.e. whether it was truly automatic, or a human was operating it. Was it typing with actual physical hands? Or had it “hacked into” the game and was playing via code?

“I Will Miss You Robot!”

16 people responded to our survey afterward. 80% of players were under 18, most were 13 to 15-year-olds. 80% of players interacted with the robot. This was extremely positive, we succeeded in making a robot that was compelling to users. 75% of players rated the robot as 3 or above, out of 5.

Viewers over time

We collected comments from the players from the survey, as well as the Minecraft game chat. They are translated from Finnish, and reflect some of the thoughts our players had on the robot.

The critical first:

“It was pretty fun and cool. But it somehow felt like a set-up.” [Referring to the robot possibly having a human operator]

A lot of players were wondering how the robot actually worked. This comment brings us back to the ethical consideration we talked about earlier: transparency about how the robot was operated.

Though we intended to be transparent at first, we decided to not inform users of the robot’s teleoperated nature for this first pilot. We made this choice because we wanted to keep the “suspension of disbelief” of the user active, meaning that we wanted the participants to play along with the fact they were talking to a “real robot” (an autonomous robot) (Duffy & Zawieska, 2012).

The negative response we received regarding the unclarity on the robot’s functioning made it clear to us that if this pilot were extended, it is highly important to be more transparent. It is possible to hold suspension of disbelief and be honest of the robot’s operation simultaneously (we do all know that television shows are not reality, after all).

“The robot was a bit simple in certain things, and sometimes talked meanly to people, and was condescending. This was a bit anxiety inducing… Was this intentional?”

Some players felt that the robot’s rude behaviour crossed the boundary into condescending. They wished that the robot would be more considerate in the future. This indicates that even a robot can hurt feelings. In future versions, making IQ_201 more empathic, and less focused on the superiority of robots over humans, could have positive results.

“The robot looked a bit blue in the face, and its voice was a bit weird.”

Two teenagers did not enjoy the robot’s appearance and voice. One remarked especially about its blue face, asking why we did not make it “normal colored”.

This may have been due to the robot falling into the “uncanny valley” for these players. Uncanny valley is a theory developed by robotics researcher Masahiro Mori (Mori et al., 2012). His theory posits that as the appearance of a robot approaches human-likeness, there is a dip when the appearance gets very close. Zombies and corpses fall into this valley.

To get rid of this effect with our robot, it could be wise to alter the robot’s appearance and voice in future solutions.

The Uncanny Valley by Masahiro Mori. Wikimedia commons.

And then the positive:

“It was really interesting gaming with the robot. :) Hopefully in the future we can have this type of event again. :)”

The majority of the teenagers enjoyed playing with the robot, and watching the stream. Their feedback complimented the robot’s sense of humour, and its playing skills. A continuation of the pilot would surely find an interested audience.

[to the robot] “Some people have trouble with new things. In this case, these players are having trouble with a robot, since you’re new.”

This player consoled the robot in-game, as other players were giving it a hard time by continuously killing it. This comment was moving: the player felt bad for the robot, and thought the robot might feel bad as well, attempting to change that. This is a clear empathic response toward the robot.

[to the robot] “I will miss you robot!”

A few players gave the robot heartfelt goodbyes when it left Minecraft. These players found the robot approachable, and even formed an emotional bond with it. This means we succeeded in creating a compelling character, even during what was only a 6 hour stream.

For future builds of the robot, the players should be informed how the robot operates. This could help them calibrate an appropriate level of emotional bond to the robot.

Are Robotic Influencers Our Future?

Players took an active interest in the robot. They approached it, interacted with it, and formed opinions on it. The robot also provoked emotional reactions — both positive and negative. Some participants really loved the robot and wished for more interaction in the future, and some were heavily critical.

This indicates that robot influencers have the capacity to affect our emotions — whether this capacity will ever reach the level of human entertainers, I do not know. Whether that is desirable, I also do not know.

What positively surprised me was that the young user base of the robot were media literate: they critically examined the robot’s mode of operation. The players had a good idea of what is possible with AI today, and what is not. They were not easily duped.

This makes me hopeful for our future, whether it contains robotic entertainers or not. When viewers remain critical, they can understand that robots are machines with no actual emotional capacity, even if viewers do choose to suspend disbelief.

Interacting with the robot can be though of as a form of parasocial interaction for the viewer — where the viewer may feel that their relationship with the robot is close — even though it is not truly reciprocal. This in itself is not necessarily harmful, as long as we are honest about what the relationship truly is. We should understand that robots are putting on a performance to elicit emotions in us — like human entertainers do

A video about the project.

First published on Towards Data Science.

Duffy, B. R., & Zawieska, K. (2012, September). Suspension of disbelief in social robotics. In 2012 IEEE RO-MAN: The 21st IEEE International Symposium on Robot and Human Interactive Communication (pp. 484–489). IEEE.

Mori, M., MacDorman, K. F., & Kageki, N. (2012). The uncanny valley: The original essay by Masahiro Mori. IEEE Spectrum, 98–100.

In Spring 2018, I conducted a pilot study of a robot teaching sign language to children with autism. This blog post reflects on the results of the study, and our team’s process of designing the robot.

Robotics, Sign Language, and Children with Autism

To start with, let’s answer the first big question: Why use a robot in autism therapy? People with autism have an attention preference to objects over people, and children with autism have shown more interest toward therapy when it involves technological or robotic components. Additionally, a robot’s operation can be strictly controlled, which can make therapy less overwhelming for autistic people.

Then, the second big question: Why teach sign language to children with autism? People with Autism Spectrum Disorder (ASD) experience problems with communication: 40–50% of people with ASD are functionally mute in adulthood. To mitigate this, they use Augmentative and Alternative Communication (AAC) methods. Assistive sign language — a simplified form of sign language — is the most common form of AAC. Other common AAC forms are symbolic pictures and photographs.

An example of a symbolic picture communication system used by children with autism: “We do assignments.”

The idea to combine these two domains — robotics for children with ASD, and sign language for children with ASD — originally came from Satakunta health care district. Satakunta is a small region in South-Western Finland, with a population of 225 000 people. The quality manager at the district had read about a robot being used to teach sign language to neurotypical children, and wanted to investigate the same method with autistic children.

Satakunta found out that the company I work for (Futurice) had built a humanoid robot. They reached out, and we assembled a cross-disciplinary team that would re-design and modify Futurice’s humanoid robot to fit this purpose. Our team had three roboticists from Futurice, and three experts of autism treatment from Satakunta: a neuropsychologist, a speech therapist, and the quality manager.

How Should We Design the Robot?

The humanoid robot Futurice had built was an InMoov robot. The InMoov is designed by French sculptor Gaël Langevin. He has made the schematics and software of the robot open source and available to anyone online.

Using these, Futurice built its own InMoov. Satakunta wanted to use the InMoov due to its nimble hands that enable it to sign. Its human-like appearance was also an advantage: one of Satakunta’s neuropsychologists thought that a human-resembling robot would be best for this solution.

The original form of the open source humanoid robot InMoov, designed by Gaël Langevin. Image Wikimedia Commons.

However, the InMoov was not ready as it was. To make the robot be suitable for children with autism, we needed to modify its form, behaviour, interactions and environment.

My job was to design these four dimensions of the robot. I also needed to design the human-robot interaction study where children would meet and sign with the robot. Luckily, the open source nature of the robot’s software and hardware would make it relatively easy to make the necessary modifications.

We wanted to take the characteristics of ASD into account while designing the robot. ASD is characterized by problems with communication and language, problems with social behaviour, inflexibility of routines, and problems forming a holistic perception of surroundings. However, autism is a spectrum, so these characteristics present differently in different people. Due to different presentations, we knew we couldn’t design a robot that would benefit everyone. Nevertheless, we wanted to find the best solution that would serve the most children with ASD.

During the project, the team created five design guidelines for the robot, which would tailor its design for autistic children. For example, the concern that a child might get distracted during the experiment was brought up by the autism specialists. The team agreed that to avoid confusing the child, the robot’s behaviour should be consistent and structured. This was defined as the guideline: “consistent, structured, simple behaviour”. To follow the guideline, the speech therapist and I created strict script of the robot’s and child’s interaction. All five guidelines were formulated in a similar manner in co-design discussions.

Design guidelines for a robot to be used with autistic children:

1. Simple form
2. Consistent, structured, simple behaviour
3. Positive, supportive, rewarding experience and environment
4. Modular complexity
5. Modularity specific to child’s preferences

The design guidelines helped the team maintain a logical and uniform design for the robot, and formed a baseline for all decisions made during the design process.

Embedded Ethics

While the team was discussing the design of the robot and the experiments, several ethical considerations were raised. These ethical considerations were embedded into the finalized robot.

Physical safety — Users can be potentially pinched or crushed by a robot. To mitigate this concern, we decided to stop the children from touching the robot during interactions. To accomplish this, we decided to have the speech therapist in the room with the child to stop them from approaching the robot, if need be.

Data security — It is paramount that the user’s data stay secure, especially in health care applications. Here, we decided to keep all data of the children attending the study encrypted. We also did not record any unnecessary data.

Appropriate behaviour enforcement — People can learn bad manners from robots. For example, children have forgotten to use polite language such as “please” and “thank you” after interacting with the voice agent Alexa. Alexa does not explicitly ask for polite language, causing people to behave badly toward it. In this case, we didn’t want children to learn to treat the robot badly, and potentially generalize that bad behaviour to people. We decided that the therapist would intervene in all bad behaviour toward the robot.

Equality across users — Artificial intelligence algorithms have been shown to be racist or sexist in the past (e.g. COMPAS, a recidivism rate prediction algorithm used in USA, was biased toward African-Americans). If a robot uses algorithms to function, designers of the algorithm need to be careful. Another thing to consider is the robot’s form. Robot designs have been shown to reinforce gender stereotypes, with “genius” robots often labeled as male, and “service” robots as female. In our case, we wanted to design a robot that was gender neutral, so that each child participating in the study could feel welcome. To do this, the robot was given no gender signifiers.

Transparency — If the user understands how the robot operates and makes decisions, they can calibrate their trust level in it. In our case, we decided to inform the children and their companions of the robot’s teleoperated nature at the end of the study. This way they could avoid forming false assumptions about the state of robotics today, and how it can be applied to autism therapy.

Emotional consideration — Research has shown that humans treat robots as is they were alive, even when they clearly aren’t. People form emotional bonds with robots. This should be taken into account in the design: how strong a bond is desirable for this use case? In this case, we didn’t want the child or their companions to think that the robot was replacing the bond between the child and the therapist. To ensure this, the speech therapist would stay in the room with the child the entire time.

A Balancing Act

Building complex technology (robotics) for a complex domain (autism therapy) is difficult. The problem space was unintuitive to both sides of the team: the autism specialists weren’t familiar with technical limitations, and the roboticists had no knowledge of the user group.

Designing was a balancing act: a seemingly small tweak in the design could result in a high amount of technical work. And vice versa, some changes that were significant design-wise could be easy to implement technically.

This is where I spent most of my time — harmonizing the team’s different viewpoints into a good design that would be realistic to execute in our time frame of 6 months. Together we made a series of decisions balancing user experience and technical complexity.

The final InMoov with modifications: new Ada hands, lights attached to its right hand, and a screen on its chest.

When designing a social robot, there are a lot of moving parts (both literally and figuratively). The robot’s features all affect each other. The robot’s operation context affects how it behaves, which affects how it interacts with the user, which affects what its form is like. Prying these design considerations apart and crystallizing them into distinct and implementable technical tasks was challenging.

Modifying the robot from its original form took 4.5 months (the original build had been assembled in 9 months). All our modifications followed the design guidelines: for example, we changed the robot’s human-like voice to robotic, to give it a “simple form” (guideline 1).

To make the InMoov a better sign language teacher, we made a few big adjustments. We gave it new Ada hands, designed by Open Bionics, and built by Metropolia University of Applied Sciences. We also embedded a screen into its chest, and lights onto its arms. The screen was added to provide another mode of communication (photographs are often used in AAC), and lights were added to capture the child’s attention.

10 Children and a Robot

10 children took part in the experiments. Some came with their parents, some came with other support people. Two roboticists (me included) were in the room adjacent to the experiment room, operating the robot and observing the experiment through a camera feed. A third roboticist was present to solve any problems that might emerge. The speech therapist was in the experiment room with each child, facilitating the interaction between the child and the robot, and intervening when needed.

The 9 words the robot signed, and the corresponding images on its screen.

The robot performed 9 signs. With ⅓ of the signs, it also flashed the lights on its arm. With another ⅓ of the signs, it displayed an image corresponding to the sign on its screen. These 3 different design conditions were examined to see which was most effective.

I was surprised by how differently each child interacted with the robot. A few of the children signed with near-perfect accuracy throughout the entire experiment, imitating all the robot’s signs in a mere 6 minutes. Some took as long as 28 minutes, struggling with each sign. One particular child — who was not too keen on signing — could not stop laughing at the robot. The child kept attempting to either hug or lunge at the robot throughout the experiment, with the speech therapist and neuropsychologist lunging after him to stop him in time.

A child signs with the robot (picture with permission).

The speech therapist signals to the robot operators that the child signed correctly (picture with permission).

Children Imitated and Paid Attention to the Robot

I measured the children’s signing accuracy and attention focus. Children and their companions also completed surveys where they gave their opinions on the robot.

What we learned:

1. Children could imitate the robot, and paid attention to it

7 out of 10 children successfully imitated the robot at least once. 70% of the time, they kept their gaze on the robot, indicating attention focus. 8 out of 8 companions who filled the survey also noted that the children had a connection with the robot.

2. The image on the robot’s screen was good

The robot displayed an image on its screen simultaneously to signing 1/3 of the time. Children’s companions found the images useful, and thought they could help the children learn.

3. The robot was seen as good, but a bit scary

Both children and their companions rated the children’s experiences with the robot as good. However, some children and their companions noted that the children felt the robot was scary. Factors that create scariness should be identified and reduced in future designs.

4. Performance of the signs needs to be better

Both the speech therapist and children’s companions noted that the robot did not sign all words well, which might affect the children’s understanding of them.

Things still to learn:

1. Are children understanding the signs?

For this experiment, I only measured whether children imitated the signs. Future experiments are needed to verify if children are understanding them.

2. Who best benefits from the robot?

Not all children responded similarly to the robot. It is unlikely that this type of robot-based sign language therapy is useful to all children with autism. Future experiments should examine who this therapy is suitable for. This varying benefit is supported by companions’ survey results: 6 out of 8 companions thought the robot could potentially be beneficial as a sign language tutor.

3. How to input speech therapist’s commands?

To use the robot in the future, the therapist needs to be able to independently control the robot. For future implementations, a remote control or UI for programming the robot’s behaviour and interactions may be useful.

4. How will guidelines 4 and 5 affect the design?

For this experiment, I used a static design of the robot. Only its interactions changed (using its screen and lights at different points). Future research is needed to examine the design guidelines “modularity of complexity” (guideline 4) and “modularity specific to children’s preferences” (guideline 5). These could help adapt the robot to different users.

The Future of Robotic Sign Language Tutors

People interacting closely with robots can induce criticism. The most prominent concern is that of robots replacing humans. In this case, the robot is not intended to replace human care — rather augment and support human care.

Therapy sessions are demanding for therapists. They need to plan and conduct the session, while dealing with a potentially uncooperative participant. In the future, technological tools could be used to reduce the cognitive load of the therapist. The robot perform the signs, while the therapist could focus on encouraging, tutoring and managing the child. With reduced cognitive load, sessions could be longer and thus more in-depth.

The same effect could potentially be achieved if the robot were situated in the child’s home: the child would get a consistent tool to practice signs with, and the robot would not be bored or frustrated by repetitive practice.

To my knowledge, this was the first instance of using a robot to teach sign language to children with autism. I hope this research is continued further. I hope our pilot can shed some light onto how to develop this application in the future.

Science — especially pilot studies — sometimes requires improvisation: lights from the robot’s back were reflected on a window behind it, which might have distracted the children. We fashioned a light-reflection-blocker for the robot’s back, out of a towel stolen from our hotel (we did pay for it).

The entire study is available as:

Axelsson, M., Racca, M., Weir, D., Kyrki, V. (2019). A Participatory Design Process of a Robotic Tutor of Assistive Sign Language for Children with Autism. In 2019 28th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN). IEEE. Accepted.

The study is based on my master’s thesis at Aalto University, available here.

The project was funded by Prizztech’s Robocoast and ERDF-fund, and Futurice.

As the Chief Technical Architect of the Shadow Robot Company, I spend a lot of time thinking about grasping things with our robots. This story is a quick delve into the world of grasp robustness prediction using machine learning.

First of all, why focus on this? There are currently much more exciting projects using deep learning for robotics. For example, the work done by Ken Goldberg and his team at UC Berkeley on DexNet is very impressive. They manage to reliably grasp 99% of their test set using deep learning. But when we work on delivering a “Hand that Grasps” as a product, we have to focus on delivering smaller robust iterations first. Being able to predict whether a grasp is stable or not, dynamically, is an interesting topic for the industry. For example, if you can determine that a grasp has a high chance of failing before it actually does, you can save a lot of time by re-grasping.

The goal isn’t to give you the best solution for grasp quality using machine learning, but rather to give a gentle introduction to using machine learning in robotics for a directly useful purpose which is hard to solve with existing standard algorithms.

If you want to skip the explanations and get your hands dirty with the open source dataset and code, head on over to Kaggle.

Gathering the dataset

Using the Smart Grasping Sandbox, we gathered a large dataset for our purpose. Since our goal is to classify whether a grasp is stable or not, we need to gather a dataset containing both stable and unstable grasps. We also needed to quantify the stability of a grasp automatically in order to annotate our data easily — instead of annotating it manually.

Which data should we record? We get plenty of data from the simulation. To simplify things, we’ll be looking at the state of the joints only. This state contains — for each joint — the position, the torque and the velocity. Since we want a grasp quality that’s object-agnostic, we won’t be using the joint position: the shape of the hand is purely object specific. So we’ll be recording each joint velocity and torque.

Our dataset will look like this:

Dataset overview

An objective grasp stability measurement

In simulation, there’s an easy way to check whether a grasp is stable or not. Once the object is grasped — if the grasp is stable — then the object shouldn’t move in the hand. This means that the distance between the object and the palm shouldn’t change when shaking the object. Lucky for us, this measure is very easy to get in simulation!

1. first grasp 2. then shake, whilst 3. measuring the distance between the palm and the object

Let’s record some data

Now that we know what we’re doing, we’ll be using the Sandbox to record a large dataset. You can have a look at the code I use to do this over here. Since the Sandbox is running on Docker, it’s very easy to spawn multiple instances in parallel on a server and let them run in parallel for a while.

Since I don’t trust simulations to run for too long — call it a strong belief based on personal experience, same as the demo effect — I also only run 100 grasp iterations before restarting the Docker container with a pristine environment.

In order to get a relevant dataset, I randomise the grasp pose around a good grasp — which I found empirically: we want enough bad grasps in there. I’m also using different approach distances. This gives me — roughly — a 50/50 ratio of stable and unstable grasps — with plenty of grasps in between.

For your convenience, I’ve made this dataset public and you can find it on Kaggle.

Let’s learn!

Now that we have gathered a good enough learning set, we want to teach a Neural Net in order to predict whether a grasp is stable or not, based on the current joint state.

What’s a Neural Net?

With all the current hype around Deep Learning, it’s easy to imagine a computer with a brain learning new things auto-magically. Let’s demystify the Neural Net quickly.

As shown above, a Neural Net takes as an input a vector — in our case the torque and velocity for each fingers. Then this vector is transformed a few times — as many times as there are layers — and a final vector is the output of the Neural Net: the classification of the data that was given. In our case the output we want is whether the grasp is robust or not — so it’s a vector of size 1.

During the learning process, we will feed the network values we’ve gathered in the dataset. Since we know whether those joint values are for stable or unstable grasps — our dataset is annotated — the training process adjusts the parameters of the different transitions between the layers.

The art of machine learning consists of choosing the network topography — how many layers and neurones, plus which transition functions to use in our network — as well as gathering a good dataset. If we have all this, then we can train a network that will generalise well to cases that haven’t been seen during the training.

Training the network

The goal of this exercise isn’t to create the perfect grasp quality prediction algorithm, but rather to simply show how it’s possible to use the smart grasping sandbox for machine learning. I chose a very simple topology for the network: I’m using one single hidden layer between the input and output layer. For more details, refer to the iPython notebook.

For simplicity’s sake, I’m using the excellent Keras library. If you can’t wait and want to see the actual code, go to Kaggle. Otherwise, read on!

After loading the dataset in memory, I split it between the training set and the test set. Validation will be run on part of the training set, and I’ll use the test set to see whether it generalizes well.

Since my network is small, training is relatively fast, even on my laptop. When I train deeper networks, I spawn the docker image on a beefy machine in the cloud, using NVidia’s GPUs to speed up training.

After training my network, I get an accuracy of 78.87%.

Testing my trained network on live data

Now that we trained our Neural Net we can use it to predict the grasp quality in real time on the simulation. As you can see in the video below the prediction is working nicely most of the time.

As you can see in this video, the live prediction of the grasp — the blue plot on the left — is higher than 0.5 when grasping the ball the first time. This results in a very stable grasp. On the contrary, during the second grasp, the metrics stays under 0.2, rightfully predicting that the grasp will fail.

Final words

I hope this story piqued your interest. If you want to try to train your own algorithm on this dataset, the easiest thing to do is to go to Kaggle.com where it’s already set up for you.

Obviously, there’s much more that should be done to deploy this method in production. The first thing to tackle is to have a better simulation in order to pick up a wide variety of objects. I’ll also be looking at having a live objective grasp quality — the one that’s used to annotate our dataset — in order to be able to use time-series prediction instead of one shot prediction. And the final challenge will be transferring that learning to the real robot.

There are so many interesting topics to explore machine learning with in robotics: grasp quality, slippage detection, amongst others.

I hope that you can’t wait to test your ideas on the sandbox. If you do, let me know on Twitter!

Getting started with robotics is probably a lot easier than you think. Here’s a simulation sandbox that’s cross-platform and provides a simple high level API. It should help you get started experimenting with robot grasping tasks.

As the Chief Technical Architect at the Shadow Robot Company, I spend a lot of time playing with different algorithms to see how they’d fit our robots. Controlling a complex robot to make it behave the way you’d want in a complex environment is… complex!

An important part of our roadmap relies on machine learning. I don’t want to have to specify every aspect of a problem — I’d rather the system learn the best way to approach a given problem itself.

Setting up the environment to easily try different machine learning algorithms — for example for refining grasps — is not trivial. Here were my requirements:

• a good simulation scene to get started: a robot, a 3d sensor, some objects to interact with, some furniture to plan around and use
• a variety of tools and libraries to get started quickly with playing with the robot: robotic framework (ROS), simulator (Gazebo), planning libraries (MoveIt!)
• a way to run it headless while being able to visualize the data
• easily deployable, well documented… all the usuals!

If you don’t want to read through the specifics but just want to get your hands on the sandbox you can head over to the github repository which contains the instruction for a quick start. For using the sandbox in the cloud you can also visit the ROS Development Studio where it’s deployed.

The simulation

The Smart Grasping Sandbox will be running on the Robotic Operating System — ROS. As a long time ROS user and contributor I’m very partial to using that framework. It’s the de facto open source framework designed for robotics. It takes the hassle of connecting the different components of a robotic system with a modular approach. This makes it possible to swap a given component easily. On top of that it’s driven by a thriving community, so you can always find the latest algorithm or driver.

In order to teach a robot, it’s a good idea to start with a simulator. Running the algorithms on the actual hardware is not only more expensive but often less convenient: it’s harder to reset the environment in real life than in simulation. It’s also often harder to characterize the impact of the robot on the scene: did the ball get picked up? Was the grasp stable? All of this information is available out of the box in a simulator.

The simulator we’ll use is Gazebo; this is a physics simulator for robotics and is also tightly integrated with the ROS framework. Plenty of robot models are available in the simulator, from arms and grippers to quadcopters! In the Smart Grasping Sandbox, the robot I’m providing is a UR10 from Universal Robot with Shadow’s Smart Grasping System. The scene only contains two useful objects for now: a cricket ball and a drill. This is just a starting point. The scene will evolve with time. The 3d Vision sensor in the simulation is a Microsoft Kinect as it’s often used in robotics (yes, the very same Kinect you used to play Just Dance on your Xbox).

Docker container

Setting up the different frameworks and libraries is not trivial and takes time. To simplify the deployment, I’m automatically building a Docker image. If you’re not familiar with Docker, I won’t go into the specifics as it’s out of scope for this article, but let’s say it’s a super lightweight Virtual Machine like environment: you can spawn images very quickly while exploiting your computer’s full potential.

Deploying the image with Docker also makes it OS agnostic — ROS and Gazebo are more Linux friendly. It’s also a great way to test things on your laptop, then once you’re ready to start a longer experiment, simply spawn in in the cloud. Since I included a web interface to the simulator, you can even visualize what’s happening in simulation by connecting through your browser. To ease the development process, I included a Jupyter notebook which you can access through your browser.

Libraries and tools

In order to speed up your development process, I’ve developed a simple high level library — boldly called the SmartGrasper. This library makes it possible to interact directly with the simulated sandbox sending commands such as pick the ball, open the hand, move above the ball… For the path planning, it relies on ROS’ planning library: MoveIt!, so that you can safely move the robot from A to B without hitting things.

iPython notebook

The sandbox comes with an example iPython notebook that shows how to pick up the ball using the SmartGrasper library. You can use this example as a base to do your own development.

Final words

So I spent some time preparing that sandbox and now that it’s ready, I’m sharing it with you! Head over to the shadow-robot/smart-grasping-sandbox github repository to get started. Feel free to play with it, submit issues, pull requests...

There’s still a lot more to be done: adding OpenRave for grasp planning, adding more complexity to the scene to be able to learn different actions, adding some vision algorithms for recognizing the different objects… But this is just the first release of the Smart Grasping Sandbox!

The Smart Grasping Sandbox in the Construct

I’m also working with the amazing team at The Construct to make the Smart Grasping Sandbox available on the ROS Development Studio for an even quicker way to test your ideas.

If you’re doing something cool with the Smart Grasping Sandbox, or have any questions, let’s connect on Twitter! If you enjoyed this article, how about liking and sharing it?

Image credit: http://www.esa.int/images/DSC03062.JPG

What it’s like to be a robot in 2017

What will a state-of-the-art robot be able to do in 2017?

There are many different types of robots out there, from humanoid robots to industrial arms that can move with an amazing accuracy and speed.

Given my area of expertise, I’ll focus more on grasping and using objects. These are core human skills that robots need to acquire if you want them to be truly useful.

But why’s it so hard for a robot to replicate these skills?

Here are the capabilities robots need:

For a grasping / manipulating robot

Hardware-wise, a grasping or manipulating robot needs an arm, a hand, and a 3d sensor. The arm usually has about 6 degrees of freedom (for reference, the human arm has 7). This makes it possible to get to any given point in a workspace.

On top of a good position control — being able to drive each joint towards its given target quickly and reliably — the arm and hand also need a reliable torque control. This means that you’re able to apply a given torque with a given joint.

An advanced hand will also have tactile sensing, which makes it possible to manipulate the objects.

A 3d scene acquired from a Kinect.

For a modern robot, understanding its environment is essential. You can’t grasp or use an object if you can’t see it.

The pipeline that’s used the most often for this task is the following:

• First, the vision pipeline will run a segmentation algorithm to isolate the different objects in the incoming scene, for example, a 3d point cloud.

• Then it will run through some recognition steps. The goal is to identify the objects if possible, and align some known mesh of the object.

The diagrams below are quite simple compared to the real use-case above. Recent advances in deep learning are showing great promises in this domain.

Full scene / segmented results: each object is a different color / each object is recognised, the model is aligned in blue.

Now that the robot has a rough understanding of where things are in the scene, it needs to be able to navigate the environment, avoiding the obstacles. This is the field of motion planning. There are plenty of different algorithms that deal with motion planning.

Now that the robot has a way of reaching the object you want it to grasp, you need to know how to close the hand around the object. There are different ways to address this issue, but two main approaches used most often are grasp planning and teaching by demonstration.

In the grasp planning approach, the algorithm uses some heuristics to compute different grasps and evaluate the grasps using a grasp quality measurement. In the teaching by demonstration approach, a human shows the robot how to do the action. The algorithm then takes care of extracting the information to make the action work reliably on the robot.

Teaching by demonstration — opening a bottle at Bielefled university.

Finally, it’s possible to close the loop using the different sensors available in the robot to accomplish an action, such as stabilizing a grasp when detecting slippage, or moving a finger on an object without letting go of it. Running a tight control loop, then using its data to modify a robot’s next command is a one of the most challenging aspect of robotics.

How can we move forward

It will take a lot of work to develop all of these capabilities and get them to work in all environments. Each individual and each robotics department will have its own specific area of expertise. For example, my area of expertise is in grasping and manipulation, while other people focus more on humanoid robots, mobile platforms and vision.

We all face the same challenge, though: solving complex problems with unreliable data.

Advanced robots are still too often reserved for specialists. They’re hard to program, and you need a lot of knowledge to get them to accomplish new things. But often, what you want your robot to do can be described easily. This is a task that you could do yourself.

To solve this issue, more and more companies are turning to visual programming interfaces in robotics: NAO’s Choregraphe, Robot Blockly (used at Shadow and Erle Robotics), and Franka’s Desk.

Shadow Robot’s Blockly interface.

But to be able to program advanced robots intuitively through these interfaces, you need advanced — and robust — capabilities.

I’m personally convinced that the way forward is to build more and more intelligence inside the tools themselves. This way, each capability can be implemented by experts.

This black-box approach makes it easier to combine different high-level capabilities, reusing the various state-of-the-art techniques developed by different experts. The boxes don’t need to be black, but the result should be that end users be able to just focus on that box’s functionality — its inputs and its outputs — instead of being distracted by how that functionality should be implemented.

As roboticists, if we want advanced robots to be useful to non-specialists, we need to simplify the interface. But in order do that, we first need to implement more and more advanced capabilities, then wrap them up inside the tools themselves.

If you enjoyed this article, how about liking and sharing it? Also, If you want to discuss robotics, let’s connect on Twitter.

If you’re into some high-tech tinkering, maker culture & creativity — we have something for you on Gitter as well. Check out those communities dedicated to hardware, Arduino, IoT, or even a software for drawing robots!

• Johnny-Five — Johnny-Five is an Open Source, Firmata Protocol based, IoT and Robotics programming framework, developed at Bocoup. Johnny-Five programs can be written for Arduino (all models), Electric Imp, Beagle Bone, Intel Galileo & Edison, Linino One, Pinoccio, pcDuino3, Raspberry Pi, Particle/Spark Core & Photon, Tessel 2, TI Launchpad and more!
• Firmata/arduino — A protocol for communicating with microcontrollers from software on a host computer. The protocol can be implemented in firmware on any microcontroller architecture as well as software on any host computer software package.
• platformio/platformio — An open source ecosystem for IoT development. Cross-platform code builder and library manager. Continuous and IDE integration. Arduino and MBED compatible. Ready for Cloud compiling.
• IRremote Arduino Library — Infrared remote library for Arduino lets you send and receive infrared signals with multiple protocols.
• ESP8266/Arduino— This project brings support for ESP8266 chip to the Arduino environment. It lets you write sketches using familiar Arduino functions and libraries, and run them directly on ESP8266, no external microcontroller required.
• Free Code Camp Hardware — Free Code Camp’s chat about
  chat about computer hardware and Internet of Things.
• Papparazzi — Paparazzi is an attempt to develop a free software Unmanned (Air) Vehicle System. As of today the system is being used successfuly by a number of hobbyists, universities and companies all over the world, on vehicle of various size (100g to 25Kg) and of various nature (fixed wing, rotorcrafts, boats and surface vehicles).
• PX4 Hardware — PX4 Hardware designs. PX4 is an open hardware design, following the OSHW 1.1 definition licensed under the Creative Commons Attribution-ShareAlike 3.0 Unported (CC BY-SA 3.0) license. Website at: http://pixhawk.org
• Node Pixel — Library for using addressable LEDs (such as NeoPixels/WS2812) with Firmata and JohnnyFive.
• Robopaint — Software for drawing robots, and your friendly painting robot kit, the WaterColorBot!

Looking for something else? Check out our Explore section or easily start your own channel here.

Did we miss an channel that you think should be featured? Drop us a line in the Gitter HQ and we will add it to the list.

Happy tinkering!