A-Frame is a web framework for building virtual reality experiences. Ever since its introduction in late 2015, it quickly became a favorite among artists and creators of all backgrounds who want to experiment with WebXR.

I am a creative coder with a background in audio. Diving into the sonic possibilities of this new platform was a very exciting and rewarding journey. Most of my A-Frame experience was confined to standard desktop and smartphone environments and not VR. I am sharing my findings. I want to create a sort of an unofficial manual on how to implement, use and create audio in A-Frame. In this first part, we will take a look at how to use A-Frame’s out of the box audio capabilities.

Prerequisites

This write-up assumes you have some experience with A-Frame. You don’t have to be an expert on it but knowing how the basics work will make following this tutorial easier. If you haven’t had a chance to check it out yet, you can start here.

A-Frame Sound Component

A-Frame is a framework based on Three.js. The sound component it provides is a wrapper around the Three.js positional audio component (or non-positional depending on what we choose, but more on that later), which uses the Web Audio API.

This means we get things like positional audio, volume control and audio playback out of the box, as soon as we use an A-Frame sound component.

_A-Frame sound component properties, screenshot taken from [A-Frame Docs](https://aframe.io/docs/0.8.0/components/sound.html" rel="noopener" target="blank" title=")

But how do we add this component to our scenes and provide audio feedback to user interactions and/or create a soundscape?

To demonstrate this I created an A-Frame project from scratch. It is based on user interactions with a desktop computer. (i.e. to use with a mouse and a keyboard). The principles should transfer to other types of controls for the most part.

You have the option to either start with a fresh copy of the project [Glitch][GitHub] — with no sounds attached to it — and follow along while implementing the provided sounds yourself. Or you can check out the finished version [Glitch][GitHub] and follow along by reading the code.

The Kitchen

Before we start the audio implementation process, let’s take a look at our scene. We have several 3D models here (courtesy of Google Poly). Our kitchen is a complete model by itself. Items like the espresso machine, radio, the frying pan and the egg on top of it are separate models that are added to the scene on top of the kitchen.

Now let’s see the methods we can use to add sounds to this scene.

User Interaction Sounds

User interaction sounds are usually one-shot sounds. This means they are short samples that play once when triggered. They are not continuous or looped like environment audio or music. Which means we need to find a way to put an audio file on our scene and then trigger it when our user interacts with that object. (e.g. clicks the mouse button, hovers over it etc.).

To trigger a sound on user interaction, we can:

1. Put a sound on a model
2. Put a sound on a primitive, like a box geometry
3. Put a primitive on an component

So let’s go through our options one by one and see the use cases for each one.

Putting a Sound on a Model

Who doesn’t want to start the day with a fresh cup of espresso? (If that’s not you, feel free to grab a tea model from here and go with that instead.)

I would say half of the satisfaction of having that caffeine boost is the sound our beloved espresso machine makes. To make sure our scene provides that aural feedback, we implement a sound that will respond to user interaction.

First let’s take a look at how our espresso machine works under the hood

We start by loading our model with the A-Frame loader. Then we set the position, rotation and scale of the model within the scene, by using the named properties.

Our model is referenced by the id “#coffeeMaker”, instead of the relative path of the file. All the models for this project is registered in the Asset Management System beforehand.

Easiest way to add a sound to our espresso machine is by adding the sound component to it by using it as an HTML attribute on the object.

We reference our sound by using the asset management system again, and setting the volume of it to 1. This means it will be heard at 100%. Now for the most important part, we set the “on” property of the sound to the “click” value. This means when a user clicks on this object, the sound will be triggered.

We don’t use the autoplay or loop properties here as this is a one-shot that depends on user interaction. We leave the positional property as “on” by default. Otherwise we would hear the sound at the same volume level at all times, no matter how distant we are from the object. Sound wouldn’t have any panning and would always play exactly where we are, instead of to the left, right or behind us depending on our position relative to the object.

NOTE: Positional sounds like these should be rendered mono to make the spatial cognition of the sound easier, while background music or audio ambiances that are not positional preferably should be rendered stereo.

Time to Crack a Few Eggs

Now it’s your turn, go ahead and implement the provided frying egg sound to the egg model, which is placed on top of the frying pan. For a challenge, you can also try to use a few additional sound properties of your choosing. Go around the scene after putting the egg sound in place and test your implementation to see if it sounds realistic (or unrealistic if that’s what you are going for!)

Did you do it?

Great!

Your code for the egg model now should look approximately like this (with your desired properties and values)

Putting a Sound on a Geometry

So we can add our sounds to the models by simply adding them as an HTML attribute. What if we want to put a sound to an object that is already part of a bigger model and not a separate entity by itself?

Let’s say we want to add a sound to the red bottle on the right, which should make it sound like we are picking it up when we click on it.

If we put the sound on the kitchen model like we did with the smaller models, it would mean that the sound would trigger no matter where we clicked on the model itself, instead of only on the bottle like we want.

We can solve this problem by using a geometry which we can click on, aligning it to where the bottle is, and adding the sound property to it.

A-Frame Inspector to the Rescue

For the next steps I highly recommend that you start creating and positioning objects inside your scene right in the browser by using the A-Frame Inspector. You can then copy and use this code instead of guesswork as to where your object should be positioned.

Open the A-Frame Inspector by using the <ctrl> + <alt> + i shortcut.

Let’s start by creating a geometry primitive on our scene, a cylinder should work well given our bottle’s shape.

Now scale and position this cylinder by using the A-Frame inspector. It can cover our bottle and respond to user interaction on the correct spot.

NOTE: If you add this geometry inside the model, it would enable you to move or rotate the parent model and preserve the correct position of everything inside.

Scale and position the cylinder inside the scene by using the controls top right

You can paste the code you copied from the inspector and just move the scale, position and rotation values inside the .

Copy the entity and its properties by using the button on the top right

Make sure to bring down the opacity of the material to 0, so our bottle can stay visible inside the cylinder geometry. As a final step, add the provided bottle sound to the object, like you did before with the espresso machine and the egg.

Now your cylinder code should look like this…

…and respond to user interaction correctly.

Using an Component

<a-sound> is a primitive wrapper around the A-Frame sound component and allows us to use audio without attaching it to anything else on the scene. However if we want to trigger this sound on interaction, we still need a type of geometry which will allow us to interact with this object.

We can put this geometry on anywhere else on the scene. (think of a user interface menu button which will trigger a non-positional sound). Or we can put it directly on to and then position it on top of the object we want to hear audio feedback from.

Which means to add a sound to our toaster, we can do the reverse of what we just did and create an component and attach a clickable geometry to it.

Environment Audio

We saw a couple of the ways we can use to implement sounds that respond to user interactions. But what about sounds that don’t require the user to do anything to play? These are sounds that go through the scene from beginning to end, to create atmosphere, mood, a sense of place?

If we take a look at our kitchen, we can see that there is a big fridge to the left, which is dead silent right now. Not to mention, whether we are consciously aware of it or not, almost all the environments we occupy (unless they are an anechoic chamber) have something called a presence or a room tone.

Electric hums, machinery, air conditioning units and alike create the room tone with or without us adding on top of it acoustically. So, to reflect that, our choice should be an ongoing. In this case a looping sound which is present throughout our scene.

NOTE: .mp3 format by its nature does not loop seamlessly in most platforms, there’s always a very short gap between loops which breaks the audio consistency, and therefore for looping sounds other formats like .wav or .ogg should be used.

Adding Room Tone

Now let’s add the provided room tone to our scene. We have several options depending on what type of A-Frame project we are working on. If we have only one room-as is the case here-we can put our room tone directly to our . Or if we have several rooms in our scene with different characteristics, we can put the room tone in to a plane geometry which could act as the ground. Or in an object which we can put in the center of the room. In all cases we have to make sure our room tone is not positional, as opposed to all the other sounds we used previously on this project.

Given that we are only working with a single room/environment in here, we can put our room tone directly into the scene itself. We set the positional property to false, to make sure sound is heard equally throughout the room.

We also set the autoplay property to true since we don’t need user interaction to hear the room tone.

NOTE: Audio autoplay in most browsers either already require or will require user interaction to start. Which means to automatically play sounds like background music or ambient audio you need to use a menu screen or a mute/unmute button or something alike to enable audio autoplay.

Music

For the final piece of our audio puzzle, let’s add some music to our scene. We can either choose to add this as diegetic music that’s coming from the radio in our scene, or as extra-diegetic music which is non-existent in the virtual world we created, but specifically for the user/viewer/player who is experiencing and controlling this virtual world.

To do the latter, we would need to put our music either onto our camera () or to our first-person character (if that is an option) and make sure the audio is non-positional. In this scene I chose to go with the former and put the music to the radio model as a source inside the virtual world.

We can add the provided music track (composed by yours truly) to our radio model just like we added all the sounds up to this point.

Playback Control

Our music plays whenever we click on the radio, but what if we want to pause it and then play it again?

To have this functionality we need to write a custom A-Frame component. Then add that component as an HTML attribute to any object we want to add that functionality to, just like we added the “sound” component to our models before.

You can either write your A-Frame components inline, or write it in an external JavaScript file and then link to it from the html, like I did for this project.

Now we have an audio-toggle component that enables us to play and pause the sound we are hearing. It also allows us to change the autoplay ability via the “playing” property.

And that’s it! We have an A-Frame scene with music, environment sounds and interactive audio.

In the next part we will take a look at how to integrate Tone.js to an A-Frame project and writing more custom components with advanced audio functionality.

Thanks for reading!

Advancements in technology have made Virtual Reality (VR) more immersive and affordable than ever. This immersive environment can be similar to the real world. Or it can be fantastical, creating an experience that is not possible in ordinary reality.

Better yet, high-quality VR devices are available at low prices these days. With a number of smartphone-compatible VR headsets such as Google Cardboard, Samsung Gear VR, Oculus Rift and HTC Vive, VR is showing to be the next big thing.

In this tutorial, we’ll take advantage of that and build a real-time, multiplayer VR game using A-Frame, PubNub, Glitch, and WebVR.

The full GitHub code repository can be found here.

WebVR

WebVR is an open specification that makes it possible to experience VR in your browser. It is a JavaScript browser API that acts as an interface for the VR hardware. WebVR is cross-platform and can be used to develop, view and share VR content on any browser that supports VR. With WebVR, you can open up a browser and get into VR just by clicking a link. Working with WebVR directly requires knowledge of JavaScript and WebGL.

A-Frame

A-Frame is a virtual reality framework that is built upon the WebVR API. It uses the WebVR API to gain access to VR headset sensor data (position, orientation) to transform the camera and to render content directly to VR headsets. A-Frame is an open community project that uses the WebVR API along with HTML, CSS, JavaScript, and Three.js. A-Frame aims for highly immersive and interactive VR content with native-like performance. At the same time, A-Frame wants everyone to be able to get involved with VR content creation. A-Frame supports all major headsets with their controllers.

Glitch

Glitch provides an online code editor with instant deployment and hosting of websites. The editor supports both front-end and back-end code as well as multiple files and directories. Glitch lets you remix (i.e., copy) existing projects and make them our own and instantly host and deploy changes for everyone to see. Firefox Nightly allows you to debug the VR content using debug console.

Gaming Environment

A-Frame Physics System

aframe-physics-system is middleware that initializes the physics engine and exposes A-Frame components for us to apply to entities. When we use its static-body or dynamic-body components, aframe-physics-system creates a Cannon.Body instance and attaches it to our A-Frame entities, so on every frame, it adjusts the entity’s position, rotation, etc. to match the body.

Ball

<a-sphere> primitive creates a spherical shape. You can define its radius color and position. Because of aframe-physics-system, the ball can be converted into a dynamic-body with a certain mass.

Bowling Lane

<a-box> creates shapes such as boxes, cubes, or walls. You can create a rectangle box and make a bowling lane out of it by placing pins and ball on top of it.

Pins

<a-cylinder> primitive is used to create tubes and curved surfaces. These cylinders can be used as bowling pins in the game. Be sure to define the radius, height, position, and mass of the cylinder.

Tracks

The ball cannot roll in the same direction every time you throw it. You can define any number of tracks for the ball to roll, and this track can, in turn, define the direction. This game has 5 tracks, and the movement of the ball on these tracks is controlled by 5 triangles or, let’s say, pointers on the bowling lane.

Surroundings

A scene is represented by the <a-scene> element. The scene is the global root object, and all entities are contained within the scene. The objects’ friction, restitution, and iterations are set to values of 0.001, 0.3 and 30 respectively.

A-Frame has an asset management system that allows us to place our assets in one place and to preload and cache assets for better performance. We place such assets within <a-assets>.

The scale component defines a shrinking, stretching, or skewing transformation of an entity. You can use the scale component to transform a box into a wall behind the bowling lane.

On similar lines, a box can be converted into a button attached to the wall by using the scale component. <a-text> can add text into your virtual environment.

<a-box> can also be used to build borders next to the bowling lane.

The Game

Rolling the Ball

As discussed earlier, the ball can roll over 5 imaginary tracks on the bowling lane. This can be achieved using <a-animation>. Animations can be attached in A-Frame through <a-animation> element by making it as a child of the entity to animate.

Now you can bind these animations of the ball with the 5 pointers so that the animation begins every time one of the triangles is clicked. This can be achieved by writing a component. We can register the component in JavaScript and use it declaratively from the DOM. Components are configurable, reusable, and shareable.

Falling of Pins

When a dynamic-body of mass 17.5 rolls towards 10 dynamic bodies of mass 1.25, some of them tend to fall. After every knockdown, one can count the number of pins that are down. We can check the position of the pins at the end of the animation. If any of the pin’s rotation has its x-value not equal to 0 or -0 then it means that the pin isn’t standing upright. By counting the number of pins that are lying down, you can calculate the score of the player.

The above line captures the x-value of the rotation attribute of a pin. This way you can fetch x-value of rotation attribute of all the pins and save it into an array. Now you can loop through the array and check every value and increment the strike counter.

New Game

The player can start a new game at any point of time by clicking on the New Game button on the wall. It automatically refreshes the game.

Moving Camera

You can move the camera at any point during the game. Here, I have chosen to move the camera every time the player rolls the ball for a better view of the falling pins.

PubNub

With less than 1/4th of a second latency, PubNub can smoothly publish and subscribe messages between multiple VR devices. Let’s convert this single-player game into a 2-player game.

You’ll now have to initialize your PubNub keys. Sign up for a PubNub account and create a project in the Admin Dashboard.

Deciding the Turns

Every player gets two turns. The player switch turns after every two shots. So after every two shots, PubNub can notify the other user that they can take control. In this game, every time the player gets his/her turn the 5 triangle pointers surface on the bowling lane. And when it’s not their turn, the 5 triangle pointers are hidden.

Hide the pointers when it’s not your turn. Here, instead of hiding, I am setting the position to 0.

Make the pointer surface back to the bowling lane when it is your turn. By doing this, you’ll be taking control of the tracks again.

Replicating the State of Pins After Knockdown

After every knockdown, you can capture the position of pins that are down and send it to the other user using PubNub. By doing so, you can replicate one player’s screen on other players’ screens. In the code below, you can see that the position and rotation values of pin 1 are passed to other players using PubNub. On similar lines, you can send rotation and position values of all the pins through PubNub.

Switching Between Static and Dynamic Bodies

Earlier we used aframe-physics-system to convert the A-Frame objects into dynamic bodies. When the player isn’t rolling the ball and is just replicating the screen of another player, the ball should not be a dynamic body in order to avoid falling of extra pins.

When it is the current player’s turn, dynamics is set to true, and the dynamic-body properties are added.

When it’s not the player’s turn, dynamics is set to false, and the dynamic-body properties are removed.

Player 2

Once you are done with publishing data through PubNub from Player 1’s screen, you can read the data by subscribing to PubNub’s channel.

When PubNub receives data related to the position of fallen pins’ position and rotation, you can set the attribute of pins on player 2’s screen to the same values as Player 1 and hence make the two screens identical.

Conclusion

Congratulations! Every time you roll the ball on Player 1’s screen, you can see Player 2’s screen replicating all the movements. Now you can revert this by publishing Player 2’s data back to Player 1 and convert your game into a fully functional 2-player game. It can be converted into a multiplayer game as well. Happy VR gaming!

The full GitHub code repository can be found here.

_Originally published at www.pubnub.com._

In this article, we’ll learn about three great frameworks/libraries that allow any web developer to build a VR app that works on any device in minutes. It will also allow networked realtime interaction by peers from all over the world.

Virtual reality (VR) has come to an amazing level where it does not need any introduction. However, from a developer’s perspective, building simple VR apps still seems to be a complex task, let alone networked multiplayer ones.

What will we build?

_Picture taken at [Martin Splitt](https://www.wearedevelopers.com/congress/" rel="noopener" target="_blank" title="">WeAreDevelopers World Congress by <a href="https://twitter.com/g33konaut/status/996753762172760065" rel="noopener" target="blank" title=")

By the end of this tutorial, you will have a VR app similar to the one seen in the above picture. It will have a basic VR scene where multiple users can connect to your app from their mobile phones by simply hitting a URL on their mobile phone’s browsers.

Don’t know or understand a lot of the terms I just used? Don’t worry, we’ll have a look at all of it shortly, and it will all start to make sense soon!

Basically, for every user who joins your application, a new avatar will appear in your VR scene (Note: I’m just using a fancy term ‘avatar’ for that set of boxes which barely resembles a real human :P ). These avatars will rotate/move in realtime, according to the movement of users’ phones in real life.

This app was demo-ed during my talk at WeAreDevelopers World Congress 2018. You can check out the slides below.

Skip to live demo

The complete project is hosted on Glitch — I think it’s the easiest way to host your community projects or demos. It also allows multiple developers to collaborate on a project remotely. You should totally check it out.

Instructions for the live demo:

• open up the link in a browser window on your computer/mobile.
• open up another instance of the application in another browser window/mobile.

You can see the avatar of each of these instances in the other. Try moving the mobile phone in the air and you’ll see that the corresponding avatar seen on the computer browser moves as well. If you have two VR headsets (even cardboards are fine), you and a friend can put them on and see each other’s avatars move in the scene as you are moving your head in real life.

Play around a bit or read on further to understand what’s happening and how. You can also check out the complete code, hosted on GitHub. Make sure to read the readme file:

Jump to the complete source code on GitHub.

What will we use?

As mentioned before, our aim is to access VR directly in the browsers, without having to download anything. We shall use the WebVR library in order to achieve this.

WebVR is a web framework that allows us to build VR applications which are accessible directly on the web. This completely eliminates heavy downloads and installs, as well as making the VR app device-independent.

_[WebVR](http://webvr.info" rel="noopener" target="blank" title=") Logo

However, even though WebVR gives us the ability to leverage the many advantages of the web, it still requires a considerable amount of complex work. This may, in fact, require knowledge of WebGL and other libraries to be able to build a smooth experience.

A-Frame

This, again, tends to be a bottleneck for the developers of the web to build something that will be eventually served on the web itself. And so, Mozilla’s VR team built a framework on top of WebVR called A-Frame.

A-Frame completely eliminates WebVR’s boilerplate code and allows developers to build VR apps with simple HTML custom tags. With, of course, JavaScript making various bits and piece work together, as always.

_[A-Frame](http://aframe.io" rel="noopener" target="blank" title=") VR logo

Ably

Further, we’ll use Ably Realtime to implement all the realtime —and multiplayer — functionality in the application. Ably is a realtime data delivery platform that solves the realtime message delivery problem by offering features like Pub/Sub and Presence out of the box.

_[Ably Realtime](http://ably.io" rel="noopener" target="blank" title=") logo

Note: The limits for an Ably free account (that is used in the Live Demo) allow you to have a maximum of two instances of the app running at any time. If you wish to have more instances running, have a look at the Ably self-service package and buy more messages accordingly.

Glitch

As you know by now, every WebVR application can be accessed from just a browser. This means we’ll need to host our files in order to be able to access them on a mobile device using a URL.

Glitch is a very convenient way of doing this. You can simply create a new project. When you are done, a URL is readily available for use on any platform or device instantly.

Authentication and assigning unique ids

First things first. We’ll need to setup an auth server that verifies our users’ credentials and authenticates them with Ably, while also assigning a random clientId to each of those. This clientId will serve as a way to identify each of these avatars separately, and handle information such as their respective position updates, and appearance/disappearance as per user login/logout. We’ll set up a simple express server for this, as shown below:

If you observe closely, this express server serves the files present in the root directory of the project. So make sure the “index.html” file that you build further on is in the same folder as the “auth-server”.

If you wish to serve these files locally, rather than on Glitch, replace the code for the listener variable with the following:

Since this is only a tutorial, we are not actually checking any credentials before authenticating the clients. Ideally, the auth server would have a validation step.

The project structure is simple, with the following files all in the same parent folder:

• auth_server.js
• index.html
• main.js

Getting started

Let’s start by building the basic VR setup for the application. We’ll use A-Frame’s entity-component-system (ECS). ECS makes it easy to build any objects in the scene. Every object is treated as an entity which differs from other entities by the various components (or attributes) that are attached to it.

In your HTML file, start off by adding the HTML skeleton code:

The references refer to the following, respectively:

1. Ably’s JS client library
2. JQuery Framework
3. A-Frame’s JS build
4. A local JS file (“main.js”) where we’ll add the logic of the app
5. A-frame’s community contributed text component to conveniently add stylised text to our VR scene.

All the objects we wish to include in our VR scene must go within the a-scene tag in our HTML file, as scene above (pun intended)! This is analogous to the body tag in regular HTML documents.

Next, we will add all the assets we wish to use within the a-assets tag. Adding all the resources under this tag makes sure that all your assets are pre-loaded before your app shows up. This prevents a crappy first look due to slow loading of part of the resources.

Feel free to use your own resources to give the app a customised feel! You can see that we have added two new tags in the above code snippet, let’s dig into them:

a-asset-item — invokes the three.js file loaders. You can use this to load all file types.

a-mixin — is a very useful tag that allows code reuse by letting you specify a set of properties (components) to be applied to a single entity. You can give it an id and reference it multiple times as we’ll see. We will have three mixins, each specifying certain attributes for the avatar that we intend to create — the eyes, pupils, and arms.

Now, let’s add all the static visual elements in our VR scene.

As you can see, we have implemented the complete app using ECS. However, this is not the only way that you can add objects to the scene. A-Frame comes with a few custom entities such as box, sphere, and so on. These custom entities are called primitives.

The code contains very easy to follow comments that explain what each of the entity-component sets is trying to implement in our app.

For VR newbies, here’s something interesting — a sky is like a layer that covers your 360 deg sphere inside which you assume yourself to be standing while experiencing a VR app. It is generally analogous to the sky in real-life which can be seen on the top and appears to drop down near the horizon. We use a-sky in A-Frame to add a sky and the resource to be used can be either a 360 deg image or just a solid colour.

Now comes another interesting part. We require a camera entity. This is a special entity offered by A-Frame. It grabs the continuously changing position as well as rotation values of your mobile phone while you are using an A-Frame powered VR app in the browser. The entity takes advantage of the various gyro sensors in your phone to achieve this under-the-hood. In a computer, the camera entity follows the WASD controls to capture the position and rotation.

Here’s how we can add a camera entity. We can optionally give it a shape and animation, which helps us track its movement by serving as a cursor.

By the end of this section, your VR app should ideally look like what’s shown below. But only if you haven’t switched the resources with your custom ones, of course!

First Look

Voilà! We just finished setting up the basic VR scene.

It’s now time to add some functionality to it — to make the avatars appear, disappear, and move around in realtime as the users log in and out of your application, or simply move their phone around.

Adding realtime functionality

It’s time to spin up some magic into our VR scene. Using Ably, it is very easy to implement this. We’ll use Pub/Sub and Presence, which are both offered as direct use out-of-the-box features by Ably.

Start by connecting your client/user to Ably. Since we are using Token Authentication, we’ll simply add a route to the auth server, as shown below:

Note: we have specified echoMessage: false. This prevents your client from being subscribed to messages published by itself, ensuring lower message count/usage on the whole app. By default, this option is always true.

After successfully authenticating the client, we’ll store the id returned by the auth server in a variable so we can use it later on.

Next, let’s set up the first function that initialises our application. In this function, we set an initial position of the avatar. For simplicity, we’ll restrict the avatar’s rotation/movement to x-axis only, while keeping the coordinates on the other two planes as zero. The initial position on the x-axis is chosen randomly so that multiple avatars do not clutter at the same point in the scene as soon as they appear. We also set some initial attributes such as colour and dimensions.

In order to send this data to Ably, we need to create a channel. I’ve called it vr-channel. After that’s done, we can publish the initial attributes on this channel.

However, we want to continuously publish this data so all the other users are able to receive it in realtime. In other words, we want to publish data as the position and rotation attributes are changing. This data is directly handed to us by the camera entity in A-Frame. We just need to publish this data on the same channel at a high frequency. In this case, I’m publishing at every 100ms.

a-box is a primitive in A-Frame that can be conveniently used to create a 3D box with basic attributes such as dimensions, position, rotation, color, and so on.

You can see that there are three subfunctions within the above function that we have not yet discussed. Presence is a common term in the realtime world which gives you the information about a user/device’s online or connection status. In our case, we would:

• create and make the avatar appear in the scene only when a user comes online (hits the URL)
• and, likewise, make it disappear as soon as a user quits the app (closes the browser window on their phone or browser).

Additionally, Ably allows you to subscribe to presence events. A callback is fired every time a new user logs in or an existing user logs out. This is exactly what we require for our app.

Using channel.presence.get() you can get a list of all the members currently connected to Ably (are online).

Using channel.presence.subscribe('enter') you can get notified whenever a user becomes connected to Ably (comes online).

Using channel.presence.subscribe('leave') you can get notified whenever a user becomes disconnected from Ably (goes offline).

As soon as a user comes online, we need to subscribe everyone else to the attribute changes of the new user’s avatar. These changes, as you can observe, will be in the attr object due to changing position and rotation. Our goal is to update the avatar as these attributes update.

Before that, we need to ensure that the avatar exists at all, or if a new one needs to be created. We do this by using a simple map of avatars where we store the IDs of all the existing avatars, as shown:

Next, we need to make all the users subscribe to changes in the attributes of everyone else, apart from themselves. We do so by making them subscribe to the specific event on the same channel to which it is being published, like so:

When a new user enters, we need to create a new avatar with all the necessary attributes. The following function gets all the initial attributes over Ably. It creates a new avatar with these attributes, and attaches other parts like eyes, pupils, and arms relative to the position of the main box (representing the head of the avatar). This manual positioning becomes easier with the use of a visual inspector tool that comes with A-Frame.

After we have built all the different parts of an avatar, we bind them all together by attaching them to a root avatar. This gives us a way to perform actions like position updates and so on on the avatar as a whole. You wouldn’t want a zombie-like situation with the head moving in one direction and eyes moving in the other, right? ;) This also makes it easy to delete the whole avatar when a user logs out.

If an avatar already exists, we simply update its position and rotation from the continually updating data in the attrobject of the respective users.

Scroll back up to the subscribeToAvatarChanges() function. You will observe that the updateAvatar() is a callback function to a channel subscription which is invoked when the attributes of an existing avatar change. This makes it very easy for us to continually update the actual avatar as well, according to the changing data. We simply update the position and rotation with new values, as shown below:

Finally, we need to ensure that the avatar is removed from the scene whenever a user logs out/ goes offline. This can be done using presence again, by handling the leave event mentioned earlier. Here are a few things you need to do when a user logs out:

We start by deleting the corresponding entry in our array and follow it by deleting the complete avatar from the scene.

That’s it!

We have now successfully implemented a Multiplayer VR app that runs on the web and works in realtime. Go ahead and test it out with your friends and let them witness the magic! If you have been working in a local envrironment, you might need a local server to host your files, as mentioned above. I personally use Glitch for all my VR projects.

You now know the basics of both A-Frame and Ably, allowing you to build both VR apps and realtime apps or — even better — a collaborative app, like the one we just did.

Ideas are already brewing in your mind? Go ahead and build that app you’ve always wanted to! Here’s the complete source code for this application. Feel free to give a shout to me on Twitter if you get stuck or would like to know more about something.

_[Ably Realtime](http://twitter.com/Srushtika" rel="noopener" target="_blank" title="">Srushtika Neelakantam is a Dev Advocate for <a href="http://ably.io" rel="noopener" target="blank" title="). Photo Credits: Radka Klein

More than two months after the iPhone X launch, it remains a conundrum for some to put a finger on exactly which features might justify the $1000+ price Apple has given its flagship phone.

Some might wager on the virtually edge-to-edge OLED display, which for the first time on an iPhone, does away with the tactile home button and prominent top and bottom bezels of every previous model. It is admittedly gorgeous.

As it turns out, however, the iPhone X’s most groundbreaking feature is not even discernible at first glance, even though it literally stares you in the face.

Meet the TrueDepth IR Camera

Apple has been dogged by perceptions that the company’s product roadmap has grown conservative and predictable in the six years following Steve Jobs’ passing. The new TrueDepth IR camera, as Apple calls the iPhone X’s secondary front-facing camera, could well help Apple dispel those rumors of complacency.

New to the iPhone X and housed in the black notch at the top of the display, the technology includes a number of high tech components such as a “flood illuminator,” infrared (IR) camera, and an infrared emitter. The latter emits 30,000 infrared dots in a known pattern when a face is detected, enabling the iPhone X to generate a 3D map of a user’s face. Notably, the TrueDepth IR camera can do this fast enough to support the creation of 3D motion data as well.

TrueDepth IR can create a 3D map of your face using an infrared camera and emitter. (Photo credit: Apple)

At present, there are currently two supported use cases for the new TrueDepth IR camera on the iPhone X:

• Animoji, or animated emoji — the camera animates a 3D emoji with your captured facial expressions, which you can then send via Messenger or SMS.
• Face ID — the camera uses a locally stored 3D image of your face to authenticate and unlock your iPhone X. Because it uses 3D image data, it’s much harder to trick than previous mobile face recognition systems that used a 2D image and therefore could be fooled with a photo.

Animojis are certainly fun, but they only scratch the surface of what’s possible. (Photo credit: Apple)

“What — that’s it?”

Admittedly, the current use cases for this technology are limited.

Animoji are just emoji with a fun twist, and unlocking phones hasn’t exactly been a pain point for modern smartphone users, with the prevalence of fingerprint sensors on many phones and pattern drawing as an adequate secondary option.

What’s incredibly intriguing is that the TrueDepth IR camera technology can and will have much stronger applications in the very near future. As with the iPhone’s previous Touch ID (fingerprint sensor), an iOS SDK makes it easy for app developers to use Face ID and the TrueDepth IR camera for a myriad of other more compelling use cases. In fact, Face ID uses the exact same API as Touch ID, making implementation into mobile apps incredibly simple.

Here are some of the areas where the TrueDepth IR camera could spawn useful or interesting mobile applications:

Financial Services and Internet Banking

For more than a decade, the gold standard in Internet banking authentication has been two-factor authentication. One of the most common forms of two-factor authentication has involved sending a verification code sent to your phone or email address, which must then be manually entered along with your password to access your account.

In the past few years, mobile banking apps have been quick to add support for fingerprint-based authentication. This has made it easier than ever to access your bank accounts online using your mobile phone. For the banks, this means customers have easier access to their services, resulting in increased revenues.

Why, given the already enhanced convenience of fingerprint-based authentication, would banks care about implementing face recognition into their mobile apps? To understand why, let’s consider false positive rates — the rate at which any given authentication protocol provides a false authentication. Apple claims that Touch ID, its fingerprint based authentication, has a false positive rate of 1 in 50,000. Compare that to the chance of guessing a four-digit PIN, 1 in 10,000.

Now, let’s look at the iPhone X’s new Face ID feature. Apple claims Face ID has a false positive rate of approximately 1 in 1,000,000. If true, this represents a 20-fold improvement in accuracy compared to fingerprint based authentication and 100-fold improvement over a four-digit PIN.

That’s sure to get the attention of the banks offering Internet banking, as well as of the consumers who also benefit from enhanced fraud protection. In fact, many banks have already implemented Face ID support into their iOS apps. Chase, for example, added Face ID support within two weeks of the iPhone X’s launch.

Entertainment and Fun

iOS 11 introduced ARKit, a new framework that allows developers to easily create unparalleled augmented reality experiences for iPhone and iPad. Combined with the iPhone X’s TrueDepth IR camera, ARKit opens up a revolutionary capability for robust face tracking across multiple app categories.

Imagine:

• Entering the character creation screen for Assassin’s Creed: Identity and being able to not only choose your assassin’s class but also their visual identity, using your own face in place of the default character model’s
• Having a more accurate view of a prospective match’s face before you choose to swipe left or swipe right on Tinder, Bumble, or POF
• Being able to skip a trip to Sephora or ULTA and experiment with new makeup without shelling out for everything up front

A few years ago, virtual reality (VR) technology was in the same place that the technology behind TrueDepth IR is now: a promising technology needing stronger applications in video games and entertainment.

Oculus, a virtual reality company, was founded in 2012 and ran a successful Kickstarter campaign that year, raising $2.5 million. Just two years later in 2014, Oculus was acquired by Facebook for $2 billion, or nearly one thousand times the initial financing round. Could companies that invest in entertainment applications using TrueDepth IR technology find a similar trajectory?

There are already precedents in the space, too. Two years ago — when face tracking was still based on two-dimensional images — Facebook purchased (for an undisclosed amount) Masquerade, a startup whose MSQRD app lets users create self-altered images and video using face tracking technology.

Medical and Fitness

Imagine being able to have a medical examination at any hour of the day, on-demand, from the convenience of your bed, sofa, or office chair. With the help of the iPhone X’s TrueDepth IR camera, this could become a reality.

TrueDepth IR technology has the potential to be used for medical imaging, because it can create detailed 3D maps of your face, including the surface of your skin. Notably, it can also capture 3D movement, including subtle changes in facial expression.

Consider the following potential medical use cases:

• Have a dermatologist examine an abnormal skin growth, with the ability to rotate the image for a better look at the structure of a potential tumor, for example.
• Have your phone automatically alert you when it notices facial cues or subtle movements that could be associated with the onset of a stroke, seizure, or heart attack.
• For recovering stroke patients, you might have an app that tracks any muscular degeneration over time and provides your doctors with this detailed 3D image data on demand.

While these use cases would certainly not replace the need to have in-person examinations, regular screening via phone camera could play a supporting role in enabling early detection or regular monitoring of known medical conditions.

Consider also that Apple has published a software development kit (SDK) called Apple Healthkit, which allows app developers API access to rich health and fitness information across a variety of sources: fitness wearables, health devices, and software applications. New or updated mobile apps might combine health vitals from Apple Healthkit with 3D facial image data to offer a more complete assessment of users’ health or fitness.

Apple Healthkit integrations will allow TrueDepth IR to be used for health and fitness assessments. (Photo credit: Apple)

The Future is Here

The iPhone X is the first phone to offer depth-sensing IR cameras, but it certainly won’t be the last.

“With the wide spectrum of new use cases that the technology makes possible, the impetus is now on software developers and bold businesses to bring these applications to life,” said Harry Lee, CEO of CitrusBits. Based in Los Angeles and San Francisco, CitrusBits is a mobile app development agency that designs and develops award-winning mobile apps for some of the world’s best known brands, including Burger King, Symantec, and Quiksilver.

“Those that do embrace the technology now can enjoy first mover advantages such as the ability to build a sizable market share and enjoy strong network effects that discourage future market entrants — not to mention the opportunity to get bought up by Facebook, Microsoft, or another titan of industry,” Lee explains.

To see more stories like this one, subscribe to CitrusBits’ newsletter, Bits & Bytes.

Will Patera and Moritz Kassner are the founders of Pupil — a mobile eye tracking platform, which they are building working closely with an open source community. Find out more about their exciting project & feel free to drop by the Pupil Labs community channel on Gitter.

Tell us about a little bit about yourself.

Will Patera and Moritz Kassner founded Pupil Labs in 2013. Andreas Bulling joined as the scientific adviser and partner in 2014. We develop open source software and build accessible hardware for mobile eye tracking and eye tracking integrations for VR/AR. We have backgrounds in computer science, industrial design, and architecture.

What is Pupil? How did it all begin?

Pupil is an open source eye tracking platform. Pupil started as a part of our joint Master’s thesis at MIT. We wanted to be able to record and visualize what someone was looking at (gaze) while navigating through buildings. At the time, there were no accessible or open source mobile tracking tools. So, we decided to build our own hardware and software. We developed a mobile eye tracking headset with our own cameras and proof of concept along with capture and visualization software.

After finishing our thesis, we released the code (on Google Code — yes Google Code was still a thing in 2012) and published DIY instructions, and bill of materials, so that others could build their own mobile eye tracking headset and conduct research.

The community, developed organically around the initial open source codebase and wiki. We started receiving requests from the community to purchase eye tracking hardware, and in late 2013 we formed the company Pupil Labs. The company provides a way for us continue working on Pupil and pursuing the vision of the platform.

Pupil binocular mobile eye tracking headset.

Eye tracking is exciting because it intersects with a diverse range of research fields, application areas, and development topics. It’s fun to meet new people and learn about new areas where eye tracking is employed. Research areas include (but are not limited to): psychology, cognitive science, neuroscience, UI/UX, artificial intelligence, music, education, assistive technology, optics, ophthalmology, optometry, transportation and wayfinding, urban design, architecture, computer graphics, VR, AR, and more. Here are some academic publications that cite Pupil.

On the development side it is equally diverse. Problems/projects address: ergonomics and physiology, optics, electrical engineering, computer vision, pupil detection algorithms and eye movement classification, visualization and more.

What common goals do you have as a community?

We share a common desire to make eye tracking accessible and reducing the barriers to obtaining robust eye tracking data for research and novel interfaces.

What issues related to your project are you most excited about these days?

We are really excited about our latest project called hmd-eyes — a community driven platform for open source eye tracking tools in VR, AR, and MR. We outline the goals of the project in this blog post. The github repo for this project is somewhat bare-bones right now, but more coming soon. We also have a Gitter channel for this project at hmd-eyes.

How do you expect VR to evolve in the next coming years? What will be the most interesting use cases in your opinion?

There is certainly a lot of speculation and excitement surrounding VR and AR right now, and a lot of interest for eye tracking in VR/AR.

We think that it is only a matter of time — maybe only one to two more release cycles — until eye tracking will be integrated into consumerware HMDs (VR, AR, MR). This is exciting. But, will these consumerware platforms be accessible and open, or a black box? Will researchers and developers be able to access the fundamental pupil detection and mapping algorithms or develop their own methods? Will the work that you do within one SDK be portable or locked into one system?

Pupil Labs recently released an eye tracking addon for the HTC Vive AR Headset.

We develop eye tracking hardware add-ons for VR and AR headsets, and have initiated a project called hmd-eyes to create open source building blocks for eye tracking in HMDs. Our goal is to work with and support a community of researchers and developers in creating reusable open-source software modules for eye tracking in HMDs. We know that many developers have already thought of or implemented eye tracking software for HMDs and instead of re-inventing wheels, we should form a community to share knowledge and tools.

Foveated rendering has gotten a lot of press. It is exciting, but maybe just the tip of the application-iceberg. There is so much that can be done within AR/VR that requires robust eye tracking data. There is interesting research into cognition aware computing using eye tracking.

What are the main issues discussed in the Pupil channel on Gitter?

Before Gitter, our only way to get in touch with our community was through Github and a Google Group (mailing list). We still use the Google Group, but have found that Gitter is really nice for quick feedback and sharing within the community.

A lot of the questions from our community are about setup and workflow. When both software and hardware (especially something you wear) is involved, workflows can quickly get complicated. It’s nice to be able to give and receive rapid feedback on Gitter. This helps the community to improve documentation and understand critical areas where improvements can be made.

We also maintain a wiki for Pupil. The wiki is open for anyone in the community to edit and can serve as both reference for developers and a “getting started” for new members.

The Pupil channel is also used to share development progress and post videos to project demos and research papers. It’s always fun to see video demos!

What are the factors that you have taken into account while creating and maintaining the community? What factors contribute to the success of your community?

In the beginning it required a lot of fostering. Most of knowledge was with the maintainers, and we had to answer almost all questions. Now the community has matured to a level that it is self perpetuating and there is more active exchange among its users.

How do you encourage participants’ commitment and contribution to the community?

It is our duty as maintainers to make sure the community is welcoming and supportive, and try our best to give quick feedback. That being said, we feel like we are still learning on how to best encourage participation in the community.

It’s great when contributors step forward and answer questions, share their work in progress, improve documentation, open issues, and make pull requests.

We really envision the community to be driven by others as much as us. With members taking ownership over certain areas of expertise.

Based on your experience, do you feel that the open source communities have changed and evolved over the past years? If so, how?

The great thing about community driven open source work is that, in the best case, it is always changing and evolving. There are certainly more tools now than ever before, that enable people to work together from around the world and to facilitate getting work done online. As communities evolve, so do the enabling technologies.

What advice would you give to someone who wants to start an online open source community?

Just start your project publicly. Initially you may be writing to yourself. Do your best to write docs, updates and code that are concise and a pleasure to read.

Anything else you want to add?

We ❤ Gitter. Its great to connect with community members and interested newcomers. We also rely on it heavily for internal communication; keeping our distributed team connected and coordinated.

Virtual reality is one of the biggest tech trends and a hot topic of 2016. Investment in that sector reached over 1 billion dollars early this year, while new applications of the technology are being uncovered daily — VR is now expected to transform education, travel, corporate training, healthcare, and even financial services. To give you an example, in April a surgery was livestreamed globally in VR for the first time, while attendees of this year’s Sundance film festival could enjoy more than 30 VR-related experiences.

VR craze has reached Gitter too! Check out the list of channels below and don’t hesitate to join the conversation.

• highfidelity/hifi — High Fidelity (hifi) is an early-stage technology lab experimenting with Virtual Worlds and VR. This community is dedicated to open, decentralized virtual world using sensors to control avatars and dynamically assigned devices as servers.
• pupil-labs/hmd-eyes — Building blocks for eye tracking in Augmented Reality AR and Virtual Reality VR or more generally Head Mounted Displays HMD.
• OSVR/OSVR-Unreal — OSVR is an open-source software platform for virtual and augmented reality. It allows discovery, configuration and operation of hundreds of VR/AR devices and peripherals. OSVR supports multiple game engines, and operating systems and provides services such as asynchronous time warp and direct mode in support of low-latency rendering. OSVR software is provided free under Apache 2.0 license. OSVR is maintained by Sensics and Razer.
• CoolGames/OSVR-HDK-Info-from-Cool-Games — This is a GitHub Contribution to the OSVR Project as Lee T. Davy develops his application using VR.
• wantedly/AiR-PingPong — Project to hack a ping-pong table of Wantedly in AR
• vrpn/vrpn — The Virtual-Reality Peripheral Network (VRPN) is a set of classes within a library and a set of servers that are designed to implement a network-transparent interface between application programs and the set of physical devices (tracker, etc.) used in a virtual-reality (VR) system. The idea is to have a PC or other host at each VR station that controls the peripherals (tracker, button device, haptic device, analog inputs, sound, etc). VRPN provides connections between the application and all of the devices using the appropriate class-of-service for each type of device sharing this link.
• artmobilis/ArtMobilis — Project software documentation for a digital art in public space journey — a journey of digital works in augmented reality produced by artists.

Looking for something else? Check out our Explore section or easily start your own channel here.

Did we miss an channel that you think should be featured? Drop us a line in the Gitter HQ and we will add it to the list.

Happy coding!