A virtual machine runs in a window on the host computer and gives a user the same experience they would have if they were using a completely different computer. Virtual machines are sandboxed from the host computer. This means that nothing that runs on the virtual machine can impact the host computer.

Virtual machines are often used for running software on operating systems that software wasn't originally intended for. For instance, if you are using a Mac computer you can run Windows programs inside a Windows virtual machine on the Mac computer. Virtual machines are also used to quickly set up software with an image, access virus-infected data, and test other operating systems.

A single physical computer can run multiple virtual machines at the same time. Often a server will use a program called a hypervisor to manage multiple virtual machines that are running at the same time. Virtual machines have virtual hardware, including CPUs, memory, hard drives, and more. Each piece of virtual hardware is mapped to real hardware on the host computer.

There are a few drawbacks with virtual machines. Since hardware resources are indirect, they are not as efficient as a physical computer. Also, when many virtual machines are running at the same time on a single computer, performance can become unstable.

Virtual Machine Programs

There are many different virtual machine programs you can use. Some options are VirtualBox (Windows, Linux, Mac OS X), VMware Player (Windows, Linux), VMware Fusion (Mac OS X) and Parallels Desktop (Mac OS X).

VirtualBox is one of the most popular virtual machine programs since it is free, open source, and available on all the popular operating systems. We'll show you how to set up a virtual machine using VirtualBox.

Setting up a Virtual Machine (VirtualBox)

VirtualBox is an open source Virtual Machine program from Oracle. It allows users to virtually install many operating systems on virtual drives, including Windows, BSD, Linux, Solaris, and more.

Since VirtualBox runs on Windows, Linux, and Mac, the process for setting up a virtual machine is pretty much the same in each operating system.

Start with downloading and installing VirtualBox. You can download it at this link: VirtualBox Downloads

You will also need to download an .iso file for the operating system that you want to run in your virtual machine. For instance, you can download a Windows 10 .iso file here: https://www.microsoft.com/en-us/software-download/windows10ISO

Once you have VirtualBox running, click the "New" button

Create a new virtual machine.

Next you will have to choose which OS you plan on installing. In the "Name" box, type the name of the OS you want to install. VirtualBox will guess the type and version based on the name you type in, but you can change these settings if you need to.

Configure the virtual machine.

The wizard will automatically select default settings based on the OS type and version you selected. You can always change the settings as you go through the wizard. Just keep clicking "Continue" and "Create" until you get through the wizard. It's usually fine to use the defaults.

Next, start the virtual machine you just created by clicking "Start".

Start the virtual machine.

Once the virtual machine starts up, select the .iso image file you want to use.

Install the operating system on the virtual machine.

Your virtual machine will now load your selected operating system. The operating system may require some setup, but it will be the same setup that would be required if you had installed it on a standard computer.

Windows 10 is successfully running inside a virtual machine.

Congratulations! You’ve run your first Virtual Machine in VirtualBox.

If you are a JavaScript coder who is still late to making up a definitive list of resolutions for 2019, let me give you a hand: Start figuring out how to get into Augmented Reality (AR).

The Augmented/Mixed/Virtual Reality (AR/MR/VR) combo has enjoyed frenetic growth since 2016, coming from a marginal market value of bit more than $6 Billion to one that might reach the $210 Billion in sales (including hardware) by 2022. Of all, Augmented Reality is the one experiencing steady growth.

At first, a JavaScript (web) developer wanting to get into the AR boat might feel discouraged when finding the usual required skills; and then there is who ask Machine Learning or Internet of Things. However if you are mainly a JavaScript developer, consider yourself blessed: the language is recurrently mentioned as one you should know to enter this sector. The reason? Right now a lot of AR development goes on the web. And this is where JavaScript reigns.

_Mobile and Web were the last getting AR capabilities, and are still developing (extract from a [buildAR presentation](https://www.slideshare.net/robman/web-standards-adoption-in-the-ar-market/6" rel="noopener" target="blank" title="))

Augmented Jobs for the JavaScript-fan — Really?

Maybe not too fast. There are many examples where AR/MR/VR shines on its own, specially in niche markets, but the industry hasn’t completely figured out the full value of the technology for the general consumer. Once that it solved, the industry would be certainly making more AR/MR/VR products, which would translate into more jobs.

For some analysts AR is expected to have the most pervasive impact, in part because it doesn't require specific devices and conditions to be implemented as VR does.

AR has utility for almost everything, overlaying useful and relevant information on the world around you. AR can be pervasive in a way that VR cannot.

• David McQueen -Strategy Analytics- from an interview on Twice

It rests on the industry to find how to make AR a more every day life tech. According to some companies, particularly within the mobile phone realm, exploiting better the AR potential reduces to a well-known rule: SIMPLICITY.

While Unity has become the default path for building AR apps, an increasing number need only a sprinkling of AR.

• from an article by Benjamin Devine, Homestory AR

In many cases, resourcing onto the leading AR tools could be an overkill. Instead, a bunch of good UX-driven features over some 2D/3D assets could be more than enough to make striking products. Something a JavaScript developer regularly does.

It is then possible that any JavaScript developer will be embedding (non)standard AR/VR features as an extension of their traditional duties in the future. And if required, JavaScript is robust enough for more complex tasks. The sky is the limit.

Becoming JavaScript-Augmented

Before starting, I would suggest to have a look at the several AR platforms and standards. The same technical constraints affecting the industry are also reflected in the AR world.

For example, there are several platforms, one for each Big Tech (Google = ARCode, Apple = ARKit, MS = ChakraCore, Facebook = AR Studio, React 360, Mozilla = aframe).

After having a quick look at the options, starting fully JavaScript’ed Augmented Reality projects is relatively easy. You can begin by taking any web/app dev framework like Cordova, Ionic, React Native or Vue Native to embed the AR framework of your choice — and deploy 3D assets on top of the real world.

If what you want is to deploy on the web using mostly marker-based AR, you could use GitHub repos like AR.js (free), argon.js (free but limited) or awe.js (paid PaaS but with an old GitHub repository still available). There are a few tailored ones that are harder for the novice, many of them focused on things like facial/head recognition (such as tracking.js and headtrackr).

Or you can boost your project capabilities if you are able to port available SDKs made by AR-related companies. There are many APIs that render as AR on browser too. For example, Mapbox follows that path and it is developed on JavaScript.

I would suggest you to keep it simple but interactive.

However if your ambitions point to also mastering design and animation in JavaScript, you definitively have to learn at least one 3D Javascript package, and THREE.js the most popular. Wait, though, until you have gained a good base of JavaScript and OpenGl as well as geometry, trigonometry, linear algebra or physics. And don’t expect more help from the existing 3D JS GUIs; in particular, THREE.js has none. Challenging but exciting!

Bonus Example

I wanted to prepare a quick demo just to explore the technology, so I took a nice CodePen and modified it to fit a marker-based web-rendered AR animation ported within a clone of Stemkoski's great work with AR.js.

For you to see the example you need a mobile device with a camera and internet (phone or tablet), and either a printed copy of the marker or another device to show it on screen.

Ready? Now open this link using a browser in your mobile device:

https://evaristoc.github.io/ARexample/

Give authorization to use your camera, and point the camera to a marker like below, either printed or in another screen.

NOTE: works on Android and Chrome — it might not work for other devices and browsers ?.

_The original animation can be found [here](https://codepen.io/rainner/details/LREdXd" rel="noopener" target="blank" title=").

Happy New Year!

I hope you will find this technology as fascinating as I do. If so, don't stay alone: contact us at the freeCodeCamp forum and share your questions and ideas.

And if you liked this article, don't forget to give it a ? and to share it on social media.

Thanks for reading, enjoy AR and Happy Coding!!

In this experiment, I talk about how Augmented Reality with JS can be used to make learning more fun and interactive. The case study will discuss the design process, implementation and feedback from children in the age group 2 to 10 years.

Education and Interactive Alphabets learning using Augmented Reality and Javascript

Augmented Reality (AR) has always attracted me, and in this experiment, I try to create a practical AR application. The use-case we will cover is primary education and we will see how we can make the learning fun and interactive. We will make an app to learn alphabets in three languages primarily: Punjabi, Hindi, and English.

The Javascript Augmented Reality app currently doesn’t have plane detection. For simplicity’s sake we are only superimposing our objects over the viewport with 3d motion tracking.

END GOAL

Below is a demo of our Javascript AR experiment. You can download and play with the app here.

The full code has been open-sourced for learning purposes and is available here.

Alphabets in augmented reality and javascript to make education more fun and engaging

The Design Process

To make the learning fun and effortless, I am relying on the following points:

1. Active participation of the child
2. Child’s physical activity instead of sitting in one place
3. A bit of effort in finding the alphabets.
4. Intuitive UX/UI.

The core theme of the app will be quite similar to the famous Pokemon Go Augmented reality app. Only two main components will be involved: the Camera Viewport and Alphabets.

Alphabet UX for AR Game

Iteration 1

2d Alphabets in English, Hindi and Punjabi for our JS Augmented Reality Game

In our first iteration we have 2d alphabets which we will try to merge in our camera viewport. The idea of the Augmented Reality(AR) app is to have children find these alphabet letters in a room or space around them. The prototype after merging the space with 2d alphabets will look something like this:

AR Motion sensor with 2d object

As you can see above, the immersive experience is missing with a 2d model because the human eye sees things in 3d.

Iteration 2

Lets try to create a 3d model and see if we can improve the experience of our Javascript-based Augmented Reality game:

3D Alphabets in English, Hindi and Punjabi for our AR project

And below is the comparison of the experience of a motion sensor with 2d vs 3d alphabet models. As you can see, 3D naturally gives you a much more immersive experience when it comes to Augmented reality.

2d vs 3d AR motion experience

The AR Implementation process

To implement the above AR concept, I’ll be using the following tools and technologies:

1. Ionic Framework: For building the hybrid app
2. Aframe: For bringing the Virtual reality (VR) and Augmented Reality (AR) experience to our app
3. MagicaVoxel: For creating our 3D models

The basic app building process is very simple. You can follow my earlier post to learn how to go about building and deploying an app using the Ionic framework here.

Once you have followed the above tutorial to create a simple app, it’s time to integrate Aframe to bring our 3D alphabets with motion sensors into our app.

Just load the below Aframe core and Aframe loader libraries in ionic’s project index.html file:

<script src="https://aframe.io/releases/0.8.2/aframe.min.js"></script>

<script src="https://rawgit.com/donmccurdy/aframe-extras/v2.1.1/dist/aframe-extras.loaders.min.js"></script>

With this we are ready to do some AR/VR magic in our Javascript code base.

Now in your home component’s home.html, let’s include our 3D models exported from magicavoxel:

And this should make a 3D scene ready for interaction with all motion sensors ready:

Final 3D Virtual Reality scene ready with 3D alphabets

Adding Augmented Reality Effect

The final part of this experiment is to add the Augmented Reality (AR) feeling in our Javascript-based hybrid app. As already explained, Augmented Reality is when you superimpose 3D models or other objects on top of your camera viewport. So the only thing missing is the camera viewport behind our scene.

To do this, we use the camera preview plugin as explained here. And here is the full gist after integration with the camera preview plugin:

We also need to ensure that our backgrounds are transparent so that the camera preview is visible in mobile. This is very IMPORTANT otherwise you might feel that the plugin is not working. Here is the home.scss file with transparency css enabled:

And this is what it would finally look like:

User reaction to our Augmented reality JS game

The final step to measure the success of your concept is real user validation — in our case, kids :) And below is their live feedback recorded.

It was pretty clear that each one of them enjoyed the game and we got full points on fun part. However, initially I had to tell them to move the phone in space to find the letters. Points lost in terms of intuitiveness :(

Points scored out of 10

User feedback for Augmented Reality JS game

Final Thoughts

Well it was an exciting project and I could see a lot of potential for Augmented Reality in learning and education. Children really like it and it surely adds the missing fun factor to education, especially in our monotonous Education system.

Feel free to comment and share your feedback.

Downloads

The code for this app is available in github. Feel free to play and push new features in it. I’ll be happy to push updates over production.

You can download the app for android here.

Advancements in technology have made Virtual Reality (VR) more immersive and affordable than ever. This immersive environment can be similar to the real world. Or it can be fantastical, creating an experience that is not possible in ordinary reality.

Better yet, high-quality VR devices are available at low prices these days. With a number of smartphone-compatible VR headsets such as Google Cardboard, Samsung Gear VR, Oculus Rift and HTC Vive, VR is showing to be the next big thing.

In this tutorial, we’ll take advantage of that and build a real-time, multiplayer VR game using A-Frame, PubNub, Glitch, and WebVR.

The full GitHub code repository can be found here.

WebVR

WebVR is an open specification that makes it possible to experience VR in your browser. It is a JavaScript browser API that acts as an interface for the VR hardware. WebVR is cross-platform and can be used to develop, view and share VR content on any browser that supports VR. With WebVR, you can open up a browser and get into VR just by clicking a link. Working with WebVR directly requires knowledge of JavaScript and WebGL.

A-Frame

A-Frame is a virtual reality framework that is built upon the WebVR API. It uses the WebVR API to gain access to VR headset sensor data (position, orientation) to transform the camera and to render content directly to VR headsets. A-Frame is an open community project that uses the WebVR API along with HTML, CSS, JavaScript, and Three.js. A-Frame aims for highly immersive and interactive VR content with native-like performance. At the same time, A-Frame wants everyone to be able to get involved with VR content creation. A-Frame supports all major headsets with their controllers.

Glitch

Glitch provides an online code editor with instant deployment and hosting of websites. The editor supports both front-end and back-end code as well as multiple files and directories. Glitch lets you remix (i.e., copy) existing projects and make them our own and instantly host and deploy changes for everyone to see. Firefox Nightly allows you to debug the VR content using debug console.

Gaming Environment

A-Frame Physics System

aframe-physics-system is middleware that initializes the physics engine and exposes A-Frame components for us to apply to entities. When we use its static-body or dynamic-body components, aframe-physics-system creates a Cannon.Body instance and attaches it to our A-Frame entities, so on every frame, it adjusts the entity’s position, rotation, etc. to match the body.

Ball

<a-sphere> primitive creates a spherical shape. You can define its radius color and position. Because of aframe-physics-system, the ball can be converted into a dynamic-body with a certain mass.

Bowling Lane

<a-box> creates shapes such as boxes, cubes, or walls. You can create a rectangle box and make a bowling lane out of it by placing pins and ball on top of it.

Pins

<a-cylinder> primitive is used to create tubes and curved surfaces. These cylinders can be used as bowling pins in the game. Be sure to define the radius, height, position, and mass of the cylinder.

Tracks

The ball cannot roll in the same direction every time you throw it. You can define any number of tracks for the ball to roll, and this track can, in turn, define the direction. This game has 5 tracks, and the movement of the ball on these tracks is controlled by 5 triangles or, let’s say, pointers on the bowling lane.

Surroundings

A scene is represented by the <a-scene> element. The scene is the global root object, and all entities are contained within the scene. The objects’ friction, restitution, and iterations are set to values of 0.001, 0.3 and 30 respectively.

A-Frame has an asset management system that allows us to place our assets in one place and to preload and cache assets for better performance. We place such assets within <a-assets>.

The scale component defines a shrinking, stretching, or skewing transformation of an entity. You can use the scale component to transform a box into a wall behind the bowling lane.

On similar lines, a box can be converted into a button attached to the wall by using the scale component. <a-text> can add text into your virtual environment.

<a-box> can also be used to build borders next to the bowling lane.

The Game

Rolling the Ball

As discussed earlier, the ball can roll over 5 imaginary tracks on the bowling lane. This can be achieved using <a-animation>. Animations can be attached in A-Frame through <a-animation> element by making it as a child of the entity to animate.

Now you can bind these animations of the ball with the 5 pointers so that the animation begins every time one of the triangles is clicked. This can be achieved by writing a component. We can register the component in JavaScript and use it declaratively from the DOM. Components are configurable, reusable, and shareable.

Falling of Pins

When a dynamic-body of mass 17.5 rolls towards 10 dynamic bodies of mass 1.25, some of them tend to fall. After every knockdown, one can count the number of pins that are down. We can check the position of the pins at the end of the animation. If any of the pin’s rotation has its x-value not equal to 0 or -0 then it means that the pin isn’t standing upright. By counting the number of pins that are lying down, you can calculate the score of the player.

The above line captures the x-value of the rotation attribute of a pin. This way you can fetch x-value of rotation attribute of all the pins and save it into an array. Now you can loop through the array and check every value and increment the strike counter.

New Game

The player can start a new game at any point of time by clicking on the New Game button on the wall. It automatically refreshes the game.

Moving Camera

You can move the camera at any point during the game. Here, I have chosen to move the camera every time the player rolls the ball for a better view of the falling pins.

PubNub

With less than 1/4th of a second latency, PubNub can smoothly publish and subscribe messages between multiple VR devices. Let’s convert this single-player game into a 2-player game.

You’ll now have to initialize your PubNub keys. Sign up for a PubNub account and create a project in the Admin Dashboard.

Deciding the Turns

Every player gets two turns. The player switch turns after every two shots. So after every two shots, PubNub can notify the other user that they can take control. In this game, every time the player gets his/her turn the 5 triangle pointers surface on the bowling lane. And when it’s not their turn, the 5 triangle pointers are hidden.

Hide the pointers when it’s not your turn. Here, instead of hiding, I am setting the position to 0.

Make the pointer surface back to the bowling lane when it is your turn. By doing this, you’ll be taking control of the tracks again.

Replicating the State of Pins After Knockdown

After every knockdown, you can capture the position of pins that are down and send it to the other user using PubNub. By doing so, you can replicate one player’s screen on other players’ screens. In the code below, you can see that the position and rotation values of pin 1 are passed to other players using PubNub. On similar lines, you can send rotation and position values of all the pins through PubNub.

Switching Between Static and Dynamic Bodies

Earlier we used aframe-physics-system to convert the A-Frame objects into dynamic bodies. When the player isn’t rolling the ball and is just replicating the screen of another player, the ball should not be a dynamic body in order to avoid falling of extra pins.

When it is the current player’s turn, dynamics is set to true, and the dynamic-body properties are added.

When it’s not the player’s turn, dynamics is set to false, and the dynamic-body properties are removed.

Player 2

Once you are done with publishing data through PubNub from Player 1’s screen, you can read the data by subscribing to PubNub’s channel.

When PubNub receives data related to the position of fallen pins’ position and rotation, you can set the attribute of pins on player 2’s screen to the same values as Player 1 and hence make the two screens identical.

Conclusion

Congratulations! Every time you roll the ball on Player 1’s screen, you can see Player 2’s screen replicating all the movements. Now you can revert this by publishing Player 2’s data back to Player 1 and convert your game into a fully functional 2-player game. It can be converted into a multiplayer game as well. Happy VR gaming!

The full GitHub code repository can be found here.

_Originally published at www.pubnub.com._

Microsoft’s HoloLens is pretty freaking awesome. I was introduced to it at my internship this summer, and I’ve had a blast developing for it.

HoloLens apps are created using either DirectX with C++ or Unity with C#. I found that it is generally quicker and easier to get an app up and running with Unity.

In this article, we’ll set up Unity for HoloLens development, create an interactive cube, and spin up the HoloLens emulator so you can see your creation. This article is meant for total beginners to Unity, so if you have experience with the editor, I’d recommend going through Microsoft’s tutorials.

Prerequisites

1. A Windows 10 PC with the April 2018 update
2. The HoloLens Emulator which you can download here
3. Visual Studio 2017 — Community edition is fine
4. Unity — Make sure you add the Windows .NET scripting backend component during installation
5. The HoloToolKit

You can find the full source code here.

Set up

Open up Unity. If this is your first time using the editor, you will be prompted to sign in or sign up. I’d recommend Unity Personal unless you plan on monetizing your app.

After you sign in, create a new project and enter in a name. You should be greeted with this screen after the project completes the initial setup.

Unity editor at first startup

Right click on the Assets folder in the bottom left corner. Navigate to Import Package -> Custom Package, and open the HoloToolKit unity file we downloaded earlier. A Unity popup will appear, click All and import the assets.

Next we’ll create our scene. Select File -> New Scene and save it as Main.

Unity needs a special configuration in order to build a project for HoloLens. Luckily, the HoloToolKit has utility scripts which do this configuration for us. On the top toolbar, select Mixed Reality Toolkit -> Configure -> Apply Mixed Reality Project Settings. Use the default values. After the configuration is finished, apply the Mixed Reality Scene Settings. Delete the Directional Light GameObject.

Awesome! So far we’ve imported the HoloToolKit and set up Unity for HoloLens development. In the next section I’ll introduce you to the basics of Unity, and after that we’ll get a cube to render on screen.

Familiarizing yourself with the editor

This is what the editor should look like at this point. Notice the three boxed-in areas:

The red area to the left is the scene hierarchy panel. Here we can add new GameObjects, and get a high-level overview of the current application. A GameObject is a base class for all Unity entities. They can be a physical object like a cube, or a collection of scripts like the Input Manager in the scene.

The bottom blue area contains the assets menu and console. The assets menu is exactly what it sounds like — it contains the C# scripts, materials, prefabs, and meshes required to run your app.

The right green area is the inspector menu. Here is where you can add and modify the Components of your GameObjects. Components are what make up the behavior of your GameObjects. They determine the look, interactivity, and physics of their parent.

Creating a cube

Now we’ll get our first GameObject to render on screen. In the scene hierarchy panel, right click and select 3D Object -> Cube. A cube should appear in the editor.

Double click Cube in the in hierarchy panel, and in the right inspector panel change the position to (0,0,2). Set the scale to be (.25, .25, .25). Your inspector panel should look like this:

Cube inspector panel

Nice! Let’s go through the build steps so we can see our cube in the HoloLens emulator.

In Unity, select File->Build Settings. The build popup should appear, so hit the build button. In the folder popup, create a new folder called App and choose it as the build destination.

After the build is completed, open the Visual Studio solution in the App folder. Change the debug options to Release, x86, and target the HoloLens emulator. Your options bar should look like this:

Debug options for HoloLens emulator

Run the solution, and after the emulator starts up and your application loads, you should see your cube on screen.

Cube in the emulator

Making the cube interactive

Now that we’ve figured out how to get a cube into our application, let’s make the cube actually do something. When we gaze at the cube, it will rotate, and when we click the cube, it will increase in size.

Navigate back to Unity and create a new script in the assets panel called InteractiveCube. To create a new script, right click on the assets panel and select Create -> C# script.

To add the script to the cube, make sure the cube is selected, and drag and drop the script onto the inspector panel. It should look like this:

Double click on the script in the assets tab and a Visual Studio instance should appear.

Import HoloToolkit.Unity.InputModule at the top of our script, and make InteractiveCube extend IFocusable and IInputClickHandler. Our code should look like this:

using HoloToolkit.Unity.InputModule;

public class InteractiveCube : MonoBehaviour, IFocusable, IInputClickHandler {...}

By extending the IFocuable and IInputClickHandler interfaces, our script component allows the parent GameObject to subscribe to focus and click events.

Let’s make the cube rotate when our gaze is on it. The IFocusable interface requires us to implement two public void methods: OnFocusEnter and OnFocusExit. Create a private boolean field and name it Rotating. When we focus the cube, set it to true, and when our focus exits, set it to false. Our code should look like this:

public bool Rotating;

public void OnFocusEnter(){    Rotating = true;}

public void OnFocusExit(){    Rotating = false;}

We’ll do the actual rotation in Update(). Update() is a special Unity method that is called every frame. To control the speed of the rotation, add a public float field named RotationSpeed. Any public field in a component can be adjusted and initialized in the Unity editor.

public float RotationSpeed;

void Update() {    if (Rotating)        transform.Rotate(Vector3.Up * Time.deltaTime            * RotationSpeed);}

In Unity, the transform is used to control the physical attributes such as size, rotation, and position of a GameObject. We are rotating the parent GameObject around the y-axis at one degree per second multiplied by the speed.

To make the cube respond to click events, add the OnInputClicked method required by IInputClickHandler. Create a public Vector3 variable called ScaleChange. In the OnInputClicked method, we’ll increase the cube’s scale by ScaleChange.

public Vector3 ScaleChange;

public void OnInputClicked(InputClickedEventData eventData) {    transform.localScale += ScaleChange;}

Now that we’re finished with the script, navigate back to Unity. Make sure the Cube object is selected, and set the Speed and ScaleChange variables to 50 and (.025, .025, .025) respectively. Feel free to experiment with different values! Our script should look like this:

Awesome! We’ve finished the InteractiveCube script. Build your app from the Unity editor, and run the solution from Visual Studio.

Summary

In this walkthrough, you learned how to set up Unity for HoloLens development, create an interactive GameObject, and run your application in the emulator.

If you enjoy HoloLens development, I encourage you to go through the Microsoft Academy tutorials. They go over the various HoloLens key concepts in detail and walk you through creating some pretty cool apps.

If you liked the article or have feedback, leave a comment below!

As I discussed in the previous article on hot future tech, artificial intelligence (AI), augmented reality (AR/VR), blockchain, and the Internet of Things (IoT) might seem a long way off from mainstream adoption.

Movies like Ready Player One and Avengers: Infinity Wars only perpetuate this perception by mixing real technologies with fantasy, making the tech we wield in the real world seem primitive in the process.

The reality, though?

AI, AR/VR, blockchain, and IoT are already playing an important role in our everyday lives, with countless examples hiding in plain sight.

In Part One of our series, we explored how Artificial Intelligence (AI) is already deeply integrated into apps we use every day. In this article, we’ll dive into Augmented Reality (AR).

What is Augmented Reality?

Not to be confused with virtual reality (VR), which is 100% driven by computer generated images, augmented reality (AR) is a technology that combines computer-generated visuals with a user’s live view of the real world, resulting in a composite view.

The Pixels movie was a somewhat loose interpretation of augmented reality.

For the most part, AR today is meant to be experienced through the lenses of devices like smartphones. (Microsoft’s attempt to evangelize head-worn AR, or what it calls “mixed reality”, is gaining traction in some areas but can still be considered niche.)

Modern smartphones come equipped with everything you need to enjoy an engaging AR experience:

• A digital camera for providing a video feed of the real world.
• A computer to store and run software applications.
• A powerful processor capable of computer vision (CV), for accurate placement of the computer-generated images “into” the real world.
• A high-res display capable of showing the real-time composite view.
• Gyroscope sensors for sensing changes in orientation, to make the composite view more engaging/dynamic.

There is a massive business opportunity tied to the fact that all the hardware needed to experience meaningful AR applications today can be found inside any modern smartphone.

AR in Ecommerce

It can be challenging to spend your hard-earned cash on furniture you’ve never actually laid eyes on in person. And, even if you have seen a piece of furniture in person, questions remain that might prevent you from pulling the trigger on a purchase.

How will it look with my coffee table and side tables? Is it too traditional for my living room? Will that color look tacky against the floral wall paper and shag carpet?

Likewise, with fashion ecommerce, how do you know whether that designer jacket will actually look good on you without taking it into a fitting room and seeing whether it jives with the jeans and boots you’re wearing?

This is where AR comes in.

Companies like Amazon, IKEA, and Target have already added AR functionality to their mobile apps, allowing you to “place” virtual objects into your environment to preview how they might actually look in your home or office. You can place a virtual vase onto your actual coffee table (thanks to plane detection), or “push” a virtual wardrobe against your bedroom wall without breaking a sweat.

Native mobile ecommerce apps can be upgraded to provide AR previews, as Amazon has done on both iOS and Android.

In fashion, Michael Kors will be using a Facebook AR-based ad campaign to provide users with a simple way to “try on” sunglasses and other fashion merchandise before they buy.

Soon, any Facebook advertiser will be able to launch AR-based ad campaigns.

Ecommerce companies that allow shoppers to try out merchandise with AR believe that doing so will provide a significant lift in conversion rates — and consequently will help grow sales revenues.

AR in Marketing & Advertising

Even beyond the conversion-focused world of ecommerce, businesses are finding that augmented reality can be the perfect way to attract and engage potential customers.

Snap, the photography-obsessed creator of Snapchat, has already deployed a commercial platform called Lens Studio, which allows businesses to “create branded AR experiences”. Snapchat’s AR experiences are AR-based camera filters, called “lenses”, that Snapchat users are already very familiar with, thanks in part to earlier lenses like the Internet sensation, “Dancing Hotdog.”

See BMW’s Snapchat Lens in action below:

Facebook has also begun testing AR-based advertising with select clients (including Michael Kors, mentioned previously,) and is expected to make AR ad campaigns available to a wider audience of advertisers later in 2018. Early results are promising. ASUS reported that its AR-based test campaign saw 10x the engagement of its non-AR counterpart.

Beyond these platforms, CPG companies are creating standalone AR experiences to bring life to food or beverage products, for no other reason than to increase customer engagement and create a buzz. Patrón, the Mexican tequila company, launched an AR app last year that lets you plant your own agave field, from which a distillery and bartender emerge to tell you everything you’ve ever wanted to know about creating Patrón tequila.

The Patrón Experience lets you plant your own agave field and distillery on any flat surface.

For companies planning a new product launch or corporate event, adding an AR experience can help dial up both audience engagement and the resulting media coverage.

Consider Samsung’s Galaxy S9 product launch, where attendees were able to witness their name badges become virtual and interactive Galaxy S9 phones right in their very hands.

AR in Industrial and Medical Procedures

Even if you’re not an industrial technician or a medical surgeon, you can probably appreciate the complexities inherent to these professions. One false move, and lives could be at stake.

Thanks to AR, things are becoming a whole lot easier for these professionals.

Energy giant BP has begun using AR to help technicians in the field perform equipment maintenance. With the new technology, BP engineers wear AR headsets that transmit a live video feed to a remote expert, who can diagnose issues and then place relevant instructions and diagrams onto the field technician’s visual field (via the AR headset) in real time. This keeps the technician’s hands free to perform the necessary repairs.

Similar use cases are being developed in the medical field, where AR headsets can provide real time and hands-free information on surgical procedures and vital health information without having to look away from the patient.

A glimpse of what AR-enabled surgery might look like. Source: The Augmentarium at the University of Maryland (illustration by Brian G. Payne)

AR Gaming

It’s been two years since Pokemon Go first grabbed the world with its killer combination of geolocation and AR-based exploration, allowing every man, woman, and child to live out their Pokemon trainer fantasies in the real world.

Pokemon Go’s massive popularity (with nearly $2 billion in revenue to date) has piqued plenty of interest and funding in AR gaming.

Jurassic World: Alive launched earlier this year and doesn’t diverge very far from the location-based and collections-oriented gameplay made popular by Pokemon Go.

Pokemon Go, but with dinosaurs?

A Walking Dead AR mobile game (Walking Dead: Our World) came out just last month, and Pokemon Go creator, Niantic, is expected to launch AR-based Harry Potter: Wizards Unite at some point this year.

Admittedly, one major weakness of today’s AR games is that the AR component is often non-essential to the core gameplay.

Newly-released Walking Dead: Our World is an “AR-optional” game — and one in which the vast majority of users appear to skip the AR mode altogether (partly because it’s not essential to progress in the game, but also because the AR mode seems to struggle with plane detection.)

Even Pokemon Go — which has an excellent AR experience — is AR-optional. You can catch Pokemon without firing up the AR mode, though finding a rare Pokemon sitting on your laundry machine is MUCH more enjoyable.

Some of these limitations are inherent to the software development kits (SDK) the applications are built on. Apple’s ARKit and Google’s ARCore SDKs determine the ceiling for today’s mobile AR experiences on iOS and Android devices, respectively.

Both companies appear to be committed to enhancing support for richer AR experiences, though.

Announced earlier this year, Apple’s ARKit 2 will allow for much better AR-based gameplay experiences.

For example, consider this Apple demo of an AR game built on ARKit 2 , in which multiple players are able to interact with the SAME virtual objects in a “shared experience.” This mechanic opens up a universe of possibilities when it comes to multiplayer AR games.

Why AR will Gain Mainstream Adoption Faster than VR

Most modern smartphones come standard with every hardware component you need to experience meaningful AR right now.

This low barrier to entry stands in stark contrast to virtual reality (VR).

For VR to work, users must own a virtual reality headset, which enables users to see the virtual world while blocking out all traces of the real world. VR headsets are typically purchased separately from the computers or phones that run the VR applications. While low end VR experiences can be had using a smartphone slapped into an inexpensive, VR headset (like the $15 Google Cardboard), the more immersive and believable VR experiences (on Oculus Rift and HTC Vive, for example) require modern gaming computers and headsets that can cost upwards of $1,500 in total.

The other edge AR has over VR is its versatility.

AR apps are designed for a wide variety of real world use cases. And, because you maintain visibility of your actual surroundings when using AR, safety is not much of an issue — at least no more than when you’re playing Candy Crush or catching up on your Instagram feed.

VR, on the other hand, is best enjoyed in a room without other people, tethered to devices like a desktop computer and motion sensors. This limits VR’s usefulness, relegating VR to use cases around gaming, entertainment, and complex commercial or governmental simulations (for aviation or military training, for example).

Here’s how Apple CEO Tim Cook explained his bullishness on AR relative to VR:

“I’m incredibly excited by AR because I can see uses for it everywhere. I can see uses for it in education, in consumers, in entertainment, in sports. I can see it in every business that I know anything about. I also like the fact that [AR] doesn’t isolate. I don’t like our products being used a lot. I like our products amplifying us. And I think AR can help amplify the human connection. I’ve never been a fan of VR like that because I think it does the opposite. There are clearly some cool niche-y kind of things for VR. But it’s not profound in my view. AR is profound.”

AR’s Accessibility is Profound, Too

As jaw-dropping and eye-popping as some AR experiences can be, AR experiences are actually not difficult to create or enjoy.

From the consumer perspective (demand side), anyone with a modern smartphone can download AR-enabled applications and begin to enjoy the benefits of augmented reality.

From the business side (supply side), creating an AR experience is as easy as partnering with a reputable software company with AR/VR development capabilities.

The hardest part, as it often is with emerging technologies, is taking the initial plunge.

Short answer: using a browser-based Augmented Reality (AR) application. For the long answer, read below.

The idea of playing with AR started as a random interesting experiment. In our company, we strive to stay at the edge of the curve. We share technical novelties and new technologies with each other on a semi-regular basis. Since we are mostly dealing with web technologies, the concept of AR in the browser really took off.

Since AR is mostly an entertainment technology, a practical application was not obvious to us from the start. Luckily, two unrelated things happened at the same time:

• we had just created a mascot for our product — Hubert,
• we had to do a marketing booth at devoxxPL 2018

We decided to bring Hubert to life during the event, in the form of an AR app for people to play with. In our heads, users should be able to:

• render Hubert on a wall background in their phones
• take a picture of the rendered model
• tweet the photo (not the subject of this article)

The end result is available on glitch.com, scaled down and rotated to be suitable for a desktop experience (you can also take a quick look into the source code).

Rendering Hubert in real time

We used AR.js (version from this commit) as the main building block of our AR app — it packages webRTC camera access, marker recognition, and 3D scene rendering. We liked it mostly because you can have a basic demo running in around 20 lines of code.

Under the hood, AR.js can use either three.js or A-frame implementations to render your 3D scenes.

• three.js offers fine-grained control of 3D rendering, and is JavaScript-based. You have probably heard about it in the context of rendering 2D and 3D scenes in the browser.
• A-frame is a web framework designed specifically for building VR and AR experiences. It has an HTML-like markup that is more declarative than three.js, but sacrifices some of the control in favor of ease of use.

We didn’t have a VR or 3D expert (except Mrówa, who prepared the 3D model). As A-frame’s HTML-like declarative syntax looked more familiar to us, we opted for A-frame to do the rendering.

Here you can see the code for rendering Hubert, 30 lines on the dot. We omitted some options and A-frame tweaking for the sake of simplicity. You can refer to the repo to see it all.

This gives us Hubert nicely rendered in the web browser in real time.

Capturing a photo to tweet

Unfortunately, we don’t have a single video feed rendering the whole scene. There is the video from your camera and a rendered 3D scene. We quickly figured out that we would have to capture a frame from both sources and put them together for a nice photo of Hubert.

Taking frames out of a webRTC video stream is pretty straightforward. The best material on the subject can be found here. If your browser has the appropriate API, you need two elements:

• a reference to your source tag
• a destination element in which to put your frame

Then it’s just a simple matter of drawing a 2D image from video to canvas. In our case, both of these are a bit tricky.

The video take we are using is generated and embedded by AR.js. We had no idea how to get it gracefully, so we hacked our way around it with a loop and a DOM selector:

We also needed to hack some scaling. AR.js doesn’t present the raw video feed to the user, they scale it to fill the screen without losing aspect ratio. That means we need to apply the same scaling to our frame. If not, our screenshot will have “more” of the video feed than is shown on the screen. We don’t want to confuse the users here.

What the user sees:

If we take a frame without scaling and just try to copy from point (0,0) we lose margins imposed by AR.js. This is a totally different picture from what is presented to the user:

Suffice it to say we just reverse-engineered the scaling and figured out the bounding box of what the user sees:

To achieve this final result (the same as what is presented live to the user, give or take some camera shake):

Now we just need to get Hubert in the picture. Again, the API for that is very straightforward. To capture a screenshot of a rendered A-frame scene, we need to get the scene’s canvas. The relevant part is copied to our destination canvas, on top of the previously taken video frame.

Getting the relevant part is the tricky bit in our case. Thanks to the AR.js scaling, we cannot simply get the “perspective” shot of the scene and use that. It will look too wide or too short, depending on orientation.

For landscape mode (width > height), the scaling method we used for video works perfectly well.

For portrait mode, it works great on a PC… However, once you enter the realm of mobile devices, the scaling breaks and the screenshot doesn’t look nice. You get this skinny Hubert…

…instead of our lovely, bubbly mascot in all his glory:

We are still not sure why that is the case. We made the mistake of not testing it out thoroughly on actual mobile devices, thinking it would work the same as it does on the development machine. (Yes, we know how bad that sounds, but that’s the reality of it.) During the conference, we managed to figure out the formula for portrait scaling and introduced a fix:

It’s not pretty. It’s one of those “it’s late, it works, just leave it” fixes. The values presented above produced a satisfactory result and we left it at that.

With that, we have a picture of Hubert in the real world! It can be retrieved from the destination canvas element and displayed on the page or sent to the server to tweet out.

Summary

AR in the browser is possible. Even more, it is possible on mid-grade mobile hardware (as of June 2018). Getting it to work on every phone and browser is still a long shot, so don’t count on it for wide, diverse userbases.

However, if you have a somewhat controlled environment, augmented reality on a phone can be used to create unique experiences. These don’t require special hardware or workstations and that is a big, big plus. Just make sure to test it on actual devices ahead of time.

I ended up watching a video where a guy was proposing to a girl using VR. So, I decided — that was it! That was how I was gonna do it. Not the proposal part though.

While I was contributing to Mozilla, I’d created few small VR projects using A-Frame — Mozilla’s web-framework for building virtual reality experiences. And believe me, even if you don’t know much about VR or AR, you can easily create a VR scene using A-Frame. The only prerequisite is HTML, which you can learn easily here. For a better understanding, though, I recommend that you go through A-Frame School, which is a great collection of tutorials intended for beginners.

So I’d decided that I was gonna use A-Frame, but I wanted more than just a simple VR scene displaying “Happy Birthday.” In the end, I chose to create an AR scene. I found a great project called AR.js. If you wanna get started with AR.js, here is a great article for beginners.

Building a basic AR web app

AR Scene, created using AR.js

To watch the AR scene, you have to:

• Open this HIRO marker image in your desktop browser.
• Open this AR web app in your phone browser, and point it to your screen.

When you scan a marker (here, a HIRO marker), it’ll display an AR scene on your phone, just like the image above. I used a plain HIRO marker, but you can create your own marker as well.

So, after adding all these libraries, my code looked like this:

_Basic AR web app [ [Demo](https://hackyroot.github.io/A-Frame-Examples/Happy_Birthday/Basic.html" rel="noopener" target="blank" title=") ]

Please note that while accessing any AR web app, if you get any prompt asking permission to access the webcam, please allow it. Otherwise the app won’t work.

Add 3D models and fonts

So, now we have simple AR web app working on our device. But what’s a birthday without cake?! Fortunately, A-Frame supports three types of 3D models: glTF, OBJ, and COLLADA. Learn more about 3D models in A-Frame here.

I downloaded some 3D model files of cake from Google Poly. You can import any asset files in A-Frame using the _ tag. You can also import separate fonts, in case you want to use a different font.

I was pretty convinced that A-Frame couldn’t be more awesome. But, wait…

Add audio

Birthdays are also not complete without the birthday song, right? And A-Frame supports Audio files as well. You can use the _ tag to import your files, under the tag.

Import asset files

Add particles

What’s the thing that comes to your mind when you hear about a birthday — after cake, of course? A party, right? So, let’s add confetti to our AR scene, using A-Frame’s Particle System Component.

Add libraries for confetti and 3D text

Let’s put it all together

3D Models

This code will display the 3D model of the cake. But as you can see, I’ve added a few values in the rotation and scale fields. So, let’s discuss that. According to A-frame’s GitHub page:

• Rotation: the rotation component defines the orientation of an entity in degrees. It takes the pitch (x), yaw (y), and roll (z) as three space-delimited numbers indicating degrees of rotation.
• Scale: the scale component defines a shrinking, stretching, or skewing transformation of an entity. It takes three scaling factors for the X, Y, and Z axes.
• MTL: stands for Material Library File (.mtl) Material library files contain one or more material definitions, each of which includes the color, texture, and reflection map of individual materials. These are applied to the surfaces and vertices of objects. Material files are stored in ASCII format and have the .mtl extension.
• OBJ: a file format that was created as a simple way to import and export geometry from different 3D applications. This is a common file type used by many 3D design solutions.
• Suggestion: If you don’t see your model, try scaling it down. OBJ models generally have extremely large scales in comparison to A-Frame’s scale.

If you’re wondering how I knew the exact values for rotation, well I didn’t. I used an amazing tool created by the Mozilla team called A-Frame Inspector, built for this purpose only.

To learn more about <a-obj-model>, visit this link.

Display 3D model of Cake

Particles

Well, this code may look overwhelming at first sight, but believe me, it’s not. We discussed Rotation earlier, but let’s talk about other fields as well:

• Position: places entities at certain spots in 3D space. The Position takes a coordinate value as three space-delimited numbers.
• Preset: preset configuration. Possible values are: default, dust, snow, rain. Here we chose default in order to display starts.
• Color: describes a particle’s color. This property is a “value-over-lifetime” property, meaning an array of values can be given to describe specific value changes over a particle’s lifetime.
• Acceleration Value: describes this emitter’s base acceleration.
• Particle Count: the total number of particles this emitter will hold.
• Direction: the direction of the emitter. If the value is 1, the emitter will start at beginning of particle's lifecycle. If the value is -1, the emitter will start at end of particle's lifecycle and work it's way backward.
• Rotation Axis: Describes this emitter’s axis of rotation. Possible values are x, y and z.

To know more about A-Frame’s Particle Component System, visit this link.

Text and Audio

Well, you use can use _ as well, but I decided to go with Text Geometry Component for more options. It’s used to generate text as a single geometry.

• Material : The text geometry component defines just the geometry. We can apply any three.js material to the geometry.

<a-entity text="value: HELLO" material="color: red; src: #texture"></a-entity>

For more details, visit three.js’s documentation.

• Text Geometry: string and font value. (you should edit the text in this part, otherwise you’ll end up wishing happy birthday to my friend ??)
• Sound: defines the entity as a source of sound or audio.
• Autoplay: describes whether to automatically play the sound once set.
• Loop: describes whether to loop the sound once the sound finishes playing.
• On: an event for the entity to listen to before playing sound.

Resources:

• Source code
• Demo

Yes, you made it ? ? ?. You’ve created your first AR application. If you did everything correctly, you should now see something like the image below:

Finally, Happy Birthday Krupa!

If you like my work, please follow me on Medium @Pratik Parmar or add me on LinkedIn. Feel free to reach out to me on Twitter: Pratik Parmar or comment down below, in case you need any help.

Apart from Open-source contributions at Mozilla, I’m a Microsoft Student Partner and community member at GDG Baroda. I would like to thank Mozilla and the MozillaIN community for providing me a chance and the resources to learn about VR/AR and Open Source.

This is me, Pratik Parmar signing off till the next tech adventure. Over and Out…

[ Edit: Thank you Vikranth Kanumuru for drawing my attention that URL of source code was broken. It’s been updated now, please go ahead and try now. Keep coding, keep rocking ]

Augmented Reality. The future screams of it. The present is full of it: filters, games, and AR apps are popping up daily. Imagine a world where everywhere you look there are visual data aids to augment your comprehension.

Now, I would rather go all in on VR, especially after seeing Ready Player One. Oh how I wish I could be transported to Oasis right now! However, AR has more real world use cases from furniture shopping to building industrial factories — the possibilities are endless.

Augmented reality will be an essential part of your daily life. It will change everything.— Tim Cook

Now, these claims have to hold the test of time (powerful use cases, realistic visuals, processing speed of hardware, and so on). Until then, Mobile AR is possibly the test bed for developers, before a headset or something similar is part of your everyday tech, along with a community of AR developers and a bevy apps.

Today we will be using Apple’s ARKit (which has made AR dev so much easier to explore) to get our feet wet and experiment with AR. We will build a basic superimposing object app in “additive” AR style to the camera view. To make things interesting, we will be placing Harry Potter PortKeys as our AR objects, which will transport you to fantasy land (this part is left to your imagination for now. I will be building a more comprehensive version of the app in the next part of this AR series, so stay tuned!).

_Image [source](https://www.hp-lexicon.org/thing/portkey/" rel="noopener" target="blank" title=")

What is a Portkey?

(For muggles, here is the definition wink wink):

A Portkey, in Harry Potter world, is an enchanted object which, when touched, will instantly transport a person from point A to point B. The object is usually a worthless piece of junk and is randomly placed around so as not to attract attention.

Project (around 3 hours)

In this tutorial, you’ll learn to build a Unity app with ARKit. We’ll add an augmented reality layer featuring Harry Potter PortKeys. Most of the time you spend will be on installations, so grab your coffee and get ready!

Step 1: Installation

In case you don’t have it already, download and install the latest version of Xcode for your Mac and Unity.

Install the personal free version of Unity, but make sure “iOS build Support” is checked.

You will also need a iOS developer account and iPhone, preferably, to take the AR app you build out for a spin.

Step 2: Setup a Unity project with ARKit plugin

Create a new 3D project called “ARHarryPotterApp”.

Once the project is created, from the “Asset Store” tab, download ARKit into your project.

Step 3: Create the AR scene and add assets

Let’s start off with an example scene that comes packaged with the downloaded ARKit asset. Navigate to the example scene on the left panel and double click open “UnityARKitScene”.

This will open up a basic cube asset placed in the “Scene” tab, which is your field of view. The “HitCube” asset here can easily to be replaced with any asset of your choice to create your own unique augmented reality scene.

There are several properties of each asset shown on the Inspector panel on the right, such as shadow/lighting/rending and so on. We will not go into detail about these properties here (but I’ll discuss them more in an upcoming post).

You need to watch out for two things here in the example. They’ll need to be repeated for any new asset added to the scene:

1. On the inspector pane, Add “Unity AR Hit Test Example” component and attach it to a script.
2. Drag and drop “HitCubeParent” on the left panel onto the “Hit Transform” in the inspector panel on the right in the “Unity AR Hit Test Example” component.

Make sure you complete these two steps for any new asset added to the scene, as this helps place the object in the horizontal plane.

For our app, we will place random Harry Potter PortKeys in the scene from the free “Halloween Pack” downloaded from the asset store.

Place the assets that you fancy in the scene and make sure you add the components “Unity AR Hit Test Example” and “Hit Transform” to each of these assets as shown below.

Step 4: Building the app

Finally, now it’s time to build the app. Select File -> Build Settings. Check “Unity ARKitScene” and select iOS platform, and then click “Switch Platform”. This will import assets and setup the stage.

You can then click player settings and check the inspector to make sure “Target device”, “Target min iOS version”, and “sdk” are all set up as you want through your system settings.

At this point, you are ready to “Build” the app and select your destination directory…this could take a while.

Step 5: Run your very first AR app

Once the build is complete, open the xcodeproj from the destination folder of the build. Connect your iPhone (with compatible iOS and xcode versions), sign the project to your Team (you will need an iOS dev account), and launch the app.

Violà, your very first AR app is ready! Look around in the field of view, spot the Harry Potter PortKeys, and be transported away to a magical land ;)

If you enjoyed this, please clap ? so others can see it as well! Follow me on Twitter @HariniLabs or Medium to get the latest updates on other stories or just to say Hi :)

PS: Sign up for my newsletter here to be the first to get fresh new content and it’s filled with a dose of inspiration from the world of #WomenInTech and yes men can signup too!

My name is Maxime Peroumal. I’m 16 and I built my own VR headset with my best friends, Jonas Ceccon and Gabriel Combe. And it ended up costing us $100.

I started programming when I was 13, thanks to my math teacher. Every Monday and Tuesday, my friends and I used to go to his classroom to learn and practice instead of having a meal at the cafeteria.

I spent one year building a very basic 8-bit OS from scratch and competing in robotics contests with my friends.

I then got interested in VR and with my friends we agreed that it would be really cool to create our own world in VR where we could spend time after school. But facing the fact that an Oculus was $700 at the time, we decided to build our own headset.

3D printed parts of the headset

Making VR accessible to everyone?

DARROW; J. R. EYERMAN/THE LIFE PICTURE COLLECTION/GETTY IMAGES

It was because of an anime called Sword Art Online where the main character is in a virtual reality RPG that I fell in love with VR. I wanted to understand every aspect of it.

I bought the cheapest components I could and we started by learning the very basics of the physics and math behind VR (proper acceleration, antiderivatives, quaternions…). And then we re-invented VR. I wrote WRMHL, and then FastVR with Gabriel. Putting all of this together, we ended up with a $100 VR headset.

A fully hackable VR headset and development kit

To speed up VR development time, we built FastVR, an open-source SDK for developers that is easy to understand and customize. It works like this:

• The core headset computes the position of the headset in space;
• The position is sent from the headset to WRMHL, and part of the CPU’s power is dedicated to reading those messages;
• Then FastVR retrieves the data and uses them to render the VR game.

Everything you need to build the headset has been open-sourced and can be hacked.

Why open source?

I want to make VR mainstream. So I reached out to Oussama Ammar, one of the co-founders at The Family. I talked to him about setting up a company and launching a Kickstarter.

But he convinced me that for now, it’s better to wait on starting a business, to keep meeting others who have the same goals, and to keep learning.

We took a trip to Silicon Valley and Oussama introduced me to the chief architect at Oculus, Atman Brinstock. And they gave me some precious advice: make all of this open source.

The Next Step?

There are still a lot of technical points that we want to improve.

Our big focus right now is on a standalone VR headset, which we already have as a simple version, and cheaper 3D tracking.

All of this will be released soon.

How do I get started?

If you want to learn more about the technical side and build your headset, just follow the guide by clicking here. Star the repo if you liked it ⭐️

I would love to hear about what you’ve experienced while building the headset, or if you need any help or have any questions. Ping me at maxime@relativty.com or @MaximePeroumal.

We’ve all heard of Augmented Reality (AR), but at this point there are few opportunities to see this technology in action. We know that AR allows us to see virtual elements fused with the real world around us. For example, with AR we can see a virtual sofa in our room. Apple now lets us see new items added to the real world using the cameras in our phones.

Apple introduced ARKit, the framework for creating experiences in Augmented Reality, in iOS11. ARKit uses Visual Inertial Odometry (VIO) to map the surrounding environment. VIO blends the data provided by the camera sensor with Core Motion data. Core Motion data are collected through the accelerometer, gyroscope, pedometer, magnetometer, and barometer.

All these inputs allow the device to understand its movement within a room. With ARKit, the iPhone and iPad can analyze a scene and find the horizontal planes of a room. ARKit can locate tables and floors, and can trace and position objects at precise points.

ARKit also uses the room sensor to measure ambient light and apply the correct amount of light to virtual objects. ARKit is compatible with Apple’s A9, A10, and A11 processors. To develop with ARKit you can use Metal, Scenekit, and third-party tools such as Unity and Unreal Engine.

Let’s now see how to create an AR application using ARKit.

Development environment

To get started we need:

• The release version of Unity 2017.1.0 or later. ARKit is also compatible with the Experimental VR build used for macOS VR content creation and Unity 5.x versions of Unity 5.6.2 or later.
• iOS 11 or later
• XCode 9 beta or later, with iOS SDK that includes ARKit Framework
• iOS that supports ARKit (iPhone 6S or later, iPad 2017 or later)

Procedure

Start Unity. The project window will open.

At this point, we have to create a new, empty project:

1. In the window, click on New for a new project.
2. Write “ARKitty” in the Project name text box.
3. On the same window, press the Create project button.

We have created our AR project!

Open the Asset Store by clicking on that tab.

Next, search within the store: write “ARKit” in the text box and click on the magnifying glass icon.

Scroll through the Asset Store window until you find “Unity ARKit Plugin.” This is the plugin that integrates ARKit within Unity.

Click on Unity ARKit Plugin and scroll down to the import button and press it. Press import again to import the same project, and import once more on the plugin element window.

Now go back to the Asset store, press the Home symbol, and write “Cute Kitten” in the search text field. Press the search symbol and import the Cute Kitten model as you just did with the ARKit plug-in.

Scroll with the cursor to the import button and press it. Press import again on the plug-in element window.

Search the “UnityARKitScene” scene in the UnityARKitScene folder under “Asset/Examples.”

Drag the scene “UnityARKitScene” under Hierarchy.

Go to the assets and find Kitten.

Go under Model and drag “kitten” under “Hierarchy-> HitCubeParent”.

Remove HitCube and RandomCube from the scene by right-clicking and then hitting Delete. Select GeneratePlanes, ARKitControl, and click on the inspector and uncheck Tag.

Go to Main Camera under CameraParent and, in Inspector, set Near to 0.01.

Click the Scene tab. Click on the kitten in Hierarchy to select it. Now we see our kitten in the Scene view in three-dimensional space. Go to the asset “UnityARKitPlugin-> Plugins-> Helpers” and take the UnityARHitTestExample.cs script. Drag the script into the kitten inspector.

Drag the kitten in the Hit transform field of the “UnityARHitTestExample.cs” script.

Save the scene by selecting “File-> Save Scenes” and call it “ARKittyTest”.

Finished! It’s time to try the new application in Augmented Reality.

Go to the File menu and choose Build Settings. In the window that opens, under Platform, choose IOS.

Press the Player Settings button and scroll down until you find Bundle Identifier. Here, write an identifier of the type: “com. .arkittytest”.

Click the Build button in the Build Settings window and save the project to iOS as “ARKittyTest”.

At the end of the process, find the XCode project in the Finder named “Unity-iPhone.xcodeproj”. Double-click to open the project with XCode. In XCode, click Unity-iPhone and go to General. In the identity field, write the same Bundle Identifier inserted in Unity’s Build Settings.

Under Signing in the Team pull-down menu, select the registered team name.

Select the device (iPhone / iPad) before connecting to the Mac as the target device.

Press the XCode arrow key to “Build and Run the Current Scheme.”

In the end, when we frame a surface near us and tap on the phone screen, we can see our Kitten live in front of us.

Mission accomplished! The following video shows the complete procedure.

Francesco Pallotta is a senior software engineer expert in software design and development. He works in the field of Space and Defense and deals with application development techniques for virtual reality and augmented reality.

Do you want to read more about Virtual Reality, Augmented Reality and Mixed Reality? Follow me on Medium and Twitter.

Did you enjoy this post? Recommend it, by give it some claps. Thanks!

Virtual Reality (VR) is exciting. It’s also the New Frontier of app development.

VR is poised to give birth to new forms of storytelling and emotionally powerful experiences. Yet making VR is perceived as intimidating: it’s expensive and requires both special hardware and skills.

But that’s changing, as intuitive tools and affordable hardware are making VR development accessible. This tutorial will show you how to build a 360 video app on Android and Google Cardboard in just a few minutes. And barely any coding required ;)

VR development should not be a barrier to bringing your ideas to life. Let’s get started:

What you need

Here’s our grocery list:

? An Android phone with a gyroscope to sense head movements, running on KitKat or newer OS.

? A Cardboard headset. If you don’t own one, you can find many on Amazon for less than 10 dollars. This one is my favorite.

? Unity3D, a cross-platform game engine, that you need to install on your computer, version 5.6 or newer. We will use this software to build our whole project.

? The GoogleVR SDK for Unity, which you can download beforehand.

? A 360 video. Shoot one with a 360 camera (here’s one you can plug into your phone) or find one online.

How are we building this app?

Unlike regular video that has a rectangular frame, 360 video has the shape of a sphere. So, we first need to create a spherical screen to project our 360 video onto. The player (or viewer) will be located inside this sphere and will be able to watch the video in any direction.

The steps below should make you feel empowered to make your own changes, by explaining how it all works under the hood. For step-by-step instructions, refer to the video

Step 1: Build a Sphere ?

First, let’s open a new Unity Project if you are starting from scratch (or a new Scene if you want to integrate the 360 video player in an existing project.) Think of a Scene as one level of a video game, and a Project as a full game.

Afterwards, add a sphere object in the Scene, placed at its center (Position = 0, 0, 0), with a radius of 50 (Scale = 50, 50, 50). The Camera’s position should also be set to 0, 0, 0. The Camera is the player/viewer’s eyes so we want it at the center of the Sphere. Placing it elsewhere would make the video look distorted.

Once the Camera is placed inside the Sphere, the latter is no longer visible in the Scene. Don’t worry, there is an explanation for that! Indeed, most game engines do not, by default, render by default the inner side of 3D objects. This is because we rarely need to see them, it would be a waste of resource to render them. We’ll fix that next.

Step 2: Flip the Sphere’s Normals ?

In our case, we do need to see our Sphere from the inside. That’s the whole point of the app, so we are going to turn it inside-out.

In Unity, spheres are not actually spheres (what? We’ve been lied to all along!), they are polygons made with thousands of teeny, tiny facets. The external sides of the facets are visible, but not the internal ones. For that reason, we are going to make a program to flip these little facets like pancakes.

In 3D geometry, we call this transformation « reversing normals » or « flipping normals ».

We’ll use a program called a Shader, that we’ll apply to the Material of the Sphere. Materials control the appearance of objects in Unity. Shaders are small scripts that calculate the color of each pixel rendered, based on lighting and information pulled from their Materials.

Hence create a new Material for the Sphere, then a new Shader applied to this Material. We need to write custom code for the Shader… but have no fear, you can copy-paste the code below:

This little Shader is going to turn each pixel of the sphere inside out. Now our Sphere appears like a big white ball, viewed from inside, within our Scene. The next step is to turn this white sphere into a video player.

Step 3: Project your 360 video inside the Sphere ?

Here you need to have a 360 mp4 video on hand. Import it in the project, then drag it onto the Sphere. And that’s when the magic happens: a ‘Video Player’ component appears and boom, the video is ready to play.

You can play with the settings like loops and audio. It also supports streaming!

Step 4: Set up Google Cardboard ?

In this step, we’ll make the experience really feel immersive. That’s why we want to view it in a VR headset, here a Google Cardboard.

We are going to create a “stereoscopic” view (the screen will be split in two, with some fisheye effects on both sides — one side for each eye), using the GoogleVR SDK. The fisheye effect on each eye, combined with the distortion of the Cardboard’s plastic lenses, is what gives you the illusion of depth and immersion.

To add the GoogleVR SDK to our project, download and import the plugin, then we’ll adjust a bunch of Android settings:

• Go to _top bar menu > File > Build Se_ttings. Add your open scene if it isn’t already added, then select Android in the list of supported platforms.
• Click on Switch Platform. It should take a little while the first time you make the switch.
• Click on Player Settings. Components appear in the Instructor panel.

In the Player Settings’ Instructor, under the ‘Other Settings’ section:

• Check Virtual Reality Supported. Under Virtual Reality SDKs, select the + icon, then select Cardboard to add it to the list.
• Enter a package name into the Bundle Identifier field (for example, com.yourdomain.demo360). It has to be unique and is used to distinguish our app from others in the Google Play store.
• Set the Minimum API Level drop-down menu to “Android 4.4 ‘Kit Kat’ (API level 19)”.

Afterwards, take the ‘GvrViewerMain’ element from the GoogleVR\Prefabs folder in the Project Browser, and drag it into the scene. In the Inspector, give it the same Position as the center of the Sphere — (0, 0, 0).

The GvrViewerMain prefab controls all VR Mode settings, such as adapting the screen to the Cardboard’s lenses. It also communicates with your phone’s gyroscope to track your head movements. When you turn your head, the Camera and what you see also turn inside the 360 video player.

Now you can look in all directions when the video is on and the screen is split in two, to accommodate both lenses of the Cardboard.

Step 5: Run the app on Android ?

For our final step, we’ll run the app on an Android phone and share it with friends!

There are two ways to do that:

• Go back to _File > Build Setti_ngs. You can plug an Android phone with a USB cable to your computer and click on Build & Run. This installs the app straight to your phone.
• The other option is to click on Build only. This does not install it on a phone, but instead generates an APK file. You can share the APK by email with anyone who wants to try the masterpiece you just built. They have to double-tap on the APK attachment to install it on their phones.

During the build process, you may be asked to select the root Android SDK folder. If that’s the case, download the Android SDK then select its folder location.

Launch the app, pop your phone into a Cardboard headset, you’re good to go! You can replace the video with anything in 360 format and experience VR 360 immersion at home.

Going further

Congratulations, you made a 360 video app, and you are one step away from creating a VR video app! While the terms are often used alike, 360 and VR define two different experiences:

• 360 video is recorded from all angles, with a special camera or an assembly of multiple ones. The user can watch in any direction desired, but there’s no interactivity to the experience.
• VR usually refers to a computer-generated environment in which the user is immersed. It is an interactive experience: the player can move around and control objects, in addition to looking in all directions.

Your new app can serve as a starting point to build a richer VR experience. Unity has plenty of features you can leverage, such as adding 3D elements or cool particle effects ✨ to overlay and enhance your video, or throwing in some interactive elements.

You can also place a full 3D environment inside the 360 video player and use the latter as a skybox. The user can navigate in the scenery, using this nifty walking script.

Let your imagination run wild and show me your creations: tweet me @AdrianaVecc or leave a comment.

Crafting beautiful VR stories is hard. Building them shouldn’t be.

If you liked this article, please press the ? below so other people will see it.

Adriana is an artist & product designer who makes VR experiences that build empathy. If you want to bring your VR ideas to life, shoot us an email: hello@vrtiginous.com ?

This piece was written & edited in collaboration with TOPBOTS writer Nick Hall.

Regular reality is being disrupted by virtual and augmented reality (VR/AR). The biggest names in tech are battling to power the next generation of entertainment, education, and communication.

Facebook acquired Oculus to make next generation social networking virtual. Apple CEO Tim Cook claims augmented reality will be “as big as the iPhone.” Microsoft’s HoloLens, Google’s Tango, and Intel’s Project Alloy are just a few of the myriad of developments underway to make VR/AR devices as ubiquitous as computers and phones.

Another major trend, artificial intelligence (AI), boosts the technical complexity behind compelling VR/AR experiences. Despite the hype, the progress that AI drives in VR/AR are by no means artificial.

Here are 8 specific ways that AI makes our virtual realities even more real.

1. Physical environment mapping

Occipital’s Bridge is one example of how AI can help to map entire environments in real time and blend the results with a virtual world. The external ‘structure sensor’ on Bridge feeds into an AI system and allows a ‘mixed reality’ experience entirely in the headset. This works thanks to the magic of instant depth perception and precise positioning with six degrees of freedom. The result is that you get a fully immersive VR experience with real world structures.

The fledgling system can already produce CAD-quality models of your house, so you can try furniture and decorations before you buy. This mixed version of reality could provide precise AR shopping experiences in local retail stores. The company is also inviting developers to build new apps and is taking an open-source approach.

Google Tango is arguably more advanced and already in the wild on the Lenovo Phab 2. So far, most of the apps targeted at the phone have been relatively simple games. We expect that to change rapidly.

2. Precise depth perception

It’s one thing to map the walls of a building, it’s quite another to map the constantly shifting internal organs of a patient on the operating table. We’re closing in on a world where a surgeon will see your major organs through a headset, with a modern-day slant on X-ray vision. Indeed, we have the first concepts from the likes of mbits imaging and the ‘Surgery Pad’.

Looking further into the future, surgical robots will take over routine procedures. An essential part of these devices will be automatic depth perception and instant, accurate adjustments. Even regulated breathing causes slight shifts in the patient’s position, so only the precision of AI can cope with the vast number of calculations that will make the difference between a clean cut and a severed artery.

3. Selective hazard warnings

Soldiers in the field make split-second decisions under heavy fire that can separate life from death. At the same time, they are bombarded with information from eyes in the sky and the crew back at base.

Artificial intelligence has the power to make complex decisions for them and has already been employed in military strategies. An AR system powered with AI can run millions of simulations, compare current situations to archives, and determine the best course of action before any bullets are fired. Soldiers of the future will have a video game-style field of view with all the assists turned on.

AR can also highlight clear and present danger, which allows the soldiers on the ground to make informed choices and deal with the threats in order. ARC4 is a small step towards this future vision. A system connected to the might of military drones, satellites and more, the battleground of the future will be a much more high-tech environment.

The military technology will trickle down to civilian use cases. The same AR system that helps Marines win the day can save cyclists from oncoming cars. Garmin’s radar-equipped Varia Vision is an idea of what’s to come, but future models will see around corners and give you advance warning before you’re even in danger.

4. Customized simulation & training

The combination of artificial intelligence and VR/AR is a powerful tool for educating the next generation of surgeons, pilots and even youngsters in school.

Doctors already get countless hours of virtual surgery time so they can encounter every complication before they hit the wards. Emergency medical teams (EMTs) deal with car crashes and natural disasters, firemen navigate the worst blazes, and pilots experience how engines go down — all in virtual worlds without a single life at stake.

AI can improve simulated training by incorporating more data points, comparing and contrasting different techniques, and personalizing the education. The system will act more like a customizable trainer than a static simulator. Children could get their own personal tour guide through ancient Rome and the Amazon jungle. They’ll get to ask questions, see the world in action, and experience an interactive education that books just cannot deliver.

With a simple set of sensors and a headset that monitors every move, we should be able to learn everything from the perfect golf swing to the Chinese language. Virtually anybody should be able to access world class coaching at any academic or sporting discipline. The combination of AI with VR/AR has the potential to democratize education and give every student a chance to learn.

5. Truly Social Media

One day we will look back and laugh at Facebook’s measly chat window as we relax over a virtual coffee with a friend who is halfway around the world. In the near future, we can meet our friends in both real or virtual environments, from a pool hall to the Taj Mahal, and interact as if they’re in the same room. The processing power required for this is mind-boggling, but also within our grasp thanks to AI.

Facebook, which invested heavily in VR with the purchase of Oculus Rift, is already using generative models from deep learning to automatically design believable character avatars based on your photos.

6. Character Modeling

Right now, we employ two methods for animating characters: motion capture and manual CG work. Handcrafted animations are laborious, while motion capture is limited to the physical capabilities of the actor being modeled. change thanks to AI methods, such as learning by demonstration, self-teaching AI, and phase-functioned neural networks.

Motion capture involves painstakingly recording a vast array of movements that are essentially repeated over and over. New systems like phase-functioned neural networks, developed by the University of Edinburgh, uses machine learning to combine a vast library of stored movements and map them onto new characters.

The team applied the neural network to a character it called The Orange Duck and the results are remarkable.

This will open up a new world of realistic animation in video games, cartoons, and Virtual Reality environments. A relatively simple session of motion capture can turn into a full range of movement with the help of a neural network. It means realistic characters can, theoretically, be created on the fly.

Not everyone is happy with AI-driven animation. Studio Ghibli founder Hayao Miyazaki notoriously reflected that AI represented the “end of times” when shown automated character models from Japanese company DWANGO. Perhaps you shouldn’t be surprised it upsets a man who has spent his entire life drawing with AI that replaces drawing.

7. Conversational Non-Player Characters (NPCs)

Auxiliary, non-playable characters (NPCs) in video games are notorious for behaving in odd ways. They act nonchalant when crimes are committed in front of them or when strangers barge into their homes. Their conversation is stilted and they just don’t adapt to circumstances.

Even in conversation-driven games like Mass Effect, NPCs are an afterthought. The console only has so much processing power to offer, so NPCs often become part of the background. But with AI to power their realism, NPCs could adapt to events and even participate in a proper conversation. Game consoles, especially for VR, could carry neuromorphic chips that send AI-related tasks to cloud servers, so even the sideline characters can evolve based on the input of other players.

8. Rendering Optimization

One of the biggest challenges in VR / AR is rendering realistic graphics with today’s consumer hardware. Too much complexity leads to pixelated images and lag. This leads to headaches for VR headset wearers. The result is that most VR experiences are simplistic and lacking in convincing detail.

The application of AI to game rendering is so obvious that Nvidia even offers formal courses teaching 3D and graphics artists how to apply deep learning techniques to tasks like super resolution, photo to texture mapping, and texture multiplication. In VR, machine learning can be used for selective rendering, where only the portions of a scene where a viewer is looking are dynamically generated in full visual fidelity, saving on computing costs. Images can also be more intelligently compressed with AI techniques, enabling faster transmission over wireless connections without a discernible loss in quality.

High cost barriers and lagging hardware have caused VR/AR to become overhyped in the past years. By using AI to overcome technical barriers and improve realism, you can now examine the intricate and alien landscape of Mars to within 30 cm of the true topology. Given that NASA’s acceptance rate for astronauts is less than 0.08%, VR — powered by AI — is the closest you’ll probably get to a space adventure.

_Love what you read? You can find more stories like this one at www.topbots.com. Or join the TOPBOTS community to get the best bot news & exclusive industry content._

With an acceptance rate of 6%, Harvard University is one of toughest universities to get into.

But now, with the magic of virtual reality, you can pretend to be a student in Harvard’s largest on-campus class. You sit front and center, able to look around at the students in the auditorium around you.

This is CS50

CS50 is Harvard University’s “introduction to the intellectual enterprises of computer science and the art of programming.” The same class is also offered on edX (know as CS50x), where it’s edX’s largest class. Since the course debuted in 2012, more than a million students have enrolled.

It’s also one of Class Central’s Top 50 MOOC of All Time. And one of our users wrote an in-depth review of CS50x here.

The course is taught every fall by David Malan. It sometimes features guest lectures from tech luminaries like Facebook founder Mark Zuckerberg, and the previous CEO of Microsoft Steve Ballmer. Currently this class is also being taught in parallel at Yale.

Every time the course is taught, the lectures are recorded and put online. In previous years they were also streamed live directly from Harvard’s Sanders Theatre.

This is CS50 VR

Nokia Ozo Virtual Reality camera

But this year, the CS50 team is trying something different. They had previously announced that the class would be shot in virtual reality.

The lectures are being shot in stereoscopic 360º on Nokia OZO VR camera. Each camera costs $45,000.

The first lecture videos in VR finally went live a few hours ago. Now, anyone in the world can get a front row seat to probably one of the largest classrooms in the world.

You can watch the videos are on their YouTube channel and on their Facebook page. For best experience, watch it using YouTube/Facebook app via a Google Cardboard, Samsung Gear VR, Oculus Rift, or HTC Vive.

You can still watch it, if you don’t have the appropriate hardware. Just click and drag the video below to look around. Here’s the first lecture, to give you a taste:

I’m the founder and CEO of Class Central, the most popular search engine for online courses. You can see which courses he is taking here: class-central.com/@dhawal

Stay up to date with the lastest and greatest in online education by signing up for MOOCWatch, a monthly monthly digest of everything related to MOOCs: https://www.class-central.com/moocwatch

Will Patera and Moritz Kassner are the founders of Pupil — a mobile eye tracking platform, which they are building working closely with an open source community. Find out more about their exciting project & feel free to drop by the Pupil Labs community channel on Gitter.

Tell us about a little bit about yourself.

Will Patera and Moritz Kassner founded Pupil Labs in 2013. Andreas Bulling joined as the scientific adviser and partner in 2014. We develop open source software and build accessible hardware for mobile eye tracking and eye tracking integrations for VR/AR. We have backgrounds in computer science, industrial design, and architecture.

What is Pupil? How did it all begin?

Pupil is an open source eye tracking platform. Pupil started as a part of our joint Master’s thesis at MIT. We wanted to be able to record and visualize what someone was looking at (gaze) while navigating through buildings. At the time, there were no accessible or open source mobile tracking tools. So, we decided to build our own hardware and software. We developed a mobile eye tracking headset with our own cameras and proof of concept along with capture and visualization software.

After finishing our thesis, we released the code (on Google Code — yes Google Code was still a thing in 2012) and published DIY instructions, and bill of materials, so that others could build their own mobile eye tracking headset and conduct research.

The community, developed organically around the initial open source codebase and wiki. We started receiving requests from the community to purchase eye tracking hardware, and in late 2013 we formed the company Pupil Labs. The company provides a way for us continue working on Pupil and pursuing the vision of the platform.

Pupil binocular mobile eye tracking headset.

Eye tracking is exciting because it intersects with a diverse range of research fields, application areas, and development topics. It’s fun to meet new people and learn about new areas where eye tracking is employed. Research areas include (but are not limited to): psychology, cognitive science, neuroscience, UI/UX, artificial intelligence, music, education, assistive technology, optics, ophthalmology, optometry, transportation and wayfinding, urban design, architecture, computer graphics, VR, AR, and more. Here are some academic publications that cite Pupil.

On the development side it is equally diverse. Problems/projects address: ergonomics and physiology, optics, electrical engineering, computer vision, pupil detection algorithms and eye movement classification, visualization and more.

What common goals do you have as a community?

We share a common desire to make eye tracking accessible and reducing the barriers to obtaining robust eye tracking data for research and novel interfaces.

What issues related to your project are you most excited about these days?

We are really excited about our latest project called hmd-eyes — a community driven platform for open source eye tracking tools in VR, AR, and MR. We outline the goals of the project in this blog post. The github repo for this project is somewhat bare-bones right now, but more coming soon. We also have a Gitter channel for this project at hmd-eyes.

How do you expect VR to evolve in the next coming years? What will be the most interesting use cases in your opinion?

There is certainly a lot of speculation and excitement surrounding VR and AR right now, and a lot of interest for eye tracking in VR/AR.

We think that it is only a matter of time — maybe only one to two more release cycles — until eye tracking will be integrated into consumerware HMDs (VR, AR, MR). This is exciting. But, will these consumerware platforms be accessible and open, or a black box? Will researchers and developers be able to access the fundamental pupil detection and mapping algorithms or develop their own methods? Will the work that you do within one SDK be portable or locked into one system?

Pupil Labs recently released an eye tracking addon for the HTC Vive AR Headset.

We develop eye tracking hardware add-ons for VR and AR headsets, and have initiated a project called hmd-eyes to create open source building blocks for eye tracking in HMDs. Our goal is to work with and support a community of researchers and developers in creating reusable open-source software modules for eye tracking in HMDs. We know that many developers have already thought of or implemented eye tracking software for HMDs and instead of re-inventing wheels, we should form a community to share knowledge and tools.

Foveated rendering has gotten a lot of press. It is exciting, but maybe just the tip of the application-iceberg. There is so much that can be done within AR/VR that requires robust eye tracking data. There is interesting research into cognition aware computing using eye tracking.

What are the main issues discussed in the Pupil channel on Gitter?

Before Gitter, our only way to get in touch with our community was through Github and a Google Group (mailing list). We still use the Google Group, but have found that Gitter is really nice for quick feedback and sharing within the community.

A lot of the questions from our community are about setup and workflow. When both software and hardware (especially something you wear) is involved, workflows can quickly get complicated. It’s nice to be able to give and receive rapid feedback on Gitter. This helps the community to improve documentation and understand critical areas where improvements can be made.

We also maintain a wiki for Pupil. The wiki is open for anyone in the community to edit and can serve as both reference for developers and a “getting started” for new members.

The Pupil channel is also used to share development progress and post videos to project demos and research papers. It’s always fun to see video demos!

What are the factors that you have taken into account while creating and maintaining the community? What factors contribute to the success of your community?

In the beginning it required a lot of fostering. Most of knowledge was with the maintainers, and we had to answer almost all questions. Now the community has matured to a level that it is self perpetuating and there is more active exchange among its users.

How do you encourage participants’ commitment and contribution to the community?

It is our duty as maintainers to make sure the community is welcoming and supportive, and try our best to give quick feedback. That being said, we feel like we are still learning on how to best encourage participation in the community.

It’s great when contributors step forward and answer questions, share their work in progress, improve documentation, open issues, and make pull requests.

We really envision the community to be driven by others as much as us. With members taking ownership over certain areas of expertise.

Based on your experience, do you feel that the open source communities have changed and evolved over the past years? If so, how?

The great thing about community driven open source work is that, in the best case, it is always changing and evolving. There are certainly more tools now than ever before, that enable people to work together from around the world and to facilitate getting work done online. As communities evolve, so do the enabling technologies.

What advice would you give to someone who wants to start an online open source community?

Just start your project publicly. Initially you may be writing to yourself. Do your best to write docs, updates and code that are concise and a pleasure to read.

Anything else you want to add?

We ❤ Gitter. Its great to connect with community members and interested newcomers. We also rely on it heavily for internal communication; keeping our distributed team connected and coordinated.