The freeCodeCamp.org YouTube channel just launched a comprehensive new course that will teach you how to implement Augmented Reality by creating four exciting projects. This course, developed by Priyanshu Bhattacharjee, is aimed at both beginners and more experienced developers who are interested in expanding their knowledge of Augmented Reality (AR).

About the Course

This course is focused on Markerless Augmented Reality, a type of AR where users can place various digital objects in space without the need for a physical marker. You will be guided through building four distinct AR projects, each with a different theme: Engineering, Education, E-commerce, and Creativity. Each project is built using the Unity Engine and AR Foundation, setting a solid foundation for your future AR endeavors.

To ensure you receive a rounded experience, each project is designed to expose you to different uses of AR. Whether it's learning about our solar system, simulating a jet turbine, creating an AR garden, or visualizing how different furniture could fit into your home, these real-world applications give you a clear view of the potential of AR.

What Will You Learn

The course is structured into five main sections:

Introduction to the Course & Project Demonstration: In this initial section, Priyanshu introduces you to the course, sets expectations, and provides a sneak peek into the projects you will be working on.

Building Planets AR with AR Foundation: The first project is an educational AR application named PlanetsAR. You will learn to develop an AR application that helps users learn about the planets in our solar system in an immersive and interactive way.

Building Jet Turbine Simulation with AR Foundation: The second project delves into the world of engineering. You'll create an AR simulation of a jet turbine, allowing users to understand its operation and functionality.

Building GardeniAR with AR Foundation: Here, you'll focus on the creative aspects of AR by building GardeniAR, an application that lets users plant a virtual garden in their own room.

Building Furnish AR with AR Foundation: The final project centers around e-commerce and product visualization. Inspired by AR solutions used by Amazon and Flipkart, you'll develop FurnishAR, an application that lets users visualize how different furniture fits into their homes, an innovative solution to reduce product returns.

Who is this Course for?

This course is designed to cater to anyone who wants to learn about Augmented Reality. Whether you are a beginner taking the first steps into AR, or a seasoned developer interested in expanding your skillset, this course offers valuable learning opportunities.

Final Thoughts

freeCodeCamp.org is devoted to democratizing education and providing accessible, quality learning resources to all. This new Augmented Reality course aligns perfectly with this ethos.

Watch the full course on the freeCodeCamp.org YouTube channel (2-hour watch).

But that knowledge isn't half as important as understanding how you can tame the data beast and then wring out every drop of its value.

Naturally, creative and resourceful people in one place or another are always finding new processes and applications that'll make better use of their data. So we'll explore some of today's dominant data utilization trends and leave predicting tomorrow's technology for the pundits.

This chapter was taken from my book, Keeping Up: Backgrounders to All the Big Technology Trends You Can't Afford to Ignore. If you'd prefer to watch this chapter as a video, feel free to follow along here:

What Exactly Is Data?

Before priming our understanding of what's available to help us work productively with data, it's a good idea to first define exactly what data is.

Sure, we saw plenty of great individual examples in my previous article on Managing Data Storage, including the huge volumes of performance and status information produced by digital components of complex systems like cars. But that's not the same as a definition.

So then let's define it. Data, for our purposes, is any digital information that is generated by, or used for, your compute operations. That will include log messages produced by a compute device, weather information relayed through remote sensors to a server, digital imaging files (like CT, tomography and ultrasound scans), and the numbers you enter into a spreadsheet. And everything in between.

Which brings us to big data – another one of those buzz phrases that get thrown around a lot, often without accompanying context or clarity.

On a first glance, you'd probably figure that big data describes data sets that come in volumes higher than traditional data software and hardware solutions are capable of handling.

Indeed, your way of figuring it would be largely correct. Although we could add one or two secondary characteristics. The complexity of a data set, for instance, is also something that could force you to consider big data solutions. And sets of data that must be consumed and analyzed while in motion (streaming data) are also often better addressed using big data tools.

It's worth mentioning that big data workloads will often seek to solve large scale predictive analytics or behavior analytics problems. Such problems are common within domains like healthcare, Internet of Things (IoC), and information technology.

With that out of the way, we can now get to work understanding how – and why – all that data is being used.

Virtual Reality and Augmented Reality

What? Plain old reality suddenly not good enough for you?

Well yes, in some cases, plain old reality really isn't good enough. At least if you have a strong interest in engaging in experiences that are difficult or impossible under normal conditions.

A virtual reality (VR) device lets you immerse yourself in a non-existent environment.

The most common examples of currently available VR technology feature some kind of headset that projects visual images in front of your eyes while tracking your head movements and, in some cases, the way you're moving other parts of your body. The visual images will adapt to your physical movements, giving you the sensation that you're actually within and manipulating the virtual projection.

VR has potential applications in educational, healthcare, research, and military fields. The ability to simulate distant, prohibitively expensive, or theoretical environments can make training more realistic and immediate than would be otherwise possible.

VR technologies have been arriving – and then disappearing – for decades already. For the most part, they've focused on providing immersive gaming and entertainment environments. But they've never really caught on in a big way beyond the niche product level.

This might be partly due to high prices, and because some people experienced forms of motion sickness and disorientation.

But maybe – just maybe – (insert the current year here) will finally be the year VR hits the big time.

But where VR can leverage data in a really meaningful way is when, rather than blocking out your physical surroundings, the virtual environment is overlaid on top of your actual field of vision.

Imagine you're a technician working on electrical switching hardware under a sidewalk. You're wearing goggles that let you see the equipment in front of you, but that also project text and icons clearly identifying labels for each part and that show you where a replacement part should go and how it's connected. This is augmented reality.

I'm sure you can easily imagine how powerful this kind of dynamic display could be in the right conditions.

Surgeons are able to access a patient's history or even consult relevant medical literature without having to divert their eyes from the operation. Military pilots can similarly enjoy "heads up" displays that show them timely status reports describing their own aircraft and broader air traffic conditions without distraction.

Artificial Intelligence and Machine Learning

As a rule, computers are even better at performing dull, repetitive tasks over and over again than bored teenagers pretending to do homework. And they make less noise in the process.

The trick with computers is to cleverly string lots of dull, repetitive tasks together so that they can approximate intelligent and useful behavior.

The prize at the end of that road is called automation. Or, in other words, a state where computers can be confidently left alone to perform complex and useful tasks without supervision.

In many ways, we've been living in an age of sophisticated computer automation for decades. Domains as diverse as security monitoring, urban traffic control, book manufacturing, and heavy industry are already being handled with little or no human supervision.

But artificial intelligence (AI) seeks to go beyond relatively simple repetition to train computers to think for themselves – and thereby efficiently solve far more difficult problems.

Great idea. Somewhat harder to achieve in the real world.

What can AI actually do?

Understanding how effective AI can be will depend on what you expect it to do. Can you design software to search for and flag a handful of suspicious financial transactions from among the millions of credit card transactions a large bank processes? Yes. (Although I'm not quite sure that's truly AI at work and not just automation.)

Can you deploy "intelligent" chatbots on your website to help customers solve their problems without needing actual (and expensive) human interaction? Yes. In fact, I just had a surprisingly effective conversation with my mobile phone carrier's chatbot that did quickly solve my problem.

Can the first stages of a rocket you've just used to launch a payload into space use AI to guide it to a safe landing on a moving platform in the middle of the ocean? If you'd ask me, I'd say it's impossible. But SpaceX went ahead anyway and did it multiple times. Good thing they didn't ask me.

But can AI reliably make strategic decisions that intelligently account for all the many moving parts and complexity that exist in your industry? Can an AI-powered machine pass the Turing test (where a human evaluator is unable to be sure whether the machine is also human)? Perhaps not just yet. And perhaps never.

One tool used in many AI processes is the neural network. The original neural network consists of the many neurons that carry information about the state of a biological environment to the brain.

Artificial and virtual neural networks are systems for assessing, processing, and responding to the large physical or virtual data sets that feed AI-controlled systems.

Such data can come from cameras or other physical sensors, or from multiple data sources. The processed data can sometimes be used for predictive modeling, where the likelihood of future outcomes are compared.

Exciting stuff, to be sure. But the tools used for some of the most significant accomplishments attributed to artificial intelligence aren't actually artificial. Nor did they necessarily require all that much intelligence.

For example, Amazon Mechanical Turk (MTurk) is a service that connects client companies with remote freelancing "human intelligence" workers. The workers will, for what usually amounts to dreadfully low pay, perform "mechanical" tasks like labelling the content of hundreds or thousands of images. The labelling will cover areas like "is the subject a male or female?" or "is the subject a car or a bus?"

It could be that, over time, services like Mechanical Turk will become less important as improving AI methodologies might one day completely replace the human element for this kind of work. But in the meantime, MTurk and its competitors are still steaming along at full speed, churning out millions of units of "artificial" artificial intelligence.

One methodology that can help reduce reliance on human intervention is machine learning (ML).

How can machine learning help?

ML works by leveraging various kinds of manual assistance to help achieve greater task automation. An ML system can hopefully "learn" how to manage our tasks by being exposed to existing training data. Only once the system has demonstrated sufficient skill at solving the problems you have for it, will it be let loose on "real world" data.

These are some common approaches to training your ML system:

• Supervised learning lets the ML software read data sets that include both "problems" (images, for example) and their "solutions" (full labels). By seeing enough of the provided examples, the system should be able to apply its experience to similar problems that arrive without solutions.
• Unsupervised learning simply throws raw data without any associated solutions at the system. The goal is for the software to recognize enough patterns in the data to allow it to solve the problems on its own.
• Reinforcement learning learns from interactions with its environment. Ideally, the software recognizes and understands positive results and evolves its methodology to reliably and consistently produce similar results.
• Deep learning algorithms apply multiple layers of analysis to transform the raw target data. The full, multi-layer process in deep learning is known as the substantial credit assignment path (CAP).

AI in general, and ML in particular, are effective at building tools for tasks like autonomous driving, drug discovery, email filtering, and speech recognition, and for deriving sentiment analysis from massive data sets made up of human communications.

What AI and ML share in common with all the other technologies like virtual reality and augmented reality that we've discussed here – and in that other "How to Manage Data Storage" article – is the need to control and make better sense of the endless streams of information our digital products keep generating. The better we get at this kind of control, the more value we'll get from our data.

YouTube videos of all ten chapters from this book are available here. Lots more tech goodness - in the form of books, courses, and articles - can be had here. And consider taking my AWS, security, and container technology courses here.

We just published a 12-hour course on the freeCodeCamp.org YouTube channel that will teach you about augmented reality from a theoretical and practical perspective.

Priyanshu Bhattaharjee developed this course. He has many years experience in AR development and has taught classes about AR at multiple colleges.

In this course you will learn to develop AR mobile applications and AR filters for Instagram and Facebook from scratch. You will also learn about concepts such as XR Fundamentals, Unity Engine, C#, Markerbased AR Development, and AR Filter development.

Here is the full syllabus for the course:

Module 1: Introduction To Extended Reality

• Augmented Reality
• Marker Based AR
• Marker Less AR
• Virtual Reality
• Mixed Reality
• XR Market Size

Module 2: Introduction to Unity Engine

• Introduction & Installation
• Navigating through interface
• Game Objects
• Materials
• Scenes
• Project Structure
• Creating our first C# Script
• Modifying Components
• Prefabs
• Physics and Collision
• Audio
• Designing User Interfaces

Module 3: Introduction to C# Scripting

• Introduction
• Comments, Variables and Data Types
• Operators
• Conditionals
• If Else, Else If
• Inline if
• Switch Case
• Loops
• For Loop
• While Loop
• Do While Loop
• Arrays
• Functions
• Classes and Objects

Module 4: Marker Based Augmented Reality

• Introduction and Architecture
• Software Development Kits
• Vuforia Engine
• Installation and Setting Up Vuforia Engine
• Free Resources for developing AR Applications
• Image Targets and Image Tracking
• Simultanous Image Tracking
• Virtual Buttons
• Video Playback

Module 5: Projects

• AR Car Customizer
• AR Business Card
• AR Encyclopedia

Module 6: Bonus - AR Filters with Spark AR

• Introduction to Spark AR
• Interface
• Face Tracking
• Face Mesh
• Head Occluder

Module 7: Bonus - Projects

• 3D Face Filter

Watch the full course below or on the freeCodeCamp.org YouTube channel (12-hour watch).

ARKit might seem intimidating but it’s not so bad if you already have some basic experience building iOS apps.

I’m a learn-by-doing type, so I’ve been playing around with ARKit, building basic apps to get familiar with it. In this post, I’ll review what I’ve learned creating a simple face tracking app.

I’ll do this in 3 parts:

1. Initial Setup → First things first, get Camera permissions and make sure the device can use ARKit.
2. Smile tracking → Start tracking smiles with ARKit. This is probably what you’re here for.
3. User Interface → Add the UI for our app that will react to smiles.

As of this writing, the Xcode simulator does not support the front facing camera so you will need a real device to run the app. Your device will also need to have a TrueDepth camera (iPhone X or newer should be fine).

Finally, for my fellow members of the Copy Paste Club, all the code is available on Github.

Initial Setup

Start by opening up Xcode and creating a new project named “SmileTracker” (or whatever name you prefer).

Before we can get into face tracking, we’ll need to do two things:

1. Make sure your device supports ARKit
2. Get permission to access your device’s camera

In your new project, open ViewController.swift. Near the top of the file, underneath import UIKit, add the line: import ARKit. This will let us access all the goodies that Apple has provided us to make face tracking super easy.

Now add the following code inside of viewDidLoad:

guard ARFaceTrackingConfiguration.isSupported else {
    fatalError("Device does not support face tracking")
}

ARFaceTrackingConfiguration.isSupported is a boolean that will be true if the device running the app can support face tracking and false if not. In this case, if the device can’t support face tracking, we’ll crash the app with a fatal error.

Next, let’s get permission to use the camera. Add the following in viewDidLoad below our guard statement:

AVCaptureDevice.requestAccess(for: AVMediaType.video) { granted in
   if (granted) {
      Dispatch.main.sync {
          // We're going to implement this function in a minute
          self.setupSmileTracker()      
      }
   } else {      
      fatalError("User did not grant camera permission!")   
   }
}

Here we’re asking the device to request camera permissions. If the user grants permissions, we’ll run the function that will setup our smile tracking (don’t worry about the error, we’ll be implementing this function in a moment).

We wrap the function in Dispatch.main.sync because we’ll be adding UI elements in this function, which can only be done on the main thread.

We’ll also need to add a Camera Usage Description to our Info.plist. Open Info.plist and add a new row (you can do this by highlighting the last row and hitting enter).

In the row you just created, add Privacy — Camera Usage Description to the Key column and make sure the Type column is set to string. You can leave the Value column blank or add a message to explain how you will use the camera to the user.

Your Info.plist should now look something like this:

If you’d like to test your app so far, you can comment out the line where we call setupSmileTracker(). Just remember to uncomment it later.

If you run your app now, you should see a popup asking you to enable camera permissions. If you say no you’ll have to go to application settings to enable those permissions in order for the app to run.

If the app crashes, check the console for one of our two error messages to see what went wrong.

Smile tracking

Open ViewController.swift and add the following variable to the top of ViewController:

class ViewController: UIViewController {   
   let sceneView = ARSCNView()

   override func viewDidLoad() {...}
}

ARSCNView comes equipped with an ARSession that your iPhone uses to coordinate AR experiences. We’ll be using sceneView’s ARSession to analyze our user’s face through the front facing camera.

Add this function to your file underneath viewDidLoad:

func setupSmileTracker() {   
   let configuration = ARFaceTrackingConfiguration()   
   sceneView.session.run(configuration)   
   sceneView.delegate = self   
   view.addSubview(sceneView)
}

Here we’ve created a configuration to handle face tracking and used it to run our sceneView’s ARSession.

Then we set sceneView's delegate to self and add it to our view.

Xcode will tell you that there is a problem since ViewController does not conform to ARSCNViewDelegate. Go to where ViewController is declared near the top of the file and change the line to the following:

class ViewController: ViewController, ARSCNViewDelegate {   
   ...
}

Now add thisARSCNViewDelegate function in your ViewController class setupSmileTracker:

func renderer(_renderer: SCNSceneRenderer, didUpdate node: SCNNode, for anchor: ARAnchor) {
   ...
}

renderer will run every time our scene updates and provides us with the ARAnchor that corresponds to the user’s face.

To make it easier to create face tracking experiences, Apple automatically creates an ARFaceAnchor and adds it to our session when we use an ARFacetrackingConfiguration to run it. This ARFaceAnchor is then passed into renderer as an ARAnchor.

Add the following code to renderer:

func renderer(_renderer: SCNSceneRenderer, didUpdate node: SCNNode, for anchor: ARAnchor) {   
   // 1      
   guard let faceAnchor = anchor as? ARFaceAnchor else { return }

   // 2   
   let leftSmileValue = faceAnchor.blendshapes[.mouthSmileLeft] as! CGFloat
   let rightSmileValue = faceAnchor.blendShapes[.mouthSmileRight] as! CGFloat

   // 3
   print(leftSmileValue, rightSmileValue)
}

There’s a lot going on inside this function so I’ve numbered the steps (Ray Wenderlich style).

In step 1 we convert the ARAnchor into an ARFaceAnchor and assign it to the faceAnchor variable.

ARFaceAnchor contains information about the current position and orientation, topology, and facial expression of the face we’re tracking.

ARFaceAnchor stores information about facial expressions in its variable blendShapes. blendShapes is a dictionary that stores coefficients corresponding to various facial features. If you’re interested, I suggest you check out the full list of facial features in Apple’s documentation. (Hint: if you want to add frown tracking you’ll find a way to do it in here.)

In step 2, we use faceAnchor.blendShapes to get the a CGFloat that corresponds to how much the left and right sides of the user’s mouth is smiling by using the keys mouthSmileLeft and mouthSmileRight.

Finally, step 3 just prints out the two values so you can make sure it’s working properly ?.

At this point you should have an app that:

• Gets camera and face tracking permissions from the user
• Uses ARKit to track the users facial expressions
• Prints how much the user is smiling on the left and right sides of their mouth to the console

We’ve made a lot of progress, so let’s take a second to make sure everything is running properly.

When you run the app for the first time, you should be asked if you will grant camera permissions. Make sure to say yes.

You’ll then be sent to a blank screen, but you should start seeing CGFloat values being printed to the console (there may be a short delay before you see them).

When you smile at your phone you should notice the values being printed going up. The more you smile the higher the numbers go.

If it’s working properly, congratulations ?! If you’re running into an error, double check to make sure your device supports face tracking and you have camera permissions turned on. If you’ve been following this writeup from the beginning the console will print errors in both those cases.

User interface

So we’re tracking faces, now let’s build the UI to react to smiles.

First add a new UILabel called smileLabel to the top of the file, just below sceneView.

class ViewController: UIViewController {   
   let sceneView = ARSCNView()      
   let smileLabel = UILabel()

   ...
}

This will be the view that reacts to the user’s facial expressions.

Add the following code at the bottom of your setupSmileTracker function:

smileLabel.text = "?"smileLabel.font = UIFont.systemFont(ofSize: 150) 

view.addSubview(smileLabel)

// Set constraints
smileLabel.translatesAutoresizingMaskIntoConstraints = false
smileLabel.centerXAnchor.constraint(equalTo: view.centerXAnchor).isActive = true
smileLabel.centerYAnchor.constraint(equalTo: view.centerYAnchor).isActive = true

Here, we’re adding the basic UI properties to our smileLabel and setting its constraints so it is in the middle of the screen. Now when you run the app, you should see a giant ? emoji in the middle.

Once you see the emoji appear, add the following function to your ViewController:

func handleSmile(leftValue: CGFloat, rightValue: CGFloat) {
   let smileValue = (leftValue + rightValue)/2.0
   switch smileValue {      
         case _ where smileValue > 0.5:         
           smileLabel.text = "?"      
      case _ where smileValue > 0.2:         
         smileLabel.text = "?"      
      default:         
         smileLabel.text = "?"      
   }
}

This function will change the emoji in our smileLabel depending on how much the user is smiling into the camera. We calculate the smileValue by taking the average of the left and right smile values given to us by our ARFaceAnchor (very scientific, I know).

Plug that value into the switch statement, and the more the user smiles the happier our emoji gets.

Finally, go back to our renderer function and add this to the bottom to plug in our left and right smile values into handleSmile:

DispatchQueue.main.async {   
   self.handleSmile(leftValue: leftSmileValue, rightValue: rightSmileValue)
}

Again, we use DispatchQueue because we are making changes to the UI, which must be done on the main thread.

When you run the app you should now see the emoji change depending on how much you smile at it.

In the gif below I’ve added my face so that you can see it working with the camera output along with the emoji.

I added the camera output to show how it works

Your app won’t have the camera output, but you can add it by adding our ARSCNView, sceneView, to the superview and giving it dimensions.

Wrapping up

I hope this post was helpful for you to get started creating apps with ARKit.

If you want to extend this app further, check out the list I mentioned above with all the other facial features you can track. I left a hint for how to extend this to check for frowns as well.

Come back and comment with any cool projects you create on your own, I’m still getting my feet wet with this stuff so it’d be excited to see more complex apps.

I’ve posted all the code for this app up on Github for feedback and questions. Thanks for reading and good luck!

Thanks so much for reading! If you liked this story, follow me on Twitter where I post updates about stories I’m working on and what I’m up to.

If you are a JavaScript coder who is still late to making up a definitive list of resolutions for 2019, let me give you a hand: Start figuring out how to get into Augmented Reality (AR).

The Augmented/Mixed/Virtual Reality (AR/MR/VR) combo has enjoyed frenetic growth since 2016, coming from a marginal market value of bit more than $6 Billion to one that might reach the $210 Billion in sales (including hardware) by 2022. Of all, Augmented Reality is the one experiencing steady growth.

At first, a JavaScript (web) developer wanting to get into the AR boat might feel discouraged when finding the usual required skills; and then there is who ask Machine Learning or Internet of Things. However if you are mainly a JavaScript developer, consider yourself blessed: the language is recurrently mentioned as one you should know to enter this sector. The reason? Right now a lot of AR development goes on the web. And this is where JavaScript reigns.

_Mobile and Web were the last getting AR capabilities, and are still developing (extract from a [buildAR presentation](https://www.slideshare.net/robman/web-standards-adoption-in-the-ar-market/6" rel="noopener" target="blank" title="))

Augmented Jobs for the JavaScript-fan — Really?

Maybe not too fast. There are many examples where AR/MR/VR shines on its own, specially in niche markets, but the industry hasn’t completely figured out the full value of the technology for the general consumer. Once that it solved, the industry would be certainly making more AR/MR/VR products, which would translate into more jobs.

For some analysts AR is expected to have the most pervasive impact, in part because it doesn't require specific devices and conditions to be implemented as VR does.

AR has utility for almost everything, overlaying useful and relevant information on the world around you. AR can be pervasive in a way that VR cannot.

• David McQueen -Strategy Analytics- from an interview on Twice

It rests on the industry to find how to make AR a more every day life tech. According to some companies, particularly within the mobile phone realm, exploiting better the AR potential reduces to a well-known rule: SIMPLICITY.

While Unity has become the default path for building AR apps, an increasing number need only a sprinkling of AR.

• from an article by Benjamin Devine, Homestory AR

In many cases, resourcing onto the leading AR tools could be an overkill. Instead, a bunch of good UX-driven features over some 2D/3D assets could be more than enough to make striking products. Something a JavaScript developer regularly does.

It is then possible that any JavaScript developer will be embedding (non)standard AR/VR features as an extension of their traditional duties in the future. And if required, JavaScript is robust enough for more complex tasks. The sky is the limit.

Becoming JavaScript-Augmented

Before starting, I would suggest to have a look at the several AR platforms and standards. The same technical constraints affecting the industry are also reflected in the AR world.

For example, there are several platforms, one for each Big Tech (Google = ARCode, Apple = ARKit, MS = ChakraCore, Facebook = AR Studio, React 360, Mozilla = aframe).

After having a quick look at the options, starting fully JavaScript’ed Augmented Reality projects is relatively easy. You can begin by taking any web/app dev framework like Cordova, Ionic, React Native or Vue Native to embed the AR framework of your choice — and deploy 3D assets on top of the real world.

If what you want is to deploy on the web using mostly marker-based AR, you could use GitHub repos like AR.js (free), argon.js (free but limited) or awe.js (paid PaaS but with an old GitHub repository still available). There are a few tailored ones that are harder for the novice, many of them focused on things like facial/head recognition (such as tracking.js and headtrackr).

Or you can boost your project capabilities if you are able to port available SDKs made by AR-related companies. There are many APIs that render as AR on browser too. For example, Mapbox follows that path and it is developed on JavaScript.

I would suggest you to keep it simple but interactive.

However if your ambitions point to also mastering design and animation in JavaScript, you definitively have to learn at least one 3D Javascript package, and THREE.js the most popular. Wait, though, until you have gained a good base of JavaScript and OpenGl as well as geometry, trigonometry, linear algebra or physics. And don’t expect more help from the existing 3D JS GUIs; in particular, THREE.js has none. Challenging but exciting!

Bonus Example

I wanted to prepare a quick demo just to explore the technology, so I took a nice CodePen and modified it to fit a marker-based web-rendered AR animation ported within a clone of Stemkoski's great work with AR.js.

For you to see the example you need a mobile device with a camera and internet (phone or tablet), and either a printed copy of the marker or another device to show it on screen.

Ready? Now open this link using a browser in your mobile device:

https://evaristoc.github.io/ARexample/

Give authorization to use your camera, and point the camera to a marker like below, either printed or in another screen.

NOTE: works on Android and Chrome — it might not work for other devices and browsers ?.

_The original animation can be found [here](https://codepen.io/rainner/details/LREdXd" rel="noopener" target="blank" title=").

Happy New Year!

I hope you will find this technology as fascinating as I do. If so, don't stay alone: contact us at the freeCodeCamp forum and share your questions and ideas.

And if you liked this article, don't forget to give it a ? and to share it on social media.

Thanks for reading, enjoy AR and Happy Coding!!

In this experiment, I talk about how Augmented Reality with JS can be used to make learning more fun and interactive. The case study will discuss the design process, implementation and feedback from children in the age group 2 to 10 years.

Education and Interactive Alphabets learning using Augmented Reality and Javascript

Augmented Reality (AR) has always attracted me, and in this experiment, I try to create a practical AR application. The use-case we will cover is primary education and we will see how we can make the learning fun and interactive. We will make an app to learn alphabets in three languages primarily: Punjabi, Hindi, and English.

The Javascript Augmented Reality app currently doesn’t have plane detection. For simplicity’s sake we are only superimposing our objects over the viewport with 3d motion tracking.

END GOAL

Below is a demo of our Javascript AR experiment. You can download and play with the app here.

The full code has been open-sourced for learning purposes and is available here.

Alphabets in augmented reality and javascript to make education more fun and engaging

The Design Process

To make the learning fun and effortless, I am relying on the following points:

1. Active participation of the child
2. Child’s physical activity instead of sitting in one place
3. A bit of effort in finding the alphabets.
4. Intuitive UX/UI.

The core theme of the app will be quite similar to the famous Pokemon Go Augmented reality app. Only two main components will be involved: the Camera Viewport and Alphabets.

Alphabet UX for AR Game

Iteration 1

2d Alphabets in English, Hindi and Punjabi for our JS Augmented Reality Game

In our first iteration we have 2d alphabets which we will try to merge in our camera viewport. The idea of the Augmented Reality(AR) app is to have children find these alphabet letters in a room or space around them. The prototype after merging the space with 2d alphabets will look something like this:

AR Motion sensor with 2d object

As you can see above, the immersive experience is missing with a 2d model because the human eye sees things in 3d.

Iteration 2

Lets try to create a 3d model and see if we can improve the experience of our Javascript-based Augmented Reality game:

3D Alphabets in English, Hindi and Punjabi for our AR project

And below is the comparison of the experience of a motion sensor with 2d vs 3d alphabet models. As you can see, 3D naturally gives you a much more immersive experience when it comes to Augmented reality.

2d vs 3d AR motion experience

The AR Implementation process

To implement the above AR concept, I’ll be using the following tools and technologies:

1. Ionic Framework: For building the hybrid app
2. Aframe: For bringing the Virtual reality (VR) and Augmented Reality (AR) experience to our app
3. MagicaVoxel: For creating our 3D models

The basic app building process is very simple. You can follow my earlier post to learn how to go about building and deploying an app using the Ionic framework here.

Once you have followed the above tutorial to create a simple app, it’s time to integrate Aframe to bring our 3D alphabets with motion sensors into our app.

Just load the below Aframe core and Aframe loader libraries in ionic’s project index.html file:

<script src="https://aframe.io/releases/0.8.2/aframe.min.js"></script>

<script src="https://rawgit.com/donmccurdy/aframe-extras/v2.1.1/dist/aframe-extras.loaders.min.js"></script>

With this we are ready to do some AR/VR magic in our Javascript code base.

Now in your home component’s home.html, let’s include our 3D models exported from magicavoxel:

And this should make a 3D scene ready for interaction with all motion sensors ready:

Final 3D Virtual Reality scene ready with 3D alphabets

Adding Augmented Reality Effect

The final part of this experiment is to add the Augmented Reality (AR) feeling in our Javascript-based hybrid app. As already explained, Augmented Reality is when you superimpose 3D models or other objects on top of your camera viewport. So the only thing missing is the camera viewport behind our scene.

To do this, we use the camera preview plugin as explained here. And here is the full gist after integration with the camera preview plugin:

We also need to ensure that our backgrounds are transparent so that the camera preview is visible in mobile. This is very IMPORTANT otherwise you might feel that the plugin is not working. Here is the home.scss file with transparency css enabled:

And this is what it would finally look like:

User reaction to our Augmented reality JS game

The final step to measure the success of your concept is real user validation — in our case, kids :) And below is their live feedback recorded.

It was pretty clear that each one of them enjoyed the game and we got full points on fun part. However, initially I had to tell them to move the phone in space to find the letters. Points lost in terms of intuitiveness :(

Points scored out of 10

User feedback for Augmented Reality JS game

Final Thoughts

Well it was an exciting project and I could see a lot of potential for Augmented Reality in learning and education. Children really like it and it surely adds the missing fun factor to education, especially in our monotonous Education system.

Feel free to comment and share your feedback.

Downloads

The code for this app is available in github. Feel free to play and push new features in it. I’ll be happy to push updates over production.

You can download the app for android here.

This article was originally posted here.

In this tutorial, you’ll learn how to place 3D models in the real world by setting the anchor as a specific scene instead of a regular plane. ARCore by Google lets you augment 2D images which can be recognized by ARCore to then place 3D models over them.

You provide some reference images and ARCore tracking determines where those images are physically located in the environment. Augmented Images are already widely used such as in books, newspapers, magazines etc.

But before you dive deeper into this tutorial, you must check out the following articles as a prerequisite to this one:

• What is ARCore by Google?
• Building an augmented reality app using ARCore.

Once you’re done with these two, you’ll have a basic understanding of some terminology in ARCore and Sceneform such as Scene, Anchor, Node, TransformableNode etc.

What are Augmented Images?

According to the developer docs, Augmented Images in ARCore let you build AR apps that can respond to 2D images, such as posters or product packaging, in the user’s environment. You provide a set of reference images, and ARCore tracking tells you where those images are physically located in an AR session, once they are detected in the camera view.

Basically, using augmented images, you can turn a simple 2D image into an augmented image which can be recognized by your app and be then used to place a 3D model above it.

When you might want to use Augmented Images

Here are some restrictions which you might want to consider before using Augmented Images:

• Your use case must not involve scanning more than 20 images simultaneously (since ARCore can only track up to 20 images at once).
• Size of the physical counterpart in the real world must be more than 15cm X 15cm and flat.
• You don’t want to track moving objects. ARCore cannot track moving images, although it can start tracking once the image stops.
• ARCore uses feature points in the reference image and can store feature point information for up to 1000 images.

Choosing a good reference image

Here are some tips to choose a good reference image to improve detectability by ARCore:

• Augmented Images support PNG, JPEG and JPG formats.
• Detection is based on points of high contrast and so both color and black/white images are detected, regardless of whether a color or black/white reference image is used.
• Image’s resolution must be at least 300 X 300 pixels.
• Using high-res images does not mean improved performance.
• Images with repetitive features such as patterns and polka dots must be avoided.
• Use arcoreimg tool to evaluate how good your reference image is. A score of at least 75 is recommended.

How to use the arcoreimg tool:

• Download the ARCore SDK for Android from this link:
• Extract the zip contents of the zip file anywhere you like.
• Navigate to the extracted folder and go to tools -> arcoreimg -> windows (linux/macos whatever you use)
• Open command prompt at this location.
• Now enter this command:

arcoreimg.exe eval-img --input_image_path=dog.png

Replace dog.png with the complete path to your image.

Getting started with Augmented Images Application

Now that you’ve familiarized yourself with ARCore and Sceneform and have selected a good reference image with a score of 75+, it’s time to start coding the application!!

Create a custom fragment

We will be creating a custom fragment to add to our activity. We need a custom fragment as we will be altering some properties of the default fragment.

Create a class named “CustomArFragment” and extend it from ArFragment. Here is the code for CustomArFragment:

package com.ayusch.augmentedimages;

import android.util.Log;

import com.google.ar.core.Config;import com.google.ar.core.Session;import com.google.ar.sceneform.ux.ArFragment;

public class CustomArFragment extends ArFragment {

    @Override    protected Config getSessionConfiguration(Session session) {        getPlaneDiscoveryController().setInstructionView(null);        Config config = new Config(session);        config.setUpdateMode(Config.UpdateMode.LATEST_CAMERA_IMAGE);        session.configure(config);        getArSceneView().setupSession(session);

        return config;    }

}

First of all, we are setting the plane discovery instruction to null. By doing this, we turn off that hand icon which appears just after the fragment is initialized which instructs the user to move their phone around. We don’t need it any more as we are not detecting random planes but a specific image.

Next, we set the update mode for the session to LATEST_CAMERA_IMAGE. This ensures that your update listener is called whenever the camera frame updates. It configures the behavior of update method.

Setting up the Augmented Images Database

Add your chosen reference image (which you want to detect in the physical world) in the assets folder. If your assets folder doesn’t exist, create one. Now we will be adding augmented images to our database which will then be detected in the real world.

We’ll set up this database as soon as the fragment (scene) is created. Then we check for the success and failure of this call and set the log accordingly. Add the following code to your custom fragment:

if ((((MainActivity) getActivity()).setupAugmentedImagesDb(config, session))) {    Log.d("SetupAugImgDb", "Success");} else {    Log.e("SetupAugImgDb","Faliure setting up db");}

This is what the CustomArFragment would look like:

package com.ayusch.augmentedimages;

import android.util.Log;

import com.google.ar.core.Config;import com.google.ar.core.Session;import com.google.ar.sceneform.ux.ArFragment;

public class CustomArFragment extends ArFragment {

    @Override    protected Config getSessionConfiguration(Session session) {        getPlaneDiscoveryController().setInstructionView(null);        Config config = new Config(session);        config.setUpdateMode(Config.UpdateMode.LATEST_CAMERA_IMAGE);        session.configure(config);        getArSceneView().setupSession(session);

        if ((((MainActivity) getActivity()).setupAugmentedImagesDb(config, session))) {            Log.d("SetupAugImgDb", "Success");        } else {            Log.e("SetupAugImgDb","Faliure setting up db");        }

        return config;    }

}

We will soon be creating the setupAugmentedImagesDb method in the MainActivity. Now with the CustomArFragment created, let’s add it to our activity_main.xml, here’s the code for your activity_main.xml:

<?xml version="1.0" encoding="utf-8"?><android.support.constraint.ConstraintLayout xmlns:android="http://schemas.android.com/apk/res/android"    xmlns:app="http://schemas.android.com/apk/res-auto"    xmlns:tools="http://schemas.android.com/tools"    android:layout_width="match_parent"    android:layout_height="match_parent"    tools:context=".MainActivity">

    <fragment        android:id="@+id/sceneform_fragment"        android:name="com.ayusch.augmentedimages.CustomArFragment"        android:layout_width="match_parent"        android:layout_height="match_parent" />

</android.support.constraint.ConstraintLayout>

Notice that we set the name of this fragment to our CustomArFragment. This is necessary to ensure that the added fragment is our custom fragment. This will ensure that permission handling and session initializations are taken care of.

Adding an image to the Augmented Images Database

Here, we will set up our images database. Find the reference image in the real world and then add a 3D model accordingly.

Let’s start by setting up our database. Create a public function setupAugmentedImagesDb in the MainActivity.java class:

public boolean setupAugmentedImagesDb(Config config, Session session) {    AugmentedImageDatabase augmentedImageDatabase;    Bitmap bitmap = loadAugmentedImage();    if (bitmap == null) {        return false;    }

    augmentedImageDatabase = new AugmentedImageDatabase(session);    augmentedImageDatabase.addImage("tiger", bitmap);    config.setAugmentedImageDatabase(augmentedImageDatabase);    return true;}

private Bitmap loadAugmentedImage() {

try (InputStream is = getAssets().open("blanket.jpeg")) {        return BitmapFactory.decodeStream(is);    } catch (IOException e) {        Log.e("ImageLoad", "IO Exception", e);    }

    return null;}

We also have loadAugmentedImage method which loads the image from the assets folder and returns a bitmap.

In setupAugmentedImagesDb, we first initialize our database for this session and then add an image to this database. We will name our image “tiger”. Then we set the database for this session configuration and return true, indicating that the image is added successfully.

Detecting the reference images in the real world

Now we will start detecting our reference images in the real world. In order to do so, we will add a listener to our scene which will be called every time a frame is created, and that frame will be analyzed to find our reference image.

Add this line in the onCreate method of MainActivity.java:

arFragment.getArSceneView().getScene().addOnUpdateListener(this::onUpdateFrame);

Now add the onUpdateFrame method to MainActivity:

@RequiresApi(api = Build.VERSION_CODES.N)private void onUpdateFrame(FrameTime frameTime) {    Frame frame = arFragment.getArSceneView().getArFrame();

    Collection<AugmentedImage> augmentedImages = frame.getUpdatedTrackables(AugmentedImage.class);    for (AugmentedImage augmentedImage : augmentedImages) {        if (augmentedImage.getTrackingState() == TrackingState.TRACKING) {            if (augmentedImage.getName().equals("tiger") && shouldAddModel) {                placeObject(arFragment, augmentedImage.createAnchor(augmentedImage.getCenterPose()), Uri.parse("Mesh_BengalTiger.sfb"));                shouldAddModel = false;            }        }    }}

In the first line, we get the frame from the scene. A frame can be imagined as a snapshot in the middle of a video. If you are familiar with how video works, you might know that they are a series of still pictures flipped one after the another really fast, giving the impression of the motion picture. We are extracting one of those pictures.

Once we have the frame, we analyze for our reference image. We extract a list of all the items ARCore has tracked using frame.getUpdatedTrackables. This is a collection of all the detected images. We then loop over the collection and check if our image “tiger” is present in the frame.

If we find a match, then we go ahead and place a 3D model over the detected image.

Note: I have added, shouldAddModel to ensure that we add the model only once.

Placing a 3D model over reference image

Now that we have detected our image in the real world, we can start adding 3D models over it. We will copy the placeObject and addNodeToScene methods from our previous project, and add them here.

Although I have previously explained what these methods do line by line, here is an overview:

• PlaceObject: This method is used to build a renderable from the provided Uri. Once the renderable is built, it is passed into addNodeToScene method where the renderable is attached to a node and that node is placed onto the scene.
• AddNodeToScene: This method creates an AnchorNode from the received anchor, creates another node on which the renderable is attached, then adds this node to the AnchorNode and adds the AnchorNode to the scene.

Here is our final MainActivity.java class:

package com.ayusch.augmentedimages;

import android.graphics.Bitmap;import android.graphics.BitmapFactory;import android.net.Uri;import android.os.Build;import android.support.annotation.RequiresApi;import android.support.v7.app.AppCompatActivity;import android.os.Bundle;import android.util.Log;import android.widget.Toast;

import com.google.ar.core.Anchor;import com.google.ar.core.AugmentedImage;import com.google.ar.core.AugmentedImageDatabase;import com.google.ar.core.Config;import com.google.ar.core.Frame;import com.google.ar.core.Session;import com.google.ar.core.TrackingState;import com.google.ar.sceneform.AnchorNode;import com.google.ar.sceneform.FrameTime;import com.google.ar.sceneform.rendering.ModelRenderable;import com.google.ar.sceneform.rendering.Renderable;import com.google.ar.sceneform.ux.ArFragment;import com.google.ar.sceneform.ux.TransformableNode;

import java.io.IOException;import java.io.InputStream;import java.util.Collection;

public class MainActivity extends AppCompatActivity {    ArFragment arFragment;    boolean shouldAddModel = true;

    @Override    protected void onCreate(Bundle savedInstanceState) {        super.onCreate(savedInstanceState);        setContentView(R.layout.activity_main);        arFragment = (CustomArFragment) getSupportFragmentManager().findFragmentById(R.id.sceneform_fragment);        arFragment.getPlaneDiscoveryController().hide();        arFragment.getArSceneView().getScene().addOnUpdateListener(this::onUpdateFrame);    }

    @RequiresApi(api = Build.VERSION_CODES.N)    private void placeObject(ArFragment arFragment, Anchor anchor, Uri uri) {        ModelRenderable.builder()                .setSource(arFragment.getContext(), uri)                .build()                .thenAccept(modelRenderable -> addNodeToScene(arFragment, anchor, modelRenderable))                .exceptionally(throwable -> {                            Toast.makeText(arFragment.getContext(), "Error:" + throwable.getMessage(), Toast.LENGTH_LONG).show();                            return null;                        }

                );    }

    @RequiresApi(api = Build.VERSION_CODES.N)    private void onUpdateFrame(FrameTime frameTime) {        Frame frame = arFragment.getArSceneView().getArFrame();

        Collection<AugmentedImage> augmentedImages = frame.getUpdatedTrackables(AugmentedImage.class);        for (AugmentedImage augmentedImage : augmentedImages) {            if (augmentedImage.getTrackingState() == TrackingState.TRACKING) {                if (augmentedImage.getName().equals("tiger") && shouldAddModel) {                    placeObject(arFragment, augmentedImage.createAnchor(augmentedImage.getCenterPose()), Uri.parse("Mesh_BengalTiger.sfb"));                    shouldAddModel = false;                }            }        }    }

    public boolean setupAugmentedImagesDb(Config config, Session session) {        AugmentedImageDatabase augmentedImageDatabase;        Bitmap bitmap = loadAugmentedImage();        if (bitmap == null) {            return false;        }

        augmentedImageDatabase = new AugmentedImageDatabase(session);        augmentedImageDatabase.addImage("tiger", bitmap);        config.setAugmentedImageDatabase(augmentedImageDatabase);        return true;    }

    private Bitmap loadAugmentedImage() {        try (InputStream is = getAssets().open("blanket.jpeg")) {            return BitmapFactory.decodeStream(is);        } catch (IOException e) {            Log.e("ImageLoad", "IO Exception", e);        }

        return null;    }

    private void addNodeToScene(ArFragment arFragment, Anchor anchor, Renderable renderable) {        AnchorNode anchorNode = new AnchorNode(anchor);        TransformableNode node = new TransformableNode(arFragment.getTransformationSystem());        node.setRenderable(renderable);        node.setParent(anchorNode);        arFragment.getArSceneView().getScene().addChild(anchorNode);        node.select();    }

}

Now run your app. You should see a screen as shown below. Move around your phone a bit over the reference object. ARCore will detect the feature points and as soon as it detects the reference image in the real world, it will add your 3D model onto it.

[caption id=”attachment_1000" align=”aligncenter” width=”1280"]

I used my blanket as a reference

With this, we have created our very first Augmented Images app using ARCore by Google and Sceneform SDK!!

If you want to stay updated with all the latest articles, subscribe to the weekly newsletter by entering your email address in the form on the top right section of this page.

Like what you read? Don’t forget to share this post on Facebook, Whatsapp and LinkedIn.

You can follow me on LinkedIn, Quora, Twitter and Instagram where I answer questions related to Mobile Development, especially Android and Flutter.

I’m really excited about the future of haptics for AR and VR. It feels like the missing link between my HTC Vive and jumping into the OASIS with Parzival and Art3mis. So it’s not surprising that haptics is perhaps the most hotly anticipated tech in the XR community right now. Several companies like Microsoft and HTC, as well as startups like SenseGlove and HaptX, have shown demos of increasingly promising iterations of haptic gloves that I’m itching to try out.

Unfortunately, like most AR developers today, our work at Placenote is focused almost entirely on mobile AR platforms like ARKit and ARCore. Naturally, this got us thinking, “Could haptics do anything for mobile AR?”

Haptics have been an awesome addition to touch screens, from simulating tactile button clicks to silent notifications. But, after some frantic googling we realized that there’s actually been no real discussion about haptics for mobile AR apps so far… CHALLENGE ACCEPTED ??

The challenge of mobile AR

We decided to dig into why haptics hasn’t made it’s way into mobile AR and it wasn’t hard to see why. Mobile AR is by far the least immersive AR medium. The consensus in the community is that it’s just a stop gap to the ultimate AR platform — smart glasses.

But mindset isn’t the only barrier here. We found that the mobile form-factor presents some unique challenges to the AR experience designer:

• unlike headsets, the phone screen is the display as well as the controller
• it’s impossible to bring your hands into the experience since you’re holding the phone.
• we still rely on touch screen interactions that are ambiguous in dimensionality — 2D or 3D touch?

Nevertheless, the reality is that, for the next few years and perhaps more, mobile AR is here to stay. There are a billion mobile devices in consumer pockets right now and only about a handful of AR headsets on their heads. As a developer, distribution for your apps trumps most other factors. In fact, in applications like indoor navigation and gaming, mobile has already proven itself as a viable medium for deploying AR experiences.

This brings us to the topic of haptics for mobile AR. At first, it might seem like there’s no real hope for haptics to enhance mobile AR experiences, but recent studies have actually shown otherwise.

In haptics, less is more

There’s been a myriad of methods conceived to achieve haptic feedback. In general they fall under two broad categories — kinesthetic haptics (force feedback) and cutaneous haptics (skin sensations).

Kinesthetic haptics has widely been considered to be the more realistic haptic technology. It involves physical actuators, either grounded or ungrounded. These push and pull our fingers and other appendages in response to interactions with virtual objects. Intuitively, realistic force-feedback should perform vastly better than plain old vibrations. But a study published in Science Robotics this year titled “The Uncanny Valley of Haptics” has challenged these assumptions.

The researchers found that increasing the realism of haptic sensation doesn’t necessarily increase the quality of the AR experience. It often has a negative impact due to the uncanny valley of realism in simulations. They found that cutaneous haptics, which is essentially a combination of light touches and vibrations, did a lot better in fooling the brain deeper into the illusion. Strange results, but they basically realized that we’ve underestimated how good our brain is at filling the gaps in our sensation of reality.

The situations where our brain steps in to fill the gaps is what I find most interesting about our perception of the sensation of touch. — Justin Brad, CEO of Osso VR

Bringing haptics to mobile AR

Given these findings, why not test what cutaneous haptics can do for mobile AR? After all, haptics on mobile is not just about vibrating ring tones anymore.

Micro-Electro-Mechanical Systems (MEMS) on mobile devices have gotten a lot more sophisticated and capable of some pretty nuanced behaviors. Since the iPhone 7, Apple has upgraded the old basic rumble vibrations to what they now call the Taptic Engine. This is a lot more subtle and consists of seven different types of haptic feedback with varying patterns and strengths.

The haptic feedback modes available are:

• Selection Change
• Impact Light
• Impact Medium
• Impact Heavy
• Notification Success
• Notification Warning
• Notification Failure

The new iOS Taptic Engine (iPhone 7 onwards) has 7 different kinds of Haptic Feedback

To learn more about the iOS feedback generator, check out this Apple documentation. At the end of this article, I will share some code you can use to quickly add these feedback types to your ARKit apps.

We decided to experiment with a number of these haptic feedback modes in our AR apps and I’m really excited to say that the results were a pleasant surprise to our team.The following are some examples of haptic implementations in our mobile AR apps.

Usage examples of haptics in mobile AR

In our experiments so far, we’ve found that haptic feedback for mobile AR works well in five distinct scenarios. Here’s a description of each.

1. Magnetic pointers (i.e. snap to grid)

A pointer locked along a planar surface is a commonly used feature in many ARKit apps, especially in measurement tools like Air Measure and Magic Plan. Since your phone behaves as a controller in mobile AR, the standard UX in measurement apps involves dragging a pointer along a surface to draw lines or polygons to measure things in the real world. Of course, when it comes to line drawing, magnetic pointers that snap to the end points and edges of lines are seen everywhere — from PowerPoint to Photoshop.

We found that subtle haptic feedback indicating a “snap” in pointer position is a great enhancement. It almost feels like your phone, (i.e your controller) is physically moving to snap into place.

I was really happy to see that Apple’s new app “Measure” actually uses haptic feedback in their UX. It’s an amazingly subtle implementation and you can see a GIF of it in action below. An “Impact Medium” is fired when the pointer snaps to the edge of the plane.

Apple’s Measure App

2. Hit testing (feeling real world surfaces)

Another common feature in ARKit apps is the hit-test. This is implemented as a ray-cast from a point on the screen — either a touch point or the center — to a surface in the real word. It is generally used to add a 3D object at the point of contact. A slight haptic sensation can help the user understand that a surface was “hit”. We found two methods that work well here:

Pinning
In this example, a marker is added to the scene at the hit point. An “Impact Light” helps users sense the “pinning” of the marker in 3D space. Of course, the downside to this is you can’t quite sense the “depth” of the hit point — in other words, how far the pin is from the user.

Grazing
An alternative to pinning is the grazing method of hit testing. In this case, a constantly updating marker previews where a marker might be added to a scene. We found that a series of haptic impulses, based on the magnitude of displacement of the preview marker at each frame, gives the sensation of scraping a pointer along a 3D surface and let’s you “feel” a 3D surface.

Here’s a code example of grazing in Unity:

if (distanceChange >= 0.1 && distanceChange < 0.2) 
{
    iOSHapticFeedback.Instance.Trigger(Impact_Light);
}
else if (distanceChange >= 0.2 && distanceChange < 0.4) 
{
    iOSHapticFeedback.Instance.Trigger(Impact_Medium);
}
else if (distanceChange >= 0.4)
{
    iOSHapticFeedback.Instance.Trigger(Impact_Heavy);
}

3. FPS gun recoil or explosions

This is by far the most fun example of haptic feedback. When building a first person shooter in AR, your phone is the display as well as the weapon. A great way to simulate a gun shot is a simple “Impact Heavy”, which produces a single bump or a “Notification Failure”, which creates a double bump that feels a lot like a gun recoil. Of course the example below is a laser weapon but, hey, this isn’t meant to be too realistic remember?

4. Collision with controller tip

In VR apps like Oculus Medium or Tilt Brush, one of the handheld controllers serves as a brush tip that the user moves around to draw in 3D space. I’ve spent hours painting in Tilt Brush and so naturally I have tried really hard to mimic this experience with ARKit.

The trouble is that creating an accurate drawing experience on mobile becomes really difficult. You lose the sense of depth when your phone is both the display and the controller. One of the hardest things in 3D drawing apps on mobile is knowing where your brush tip is relative to the other 3D objects in the scene.

And, again, haptics was the answer. We found that one way to give users a sense of depth is to imagine the brush is actually a cane you can use to hit 3D objects that are already in scene. Providing haptic feedback to let users know whether the brush tip is in contact with any existing objects in the scene lets users accurately pin point their brush in 3D space.

Sensing brush tip collisions

5. Re-localization snap in Persistent AR Apps.

At Placenote , we primarily build Persistent AR, or AR Cloud, apps. The core functionality of these apps is the ability to save AR content permanently in a physical place. Users can load it up in the same location every time.

This behaviour is called the relocalization of a scene.

Localization snapping into place

In order to relocalize an AR scene, a user must first point their phone’s camera to the real world, and then wait until the camera detects its location.

With Placenote, relocalization happens almost instantaneously but it all happens internally. Hence we need to design a way to notify the user of a successful relocalization. The visual cues might be enough, as seen in the GIF above. But a more subtle indication is to provide a haptic “Impact Light” to suggest that you have snapped into place in the real world.

How to add haptics to your ARKit project

If you’re working with Swift for Native iOS ARKit development, check out this tutorial on implementing haptic feedback in Native apps.

If you’re working with Unity, my favorite package so far is the iOS Haptic Feedback Package on the Unity Asset Store. It’s $5 but well worth it because Unity’s built in function Handheld.Vibrate() doesn’t actually expose the new iOS Taptic Engine functions!

The iOS Haptic Feedback Package provides a simple Prefab and Scripts to add all 7 types of haptic feedback into your app. You can get it from the Asset Store link here:

Things to watch out for

As with any design tool, here’s a few things to watch out for when incorporating haptics in your mobile AR app.

Using haptics too much can mess up ARKit tracking

Test the impact of haptics on your AR session. Since ARKit relies on inertial sensing to track the phone’s motion, adding too many vibrations during an ARKit session can throw off tracking slightly.

Using haptics too much can overheat the device

Haptics is, after all, a physical movement of your mobile device and naturally tends to use more energy. Use this sparingly to ensure your phone doesn’t overheat or run out of battery too fast.

Too much haptic feedback might confuse and desensitize your user

This is true for any haptic mechanism. Don’t overdo it. Specifically, don’t use it without a clear understanding of why haptic feedback is necessary for the action your user is performing. The danger of overuse is that your user gets confused by it and therefore gets desensitized to your feedback.

And that’s it! I hope this article has given you a helpful dose of design ideas and convinced you to venture into the world of mobile AR haptics. We’ve really enjoyed exploring the different ways we could simulate touch sensations in mobile AR and if you have any more ideas we would love to talk to you about it. If you’re interested in trying out any of our code examples for mobile AR haptics, send me an email at neil [at] placenote.com.

If you’re interested in Persistent AR apps or what we do at Placenote, message us on twitter, or check out Placenote.com

This article was originally posted here

In the previous post, I explained what ARCore is and how it helps developers build awesome augmented reality apps without the need to understand OpenGL or Matrix maths.

If you haven’t checked it out yet, I highly recommend doing so before moving ahead with this article and diving into ARCore app development.

Overview

According to Wikipedia, ARCore is a software development kit developed by Google that allows for augmented reality applications to be built.

ARCore uses three key technologies to integrate virtual content with the real environment:

1. Motion Tracking: it allows the phone to understand its position relative to the world.
2. Environmental understanding: This allows the phone to detect the size and location of all type of surfaces, vertical, horizontal and angled.
3. Light Estimation: it allows the phone to estimate the environment’s current lighting conditions.

Getting Started

To get started with ARCore app development, you first need to enable ARCore in your project. This is simple as we will be using Android Studio and Sceneform SDK. There are two major operations Sceneform performs automatically:

1. Checking for availability of ARCore
2. Asking for camera permission

You don’t need to bother with these two steps when creating an ARCore app using Sceneform SDK. But you do need to include Sceneform SDK in your project.

Create a new Android Studio project and select an empty activity.

Add the following dependency to your project level build.gradle file:

dependencies {    classpath 'com.google.ar.sceneform:plugin:1.5.0'}

Add the following to your app level build.gradle file:

implementation "com.google.ar.sceneform.ux:sceneform-ux:1.5.0"

Now sync project with Gradle files and wait for the build to finish. This will install the Sceneform SDK to the project and Sceneform plugin to AndroidStudio. It will help you to view the .sfb files. These files are the 3D models which are rendered in your camera. It also helps you in importing, viewing, and building 3D assets.

Building your first ARCore app

Now with our Android Studio setup complete and Sceneform SDK installed, we can get started with writing our very first ARCore app.

First, we need to add the Sceneform fragment to our layout file. This will be the Scene where we place all our 3D models. It takes care of the camera initialization and permission handling.

Head over to your main layout file. In my case it is activity_main.xml and add the Sceneform fragment:

<?xml version="1.0" encoding="utf-8"?><FrameLayout xmlns:android="http://schemas.android.com/apk/res/android"    xmlns:tools="http://schemas.android.com/tools"    android:layout_width="match_parent"    android:layout_height="match_parent"    tools:context=".MainActivity">

    <fragment android:name="com.google.ar.sceneform.ux.ArFragment"        android:id="@+id/ux_fragment"        android:layout_width="match_parent"        android:layout_height="match_parent" />

</FrameLayout>

I’ve set the width and height to match parent as this will cover my entire activity. You can choose the dimensions according to your requirements.

Compatibility Check

This is all that you need to do in the layout file. Now head over to the java file, in my case which is MainActivity.java. Add the method below in your class:

public static boolean checkIsSupportedDeviceOrFinish(final Activity activity) {    if (Build.VERSION.SDK_INT < Build.VERSION_CODES.N) {        Log.e(TAG, "Sceneform requires Android N or later");        Toast.makeText(activity, "Sceneform requires Android N or later", Toast.LENGTH_LONG).show();        activity.finish();        return false;    }    String openGlVersionString =            ((ActivityManager) activity.getSystemService(Context.ACTIVITY_SERVICE))                    .getDeviceConfigurationInfo()                    .getGlEsVersion();    if (Double.parseDouble(openGlVersionString) < MIN_OPENGL_VERSION) {        Log.e(TAG, "Sceneform requires OpenGL ES 3.0 later");        Toast.makeText(activity, "Sceneform requires OpenGL ES 3.0 or later", Toast.LENGTH_LONG)                .show();        activity.finish();        return false;    }    return true;}

This method checks whether your device can support Sceneform SDK or not. The SDK requires Android API level 27 or newer and OpenGL ES version 3.0 or newer. If a device does not support these two, the Scene would not be rendered and your application will show a blank screen.

Although, you can still continue to deliver all the other features of your app which don’t require the Sceneform SDK.

Now with the device compatibility check complete, we shall build our 3D model and attach it to the scene.

Adding the assets

You will need to add the 3D models which will be rendered on your screen. Now you can build these models yourself if you are familiar with 3D model creation. Or, you can visit Poly.

There you’ll find a huge repository of 3D assets to choose from. They are free to download. Just credit the creator and you are good to go.

In the Android Studio, expand your app folder available on the left-hand side project pane. You’ll notice a “sampledata” folder. This folder will hold all of your 3D model assets. Create a folder for your model inside the sample data folder.

When you download the zip file from poly, you will most probably find 3 files.

1. .mtl file
2. .obj file
3. .png file

Most important of these 3 is the .obj file. It is your actual model. Place all the 3 files inside sampledata -> “your model’s folder”.

Now right click on the .obj file. The first option would be to Import Sceneform Asset. Click on it, do not change the default settings, just click finish on the next window. Your gradle will sync to include the asset in the assets folder. Once the gradle build finishes, you are good to go.

You’ve finished importing a 3D asset used by Sceneform in your project. Next, let’s build the asset from our code and include it in the scene.

Building the Model

Add the following code to your MainActivity.java file (or whatever it is in your case). Don’t worry, I’ll explain all the code line by line:

private static final String TAG = MainActivity.class.getSimpleName();private static final double MIN_OPENGL_VERSION = 3.0;

ArFragment arFragment;ModelRenderable lampPostRenderable;

@Override@SuppressWarnings({"AndroidApiChecker", "FutureReturnValueIgnored"})

protected void onCreate(Bundle savedInstanceState) {    super.onCreate(savedInstanceState);    if (!checkIsSupportedDeviceOrFinish(this)) {        return;    }    setContentView(R.layout.activity_main);    arFragment = (ArFragment) getSupportFragmentManager().findFragmentById(R.id.ux_fragment);

    ModelRenderable.builder()            .setSource(this, Uri.parse("LampPost.sfb"))            .build()            .thenAccept(renderable -> lampPostRenderable = renderable)            .exceptionally(throwable -> {                Toast toast =                        Toast.makeText(this, "Unable to load andy renderable", Toast.LENGTH_LONG);                toast.setGravity(Gravity.CENTER, 0, 0);                toast.show();                return null;            });

}

First, we find the arFragment that we included in the layout file. This fragment is responsible for hosting the scene. You can think of it as the container of our scene.

Next, we are using the ModelRenderable class to build our model. With the help of setSource method, we load our model from the .sfb file. This file was generated when we imported the assets. thenAccept method receives the model once it is built. We set the loaded model to our lampPostRenderable.

For error handling, we have .exceptionally method. It is called in case an exception is thrown.

All this happens asynchronously, hence you don’t need to worry about multi-threading or deal with handlers XD

With the model loaded and stored in the lampPostRenderable variable, we’ll now add it to our scene.

Adding the Model to Scene

The arFragment hosts our scene and will receive the tap events. So we need to set the onTap listener to our fragment to register the tap and place an object accordingly. Add the following code to onCreate method:

arFragment.setOnTapArPlaneListener(        (HitResult hitresult, Plane plane, MotionEvent motionevent) -> {            if (lampPostRenderable == null){                return;            }

            Anchor anchor = hitresult.createAnchor();            AnchorNode anchorNode = new AnchorNode(anchor);            anchorNode.setParent(arFragment.getArSceneView().getScene());

            TransformableNode lamp = new TransformableNode(arFragment.getTransformationSystem());            lamp.setParent(anchorNode);            lamp.setRenderable(lampPostRenderable);            lamp.select();        });

We set the onTapArPlaneListener to our AR fragment. Next what you see is the Java 8 syntax, in case you are not familiar with it, I would recommend checking out this guide.

First, we create our anchor from the HitResult using hitresult.createAnchor() and store it in an Anchor object.

Next, create a node out of this anchor. It will be called AnchorNode. It will be attached to the scene by calling the setParent method on it and passing the scene from the fragment.

Now we create a TransformableNode which will be our lamppost and set it to the anchor spot or our anchor node. The node still doesn’t have any information about the object it has to render. We’ll pass that object using lamp.setRenderable method which takes in a renderable as it’s parameter. Finally call lamp.select();

Phew!! Too much terminology there, but don’t worry, I’ll explain it all.

1. Scene: This is the place where all your 3D objects will be rendered. This scene is hosted by the AR Fragment which we included in the layout. An anchor node is attached to this screen which acts as the root of the tree and all the other objects are rendered as its objects.
2. HitResult: This is an imaginary line (or a ray) coming from infinity which gives the point of intersection of itself with a real-world object.
3. Anchor: An anchor is a fixed location and orientation in the real world. It can be understood as the x,y,z coordinate in the 3D space. You can get an anchor’s post information from it. Pose is the position and orientation of the object in the scene. This is used to transform the object’s local coordinate space into real-world coordinate space.
4. AnchorNode: This is the node that automatically positions itself in the world. This is the first node that gets set when the plane is detected.
5. TransformableNode: It is a node that can be interacted with. It can be moved around, scaled rotated and much more. In this example, we can scale the lamp and rotate it. Hence the name Transformable.

There is no rocket science here. It’s really simple. The entire scene can be viewed as a graph with Scene as the parent, AnchorNode as its child and then branching out different nodes/objects to be rendered on the screen.

Your final MainActivity.java must look something like this:

package com.ayusch.arcorefirst;

import android.app.Activity;import android.app.ActivityManager;import android.content.Context;import android.net.Uri;import android.os.Build;import android.support.v7.app.AppCompatActivity;import android.os.Bundle;import android.util.Log;import android.view.Gravity;import android.view.MotionEvent;import android.widget.Toast;

import com.google.ar.core.Anchor;import com.google.ar.core.HitResult;import com.google.ar.core.Plane;import com.google.ar.sceneform.AnchorNode;import com.google.ar.sceneform.rendering.ModelRenderable;import com.google.ar.sceneform.ux.ArFragment;import com.google.ar.sceneform.ux.TransformableNode;

public class MainActivity extends AppCompatActivity {    private static final String TAG = MainActivity.class.getSimpleName();    private static final double MIN_OPENGL_VERSION = 3.0;

    ArFragment arFragment;    ModelRenderable lampPostRenderable;

    @Override    @SuppressWarnings({"AndroidApiChecker", "FutureReturnValueIgnored"})    protected void onCreate(Bundle savedInstanceState) {        super.onCreate(savedInstanceState);        if (!checkIsSupportedDeviceOrFinish(this)) {            return;        }        setContentView(R.layout.activity_main);        arFragment = (ArFragment) getSupportFragmentManager().findFragmentById(R.id.ux_fragment);

        ModelRenderable.builder()                .setSource(this, Uri.parse("LampPost.sfb"))                .build()                .thenAccept(renderable -> lampPostRenderable = renderable)                .exceptionally(throwable -> {                    Toast toast =                            Toast.makeText(this, "Unable to load andy renderable", Toast.LENGTH_LONG);                    toast.setGravity(Gravity.CENTER, 0, 0);                    toast.show();                    return null;                });

            arFragment.setOnTapArPlaneListener(                    (HitResult hitresult, Plane plane, MotionEvent motionevent) -> {                        if (lampPostRenderable == null){                            return;                        }

                        Anchor anchor = hitresult.createAnchor();                        AnchorNode anchorNode = new AnchorNode(anchor);                        anchorNode.setParent(arFragment.getArSceneView().getScene());

                        TransformableNode lamp = new TransformableNode(arFragment.getTransformationSystem());                        lamp.setParent(anchorNode);                        lamp.setRenderable(lampPostRenderable);                        lamp.select();                    }            );

    }

    public static boolean checkIsSupportedDeviceOrFinish(final Activity activity) {        if (Build.VERSION.SDK_INT < Build.VERSION_CODES.N) {            Log.e(TAG, "Sceneform requires Android N or later");            Toast.makeText(activity, "Sceneform requires Android N or later", Toast.LENGTH_LONG).show();            activity.finish();            return false;        }        String openGlVersionString =                ((ActivityManager) activity.getSystemService(Context.ACTIVITY_SERVICE))                        .getDeviceConfigurationInfo()                        .getGlEsVersion();        if (Double.parseDouble(openGlVersionString) < MIN_OPENGL_VERSION) {            Log.e(TAG, "Sceneform requires OpenGL ES 3.0 later");            Toast.makeText(activity, "Sceneform requires OpenGL ES 3.0 or later", Toast.LENGTH_LONG)                    .show();            activity.finish();            return false;        }        return true;    }}

Congratulations!! You’ve just completed your first ARCore app. Start adding objects and see them come alive in the real world!

This was your first look into how to create a simple ARCore app from scratch with Android studio. In the next tutorial, I would be going deeper into ARCore and adding more functionality to the app.

If you have any suggestions or any topic you would want a tutorial on, just mention in the comments section and I’ll be happy to oblige.

Like what you read? Don’t forget to share this post on Facebook, Whatsapp and LinkedIn.

You can follow me on LinkedIn, Quora, Twitter and Instagram where I answer questions related to Mobile Development, especially Android and Flutter.

A tree in the forest

I was recently working on a prototype where I needed to place a large amount of objects in 3D space. This was an AR project, and the thought was that these objects would be placed around you in a random fashion, scattering around you as they dropped in from the sky.

This brings up a few problems. First, we don’t know the current surroundings of the user, so we have to limit the radius of the dropped items to something that we can configure based on location or other factors.

Next, we know that built-in random generators are not very random unless specific precautions are taken.

Lastly, generating points with random generators, no matter how random, results in clustering, where many of the generated points can fall in close areas to one another, and leave you with spots that are bald of any points. And nobody wants bald spots: trust me, I know.

There are a number of ways to accomplish random equal distribution. Like any developer worth his salt, I found a couple which have simple implementations that work well for our purpose. Let’s examine each one, and implement them in Swift using ARKit and SceneKit.

Roll them dice!

Before we get to the “good” algorithms, we should look to see what we get by using just random numbers to place our points. In doing this, we’ll get our app put together and use it to test out the other implementations in the same form.

Let’s create our app and get some of the basic stuff in place. Open up Xcode and create a new project using the Augmented Reality App template. Build and run the app to make sure everything is working and you see the default ship appear in front of your phone when running the app.

Next, we’re going set up our project like we did for the first tutorial in my ARKit series. Follow the steps in that tutorial, with the one difference being the name of the scene file. Instead of naming it HoverScene, we’ll name it **MainScene** instead. Add the render delegate as described in the tutorial, and follow the Extra Credits section where the tap gesture recognizer is created.

At this point the project is almost ready, but we don’t have (nor need) the addSphere method that is referenced in that tutorial. Instead, we will begin creating our algorithm generators.

Create a file called **PointGenerator.swift**. In here we’ll put several iterations of our algorithms. Let’s begin with the random number point generator. We’ll create a protocol that our algorithms will adhere to, making it easy to switch between algorithms in our source later on.

Our protocol is simple. Given a number of points to generate and a width and length to limit the points to, give back an array of points:

Our RandomPointGenerator will adhere to this class and output our first set of results:

The code here is simple. We iterate and create points that lie within the width and length limits, placing points on either side of the midpoint of those limits. We add the created points to an array then simply return the points.

Create a class called Visualizer derived from SCNNode. This class will serve as the container that holds the objects that we’ll place in the world to visualize the point set. For the moment, we’re going to create small spheres at each point generated by our algorithms.

This is what our class should look like:

Alright, now we’re ready to create our points. Let’s go back to our MainScene class and add a method called createPointField, which takes in a SCNVector3 position:

We’re going to call this from our ViewController when we tap on the screen. Let’s go to our didTapScreen method and make the part where we previously created a sphere (in that first tutorial) look like the following:

Build and run, and we now have our first algorithm implemented.

Random placement of points. Not great.

Notice how the spheres are clustered in certain spots. This is exactly what we want to avoid.

I won’t get into detailed descriptions of each algorithm, but I’ll provide links to the sites that I found informative and gave me the inspiration to implement in Swift and AR.

Poisson Sampling and Mitchell’s best-candidate

What we need as an alternative to the random number generator way is an algorithm which returns a set of points that are close to each other, but no closer than some specified minimum distance. That’s where Poisson-disc sampling comes into the picture. There are several ways of implementing the Poisson-disc algorithm. The one we’re going to be implementing in our code is called Mitchell’s best-candidate algorithm. It’s easy to implement and runs fast.

The idea behind the algorithm is to place down points, and as you place them, check whether they meet the requirement of being at least the minimum distance away from the points already placed. To do this, you sample the point as you place it, by looking at the distance that nearby points have. If there are no points within the minimum requirement, place the new point, otherwise, try to find another location. You can read more about the algorithm here.

To implement, we’re going to create another implementation of our **PointGenerator** protocol:

Let’s head back to our MainScene class, comment out the random generator lines, and add these new lines in:

Run the app again, and let’s look at our results.

Poisson-disc using Mitchell’s best candidate method

Much better! There are no big clusters or barren areas. This is now something we can use to place items in the world. Other uses for this algorithm include things like dynamically generating textures at runtime or noise generators.

We have what we need with this implementation, and we’re going to make use of it later. But what if we wanted to have something a bit different, where we need uniform distribution within a circular boundary? This is where the Sunflower Seed algorithm comes in.

Extra Credit

Sunflower Seed Algorithm

Throughout history, we have found mathematical patterns in nature. One of the interesting features in many of these patterns that are mimicked in nature is the existence of the Fibonacci sequence in plants. These features manifest themselves in spiral patterns in leaves, seeds and petal arrangements. The study of these patterns is called Phyllotaxis. The following algorithm implements the mathematical model of one of these spirals. You can find more info here and here.

Let’s go back to our PointGenerator file and create our new implementation:

You’ll notice here that we are ignoring the width and height parameters passed in. This is because instead of constraining the points to a region, we will be evenly distributing the points in a spiral fashion until we run out of points.

Changing the alpha in the parameter passed in to the sunflower method controls the granularity of the points at the edge of the boundary. That is, we can make the boundary smoother or rougher by controlling the point distribution. The above code uses an alpha of 2, which is on the high side, and results in a more even boundary.

Let’s go to our MainScene again, and comment out the previous algorithm. Let’s add in a call to get our Sunflower pattern generating points:

Let’s run our app again and see what we get.

Sunflower Algorithm

As you can see, we have a pattern that mimics the way sunflowers hold their seeds. There is also an interesting variation we can apply to the algorithm, as detailed in one of the comments in this Stack Overflow question.

By changing the theta to a bearing, the commenter turned the algorithm into a geodesic formation.

Change the theta line in our code to the following:

Let’s run our algorithm again and see what it looks like.

Sunflower Algorithm with spiral pattern

Cool! Now we have the pattern as spirals.

Speaking of spirals, let’s check out one last algorithm.

The Vogel Spiral

Here we have another closely related algorithm which also uses a Fibonacci sequence and the Golden Angle. You can read more about the Vogel Spiral here and here.

Let’s implement it, and then we’ll tweak it to see how it influences the results.

In our PointGenerator class, add our implementation of this algorithm.

This algorithm, like the Sunflower Seed algorithm, also ignores the width and height parameters.

Let’s replace the previous algorithm with our new calls.

Let’s give that a run and see what we get.

Vogel Spiral

Nice! Now, let’s try different variations of this algorithm. By changing the formula, we can get different spiral formations. Change the it declaration to the following line:

Run that and check out what we get.

Vogel spiral, looking like the Sunflower

That spiral now looks like a flipped version of our Sunflower spiral with the updated theta. Interesting!

Let’s try it with the following formula:

Vogel spirals

Similar to the first algorithm, but the spirals are separated. Very cool stuff!

We’ve now run through 4 different algorithms, including the not-very-good strictly random number placement. Each one of these has a place in our toolbox, though, and they can be used to fulfill a variety of needs.

Extra Credit, Part Two

It wouldn’t be an article of mine if we didn’t do something cool with what we just learned, now would it?

Download some tree models from a 3D model store like Sketchfab or Turbosquid. Convert them to Collada (DAE) format as needed and add them to your project. You may need to resize them to be the proper scale when putting them in your scene, but you’ll know when you go to use them. Be sure to use a low poly model since we are talking about instantiating dozens and even hundreds of object instances.

Let’s make a Tree class that derives from **SceneObject** (we created this class back in our previous tutorial). We’ll make it load a random tree from the ones we have added into our app. We make use of the random convenience function we also added in a previous tutorial.

Here’s what my Tree class looks like:

Let’s use the Mitchell algorithm and bring down the number of models (points) we want to generate to 60. Depending on the amount of polygons in your models, this might be too large. I started with a different set of models and it took a while to place 20 models. Start low and work your way up. For the models I used I could go higher, but 60 was dense enough.

In our Visualizer code, let’s change our Tree creation to animate the scaling up a bit.

In my case, with the models I used, I had to scale them down a bit to get them to a reasonable size which is where that 0.45 comes from. I also lowered the position a bit so that they lay on the ground plane. You can adjust these numbers to whatever fits your situation.

Build and run, and now we have a happy little forest created with almost no effort.

Our happy little forest

Hope you enjoyed this little experiment. Feel free to show off your work in the comment section!

Daniel Wyszynski is a developer who’s worked on more platforms and languages than he cares to admit to. He believes in building products that create memorable user experiences. To Dan, the users are first, the platforms are second. Follow Dan on Medium or Twitter to hear more from him. Also check out the s23NYC: Engineering blog, where a lot of great content by Nike’s Digital Innovation team gets posted.

Author’s Note: Views are my own and do not necessarily represent those of my employer

Today, about 4% of the world’s population is visually impaired. Tasks like simple navigation across a room, or walking down a street pose real dangers they have to face every day. Current technology based solutions are too inaccessible, or difficult to use.

As part of a university assignment, we (myself, Louis, and Tom) devised and implemented a new solution. We used configurable WebGL shaders to augment a video feed of a user’s surroundings in real-time. We rendered the output in a AR/VR format, with effects such as edge detection and color adjustments. Later, we also added color blindness simulation, for designers to use. We also added some AI experiments.

We did a more in-depth literature review in our original research paper. ACM published a shorter, two page version here. This article focuses more on the technologies used, as well as some of the further uses, and experiments such as AI integration.

A popular approach we found in our studies of existing solutions was the use of edge detection for detecting obstacles in the environment. Most solutions fell short in terms of usability, or hardware accessibility and portability.

The most intuitive approach we could think of as feedback to the user was through the use of a VR headset. While this meant that the system would not be of help to very severely visually impaired people, it would be a much more intuitive system for those with partial sight, especially for those with blurry vision.

Edge detection

Feature detection, such as edges, are best done using 2D convolutions, and are even used in deep learning (convolutional neural networks). Simply put, these are dot products of a grid of image data (pixels) against weights in a kernel/filter. In edge detection, the output is higher (more white) when the pixel values line up with the filter values, representing an edge.

_This is done for every pixel. - [Image credit](http://jeanvitor.com/convolution-parallel-algorithm-python/" rel="noopener" target="blank" title=")

There are a few available options for edge detection filters. The ones we included as configurations are Frei-chen, and the 3x3 and 5x5 variants of Sobel. They each achieved the same goal, but with slight differences. For example, the 3x3 Sobel filter was sharper than the 5x5 filter, but included more noise, from textures such as fabric:

Left: 3x3, Right: 5x5

The web platform

The primary reason we chose the web as a platform was its wide availability, and compatibility across almost all mobile devices. It also benefits from easier access, compared to native apps. However, this trade-off came with a few issues, mostly in terms of necessary set-up steps that a user would need to take:

1. Ensure network connectivity
2. Navigate to the web page
3. Turn the device to landscape mode
4. Configure the effect
5. Enable VR mode
6. Activate full screen mode (by tapping the screen)
7. Slot the phone into a VR headset

To avoid confusing a non-technical user, we created the website as a PWA (progressive web app), allowing the user to save it to their Android home screen. This ensures it always starts on the correct page, landscape mode is forced on, the app is always full screen, and not reliant on a network connection.

Performance

Early JavaScript prototypes ran nowhere near our 60fps target, due to the very expensive convolution operations. We suspected that the bottleneck was JavaScript itself. We attempted a WebAssembly version. The resulting prototype ran even slower. This was most likely due to the overhead in passing the video frame data to the WebAssembly code, and back.

So instead, we turned to WebGL shaders. Shaders are awesome because of their extreme parallelization of a bit of code (the shader) across the texture (video feed) pixels. To maintain high performance, while keeping a high level of customization, the shader code had to be spliced together and re-compiled at run-time, as configurations changed, but with this, we managed to stay within the 16.7ms frame budget needed for 60fps.

Feedback

We carried out some user testing. We tested some basic tasks like navigation, and collected some qualitative feedback. This included adjustments to the UI, a suggestion to add an option to configure the colors of the edges and surfaces, and a remark that the field of view (FoV) was too low.

Both software improvement suggestions were applied. The FoV was not something which could have been fixed through software, due to camera hardware limitations. However, we managed to find a solution for this in the form of cheaply available phone-camera fish-eye lenses. The lenses expanded the FoV optically, instead of digitally.

Fish-eye lens

Left: no lens, Right: with lens

Other than that, the system surpassed initial expectations, but fell short on reading text. This was due to there being two sets of edges for each character. Low light performance was also usable, despite the introduction of more noise.

Some shader effect configuration examples

Some other configurations we included was the radius of the effect, its intensity, and color inversion.

The options menu

Other use cases

An idea we had was to add shader effects to simulate various types of color blindness, providing an easy way for designers to detect color blindness related accessibility issues in their products, be they software or otherwise.

Using RGB ratio values found here, and turning off edge detection, we were able to add basic simulations of all major types of color blindness through extra, toggle-able components in the shaders.

_Ishihara test - [Image Credit](http://www.colour-blindness.com/colour-blindness-tests/ishihara-colour-test-plates/" rel="noopener" target="blank" title=")

(Left) Normal (Right) Deuteranopia filter

(Left) Normal (Right) Protanopia filter

AI and future work

Although it’s an experiment, still in its very early stages, higher level object detection can be done using tensorflowjs and tfjs-yolo-tiny, a tensorflowjs port of tiny-yolo, a smaller and faster version of the YOLO object detection model.

The current categories

Example inference, using a laptop. Requires a decent GPU, to work.

The next step is to get instance segmentation working in a browser, with something similar to mask rcnn (though, it may need to be smaller, like tiny-yolo), and add it to WebSight, to highlight items with a color mask, instead of boxes with labels.

_Goals — [Source](https://github.com/matterport/Mask_RCNN#4k-video-demo-by-karol-majek" rel="noopener" target="blank" title=")

The GitHub repo is here, and a live demo can be found at https://websight.danruta.co.uk. Do note that until Apple provides support for the camera API in browsers, it might not work on Apple phones.

Of course, I had some extra fun with this as well. Being able to edit what you can see around you in real time opens up a world of opportunities.

For example, using a Matrix shader, you can feel like The One.

Everything is drawn using green digits, not lines

Or maybe you just enjoy watching the world burn.

_Playing: [Billy Joel — We Didn’t Start the Fire](https://www.youtube.com/watch?v=eFTLKWw542g" rel="noopener" target="blank" title=")

You can tweet more shader ideas at me here: @DanRuta

If you’ve read my previous post, you already have a beautiful AR floor in your dining room. That was the first thing we built while learning the basics of AR. And now, it’s time to decorate the room with some cool virtual furniture. At the end of this tutorial, you will have a dining room just like this:

Naming SCNNodes

Let’s get our hands dirty. Fire up Xcode, and open the last project where we decorated our floor. Before viewDidLoad, create a constant floorNodeName.

let floorNodeName = "FloorNode"

We are now going to set the name of the floor node to this constant so that we don’t mix up this node with other furniture nodes. Go to the createFloorNode method and name the floor node.

In line 7 we have just named the floor node — everything else remains the same.

The plan is that, once launched, the app would recognise the floor first and then the user would see through the screen to determine where they want to place the furniture. They would tap on the location and a piece of furniture would sit just there. To achieve this, we need a correlation between the points on screen and the real-world locations. Thankfully, Apple has made this process quite simple.

Gestures and HitTests

An active AR session keeps on finding nearby objects/planes. Once a new object/plane is found, it places AR anchors on top of it. To find exactly on which anchors user has tapped, we need the help of HitTest. HitTest works like this:

• a logical ray is shot from the point of touch to the anchors on the plane
• all the anchors the ray passes through are stored in an array in the format of HitTestResult.

Each HitTestResult contains the information of the real-world surface of an AR anchor. We will use this HitTestResult information to place our furniture.

Let’s create a method that adds the tap gesture to our sceneView to interact with the user. We’ll call this method from viewDidLoad.

Now define the “tapped” method to get the location of the tap and place a furniture there. For now, we print to test if the HitTest is working fine.

In the first line, we are casting tap gesture’s view to ARSCNView. As we know that the tap will come from our sceneView itself, we force-unwrap it. Then we get the location on sceneView where user has tapped. Then a HitTest is performed to get all the HitTestResults from the tapped location to the real-world anchors. “.existingPlaneUsingExtent” gives the estimated size of the detected planes. Now, we check if the user actually has tapped on a detected plane or somewhere else, and we print accordingly.

Now run the app, wait until the world origin loads and the floor is detected. Then tap on the screen to check if we are hitting the planes correctly.

If you tap on the places where the tiles are located, it will print “Touched on the plane.” Touch anywhere else, and it will be “Not a plane.” If there’s no output in the console, you haven’t called the addTapGesture method from viewDidLoad (this happed to me!). Now that we have successfully detected the clicked position, it’s time to bring home some furniture.

Importing 3D models to the project

We need some 3D furniture models. I use turbosquid, which is a great repository of 3D models. You need to create a free account there to access their free models. I’ve downloaded a 3D table for now (in the GitHub repo — I’ll add some more furniture as well).

Apple advises you to use collada-formatted(.dae) models in SceneKit. I have used other kinds of models as well in the past, and I have faced problems in many cases. Mostly, if there is a glass, SceneKit tends to make it solid. So, let’s find 3D models with a .dae extension.

Now add an asset folder under the “Home Decor” group.

Press next, and in the save dialog, save it as furnitures.scnassets and not xcassets.

Working with Xcode SceneKit Editor

All scene files or models that will be used by SceneKit should reside in a scnassets folder. Now drag and drop the table we just downloaded to the scnassets folder and rename it as “Table”. Click on the Table.dae file (if not already opened) to open it in SceneKit Editor. On the bottom left corner there is a button (marked with a red ellipse below) to open Scene Graph View. Click on it and the editor should appear like below.

In the Scene Graph there are 3 nodes: a Camera, a light source (Lamp) and the table (Small_table). ARKit has its default camera, so we don’t need this camera anymore. As we will be using sceneView’s default lighting, we don’t need the Lamp either. Remove both of them.

Let’s rename “Small_table” to “Table”, same as the file name. Now the scene graph will look like this:

There is something called point of view (marked with a red ellipse above) in the editor which determines how the object will look from a certain point of view. By default, it’s set to Perspective, but we will be seeing the table from the front. So change the point of view to Front.

Oops — it looks like we can see only the top view. Clearly, this isn’t how we would like to see it. We want to see the table as it was shown in the perspective view. Let’s fix this.

Select the Table node, open the Utilities tab (right-top corner button in Xcode) and click on the node inspector(the cube icon). You should see the below window.

I’ve numbered the steps for your reference. To view the complete table from the front, we need to rotate the table on its X axis. If you can recollect, we did same kind of thing in the first post where we rotated the capsule on its Z axis by changing its Euler angle.

If you see the Transforms matrix, the Euler angle for X is already specified to 90 degree radians, which is causing the table to be rotated incorrectly. Make it zero and the rotation will be fixed. But the positioning has a vector of (-0.35,0.348,0). We will make it (0,0,0) to place the table exactly where the user will tap. Now the Transform will look like this:

Transform matrix editing is kind of a trial and error task. You might need to go through quite a number of iterations before you reach the exact position.

Now, we change the .dae model to a SceneKit scene file (.scn) which will made it much more efficient for SceneKit to handle. Go to Editor>Convert to SceneKit scene file(.scn).

And…we are done with the 3D object editing. Head back to the ViewController file. First, since we have removed the light source of the table, we should turn on the default lighting of the scene view in viewDidLoad.

sceneView.autoenablesDefaultLighting = true

Positioning 3D objects

Then we create a method to add the table to the scene. It will accept a HitResult as a parameter and place the table based on the position of the HitResult.

Bear with me — this is the last method which needs some explanation!

1. A constant is declared to know what furniture scene needs to be added. We will change it to a variable and declare it on top of the viewcontroller when we have more furniture.
2. Then we create a scene from the selected furniture file.
3. A furniture node is created from the child node of the furniture scene’s root node with the name of the furniture. As we named the first node the same as the furniture name, it doesn’t need to traverse any further. Hence the recursive option is set to false.
4. HitResult’s worldTransform property holds the co-relation transform matrix between the real-world position and the scene anchor/node position. And the 3rd column of the transform matrix holds the position information.
5. Now that we have successfully extracted the world coordinate of the tapped location, our job is now just to place the furniture node on that exact same coordinate.
6. Then we add the node to the root node of the scene. And thats it!

Now we just need to call the method whenever the user taps on the screen. Let’s modify the tapped method to accommodate this change.

Here, if we find a plane, we are just calling the method to add the furniture. As HitTest holds all the positions the ray passes through, we are considering the top most result. Let’s run the app, wait until the floor has tiles on it, and then tap on the screen to place a table. And voilà! You have a new table on your floor. Watch your step :)

I have downloaded some more furniture and have added it into the repo. I’ve also added pinch and rotate gestures to resize and rotate objects. The complete source code will give you an appearance like this:

You can download the full source code from GitHub.

Hope you have enjoyed reading the post as much as I did writing it :)

See you at the next post. Happy reading!!

As I discussed in the previous article on hot future tech, artificial intelligence (AI), augmented reality (AR/VR), blockchain, and the Internet of Things (IoT) might seem a long way off from mainstream adoption.

Movies like Ready Player One and Avengers: Infinity Wars only perpetuate this perception by mixing real technologies with fantasy, making the tech we wield in the real world seem primitive in the process.

The reality, though?

AI, AR/VR, blockchain, and IoT are already playing an important role in our everyday lives, with countless examples hiding in plain sight.

In Part One of our series, we explored how Artificial Intelligence (AI) is already deeply integrated into apps we use every day. In this article, we’ll dive into Augmented Reality (AR).

What is Augmented Reality?

Not to be confused with virtual reality (VR), which is 100% driven by computer generated images, augmented reality (AR) is a technology that combines computer-generated visuals with a user’s live view of the real world, resulting in a composite view.

The Pixels movie was a somewhat loose interpretation of augmented reality.

For the most part, AR today is meant to be experienced through the lenses of devices like smartphones. (Microsoft’s attempt to evangelize head-worn AR, or what it calls “mixed reality”, is gaining traction in some areas but can still be considered niche.)

Modern smartphones come equipped with everything you need to enjoy an engaging AR experience:

• A digital camera for providing a video feed of the real world.
• A computer to store and run software applications.
• A powerful processor capable of computer vision (CV), for accurate placement of the computer-generated images “into” the real world.
• A high-res display capable of showing the real-time composite view.
• Gyroscope sensors for sensing changes in orientation, to make the composite view more engaging/dynamic.

There is a massive business opportunity tied to the fact that all the hardware needed to experience meaningful AR applications today can be found inside any modern smartphone.

AR in Ecommerce

It can be challenging to spend your hard-earned cash on furniture you’ve never actually laid eyes on in person. And, even if you have seen a piece of furniture in person, questions remain that might prevent you from pulling the trigger on a purchase.

How will it look with my coffee table and side tables? Is it too traditional for my living room? Will that color look tacky against the floral wall paper and shag carpet?

Likewise, with fashion ecommerce, how do you know whether that designer jacket will actually look good on you without taking it into a fitting room and seeing whether it jives with the jeans and boots you’re wearing?

This is where AR comes in.

Companies like Amazon, IKEA, and Target have already added AR functionality to their mobile apps, allowing you to “place” virtual objects into your environment to preview how they might actually look in your home or office. You can place a virtual vase onto your actual coffee table (thanks to plane detection), or “push” a virtual wardrobe against your bedroom wall without breaking a sweat.

Native mobile ecommerce apps can be upgraded to provide AR previews, as Amazon has done on both iOS and Android.

In fashion, Michael Kors will be using a Facebook AR-based ad campaign to provide users with a simple way to “try on” sunglasses and other fashion merchandise before they buy.

Soon, any Facebook advertiser will be able to launch AR-based ad campaigns.

Ecommerce companies that allow shoppers to try out merchandise with AR believe that doing so will provide a significant lift in conversion rates — and consequently will help grow sales revenues.

AR in Marketing & Advertising

Even beyond the conversion-focused world of ecommerce, businesses are finding that augmented reality can be the perfect way to attract and engage potential customers.

Snap, the photography-obsessed creator of Snapchat, has already deployed a commercial platform called Lens Studio, which allows businesses to “create branded AR experiences”. Snapchat’s AR experiences are AR-based camera filters, called “lenses”, that Snapchat users are already very familiar with, thanks in part to earlier lenses like the Internet sensation, “Dancing Hotdog.”

See BMW’s Snapchat Lens in action below:

Facebook has also begun testing AR-based advertising with select clients (including Michael Kors, mentioned previously,) and is expected to make AR ad campaigns available to a wider audience of advertisers later in 2018. Early results are promising. ASUS reported that its AR-based test campaign saw 10x the engagement of its non-AR counterpart.

Beyond these platforms, CPG companies are creating standalone AR experiences to bring life to food or beverage products, for no other reason than to increase customer engagement and create a buzz. Patrón, the Mexican tequila company, launched an AR app last year that lets you plant your own agave field, from which a distillery and bartender emerge to tell you everything you’ve ever wanted to know about creating Patrón tequila.

The Patrón Experience lets you plant your own agave field and distillery on any flat surface.

For companies planning a new product launch or corporate event, adding an AR experience can help dial up both audience engagement and the resulting media coverage.

Consider Samsung’s Galaxy S9 product launch, where attendees were able to witness their name badges become virtual and interactive Galaxy S9 phones right in their very hands.

AR in Industrial and Medical Procedures

Even if you’re not an industrial technician or a medical surgeon, you can probably appreciate the complexities inherent to these professions. One false move, and lives could be at stake.

Thanks to AR, things are becoming a whole lot easier for these professionals.

Energy giant BP has begun using AR to help technicians in the field perform equipment maintenance. With the new technology, BP engineers wear AR headsets that transmit a live video feed to a remote expert, who can diagnose issues and then place relevant instructions and diagrams onto the field technician’s visual field (via the AR headset) in real time. This keeps the technician’s hands free to perform the necessary repairs.

Similar use cases are being developed in the medical field, where AR headsets can provide real time and hands-free information on surgical procedures and vital health information without having to look away from the patient.

A glimpse of what AR-enabled surgery might look like. Source: The Augmentarium at the University of Maryland (illustration by Brian G. Payne)

AR Gaming

It’s been two years since Pokemon Go first grabbed the world with its killer combination of geolocation and AR-based exploration, allowing every man, woman, and child to live out their Pokemon trainer fantasies in the real world.

Pokemon Go’s massive popularity (with nearly $2 billion in revenue to date) has piqued plenty of interest and funding in AR gaming.

Jurassic World: Alive launched earlier this year and doesn’t diverge very far from the location-based and collections-oriented gameplay made popular by Pokemon Go.

Pokemon Go, but with dinosaurs?

A Walking Dead AR mobile game (Walking Dead: Our World) came out just last month, and Pokemon Go creator, Niantic, is expected to launch AR-based Harry Potter: Wizards Unite at some point this year.

Admittedly, one major weakness of today’s AR games is that the AR component is often non-essential to the core gameplay.

Newly-released Walking Dead: Our World is an “AR-optional” game — and one in which the vast majority of users appear to skip the AR mode altogether (partly because it’s not essential to progress in the game, but also because the AR mode seems to struggle with plane detection.)

Even Pokemon Go — which has an excellent AR experience — is AR-optional. You can catch Pokemon without firing up the AR mode, though finding a rare Pokemon sitting on your laundry machine is MUCH more enjoyable.

Some of these limitations are inherent to the software development kits (SDK) the applications are built on. Apple’s ARKit and Google’s ARCore SDKs determine the ceiling for today’s mobile AR experiences on iOS and Android devices, respectively.

Both companies appear to be committed to enhancing support for richer AR experiences, though.

Announced earlier this year, Apple’s ARKit 2 will allow for much better AR-based gameplay experiences.

For example, consider this Apple demo of an AR game built on ARKit 2 , in which multiple players are able to interact with the SAME virtual objects in a “shared experience.” This mechanic opens up a universe of possibilities when it comes to multiplayer AR games.

Why AR will Gain Mainstream Adoption Faster than VR

Most modern smartphones come standard with every hardware component you need to experience meaningful AR right now.

This low barrier to entry stands in stark contrast to virtual reality (VR).

For VR to work, users must own a virtual reality headset, which enables users to see the virtual world while blocking out all traces of the real world. VR headsets are typically purchased separately from the computers or phones that run the VR applications. While low end VR experiences can be had using a smartphone slapped into an inexpensive, VR headset (like the $15 Google Cardboard), the more immersive and believable VR experiences (on Oculus Rift and HTC Vive, for example) require modern gaming computers and headsets that can cost upwards of $1,500 in total.

The other edge AR has over VR is its versatility.

AR apps are designed for a wide variety of real world use cases. And, because you maintain visibility of your actual surroundings when using AR, safety is not much of an issue — at least no more than when you’re playing Candy Crush or catching up on your Instagram feed.

VR, on the other hand, is best enjoyed in a room without other people, tethered to devices like a desktop computer and motion sensors. This limits VR’s usefulness, relegating VR to use cases around gaming, entertainment, and complex commercial or governmental simulations (for aviation or military training, for example).

Here’s how Apple CEO Tim Cook explained his bullishness on AR relative to VR:

“I’m incredibly excited by AR because I can see uses for it everywhere. I can see uses for it in education, in consumers, in entertainment, in sports. I can see it in every business that I know anything about. I also like the fact that [AR] doesn’t isolate. I don’t like our products being used a lot. I like our products amplifying us. And I think AR can help amplify the human connection. I’ve never been a fan of VR like that because I think it does the opposite. There are clearly some cool niche-y kind of things for VR. But it’s not profound in my view. AR is profound.”

AR’s Accessibility is Profound, Too

As jaw-dropping and eye-popping as some AR experiences can be, AR experiences are actually not difficult to create or enjoy.

From the consumer perspective (demand side), anyone with a modern smartphone can download AR-enabled applications and begin to enjoy the benefits of augmented reality.

From the business side (supply side), creating an AR experience is as easy as partnering with a reputable software company with AR/VR development capabilities.

The hardest part, as it often is with emerging technologies, is taking the initial plunge.

Short answer: using a browser-based Augmented Reality (AR) application. For the long answer, read below.

The idea of playing with AR started as a random interesting experiment. In our company, we strive to stay at the edge of the curve. We share technical novelties and new technologies with each other on a semi-regular basis. Since we are mostly dealing with web technologies, the concept of AR in the browser really took off.

Since AR is mostly an entertainment technology, a practical application was not obvious to us from the start. Luckily, two unrelated things happened at the same time:

• we had just created a mascot for our product — Hubert,
• we had to do a marketing booth at devoxxPL 2018

We decided to bring Hubert to life during the event, in the form of an AR app for people to play with. In our heads, users should be able to:

• render Hubert on a wall background in their phones
• take a picture of the rendered model
• tweet the photo (not the subject of this article)

The end result is available on glitch.com, scaled down and rotated to be suitable for a desktop experience (you can also take a quick look into the source code).

Rendering Hubert in real time

We used AR.js (version from this commit) as the main building block of our AR app — it packages webRTC camera access, marker recognition, and 3D scene rendering. We liked it mostly because you can have a basic demo running in around 20 lines of code.

Under the hood, AR.js can use either three.js or A-frame implementations to render your 3D scenes.

• three.js offers fine-grained control of 3D rendering, and is JavaScript-based. You have probably heard about it in the context of rendering 2D and 3D scenes in the browser.
• A-frame is a web framework designed specifically for building VR and AR experiences. It has an HTML-like markup that is more declarative than three.js, but sacrifices some of the control in favor of ease of use.

We didn’t have a VR or 3D expert (except Mrówa, who prepared the 3D model). As A-frame’s HTML-like declarative syntax looked more familiar to us, we opted for A-frame to do the rendering.

Here you can see the code for rendering Hubert, 30 lines on the dot. We omitted some options and A-frame tweaking for the sake of simplicity. You can refer to the repo to see it all.

This gives us Hubert nicely rendered in the web browser in real time.

Capturing a photo to tweet

Unfortunately, we don’t have a single video feed rendering the whole scene. There is the video from your camera and a rendered 3D scene. We quickly figured out that we would have to capture a frame from both sources and put them together for a nice photo of Hubert.

Taking frames out of a webRTC video stream is pretty straightforward. The best material on the subject can be found here. If your browser has the appropriate API, you need two elements:

• a reference to your source tag
• a destination element in which to put your frame

Then it’s just a simple matter of drawing a 2D image from video to canvas. In our case, both of these are a bit tricky.

The video take we are using is generated and embedded by AR.js. We had no idea how to get it gracefully, so we hacked our way around it with a loop and a DOM selector:

We also needed to hack some scaling. AR.js doesn’t present the raw video feed to the user, they scale it to fill the screen without losing aspect ratio. That means we need to apply the same scaling to our frame. If not, our screenshot will have “more” of the video feed than is shown on the screen. We don’t want to confuse the users here.

What the user sees:

If we take a frame without scaling and just try to copy from point (0,0) we lose margins imposed by AR.js. This is a totally different picture from what is presented to the user:

Suffice it to say we just reverse-engineered the scaling and figured out the bounding box of what the user sees:

To achieve this final result (the same as what is presented live to the user, give or take some camera shake):

Now we just need to get Hubert in the picture. Again, the API for that is very straightforward. To capture a screenshot of a rendered A-frame scene, we need to get the scene’s canvas. The relevant part is copied to our destination canvas, on top of the previously taken video frame.

Getting the relevant part is the tricky bit in our case. Thanks to the AR.js scaling, we cannot simply get the “perspective” shot of the scene and use that. It will look too wide or too short, depending on orientation.

For landscape mode (width > height), the scaling method we used for video works perfectly well.

For portrait mode, it works great on a PC… However, once you enter the realm of mobile devices, the scaling breaks and the screenshot doesn’t look nice. You get this skinny Hubert…

…instead of our lovely, bubbly mascot in all his glory:

We are still not sure why that is the case. We made the mistake of not testing it out thoroughly on actual mobile devices, thinking it would work the same as it does on the development machine. (Yes, we know how bad that sounds, but that’s the reality of it.) During the conference, we managed to figure out the formula for portrait scaling and introduced a fix:

It’s not pretty. It’s one of those “it’s late, it works, just leave it” fixes. The values presented above produced a satisfactory result and we left it at that.

With that, we have a picture of Hubert in the real world! It can be retrieved from the destination canvas element and displayed on the page or sent to the server to tweet out.

Summary

AR in the browser is possible. Even more, it is possible on mid-grade mobile hardware (as of June 2018). Getting it to work on every phone and browser is still a long shot, so don’t count on it for wide, diverse userbases.

However, if you have a somewhat controlled environment, augmented reality on a phone can be used to create unique experiences. These don’t require special hardware or workstations and that is a big, big plus. Just make sure to test it on actual devices ahead of time.

If you look around, this is the golden era of technology. Every keynote adds something new to the existing stack of technologies. It’s exciting to see how these emerging technologies have enhanced the boundaries of our imagination. As a developer, we must be proud that we are the firsthand users of these technologies.

But every new technology comes with quite a steep learning curve. You just can’t watch a keynote or a video on Youtube and start developing an app. But the good news is that, with AR in Swift, it’s remarkably easy to work with basic AR apps. Apple has done most of the heavy lifting for you. Follow along and you’ll see how easy it can be.

Let’s dig in…

In this tutorial, we will learn the necessary tools and techniques of AR in Swift that will allow us to create an app that decorates your floor with some cool floor tiles and wooden textures. The finished app will look something like this:

Let’s start by creating a single view application in Xcode and name it Home Decor.

Adding camera permissions

Now the very first thing we will do is to navigate to the info.plist file and enable camera usage. Camera capability is the first thing you need for an AR app. Find the Camera Usage Description key, like the below image, and give it a suitable message. This message will show up in the very first launch of the app while asking for the camera permissions from the user.

Adding ARKit Capabilities to the app

Go to Main.storyboard. Drag and drop an ARKit SceneKit View on the ViewController and pin the ARSCNView to the edges of the ViewController.

Create a IBOutlet to the ViewController class, and name it sceneView. As soon as you do that, an error stating undeclared ARSCNView, will popup, as our view controller doesn’t recognise anything of type ARSCNView. To resolve this, and to use other ARKit features, we need to import ARKit into the view controller.

Now move from storyboard to the view controller.swift file. Declare a property of type ARWorldTrackingConfiguration before the viewDidLoad() method and name it config. And our view controller will look like this (I’ve removed the didReceiveMemoryWarning method):

import UIKitimport ARKit

class ViewController: UIViewController {

@IBOutlet weak var sceneView: ARSCNView!let config = ARWorldTrackingConfiguration()

override func viewDidLoad() {super.viewDidLoad()}

Allow debugging

This config variable will determine the configurations of the scene session. We will see it’s usage later in the section. Now, in the viewDidLoad method after super.viewDidLoad(), add the following:

sceneView.debugOptions = [ARSCNDebugOptions.showFeaturePoints, ARSCNDebugOptions.showWorldOrigin]

Here we are enabling debug options for our sceneView, which is nothing but the camera view with the capabilities of AR framework. ARSCNDebugOptions.showWorldOrigin will display world origin on the screen. This will help us find the reference point of all other positions. ARSCNDebugOptions.showFeaturePoints will display all the points on the screen which the AR camera has recognised in the surroundings.

Now to start the AR session, we need to run a session in our sceneView with the configurations mentioned in the config variable. Just below sceneView.debugOptions line, write:

sceneView.session.run(config)

Now run the app on your device (not on a simulator, as it doesn’t have the camera). The alert asking for camera permissiosn with the message you wrote will show up, and you need allow it. Wait for a bit while it loads the world origin.

If you are here, you have already have an AR app running. Congratulations!

How AR Axes work

The red bar or X axis is used to position objects left or right of the world origin. The green bar or Y axis is used to position objects to the top or bottom of the world origin. And the blue bar or Z axis is used to determine how close or far an object will be placed from the world origin.

A positive value of X will position an object to the right of the world origin, and negative will place it on the left. Positive for Y will place it on top and negative will place it on the bottom of the world origin. Positive for Z will place it nearer, and negative will place it farther from the world origin.

Adding a virtual object

Let’s add some virtual objects to the scene. 3D capsule would be a good choice. Declare a capsuleNode of type SCNNode and give it a geometry of capsule. Give it a height of 0.1 meter, and radius of 0.03 meters.

let capsuleNode = SCNNode(geometry: SCNCapsule(capRadius: 0.03, height: 0.1

Now position it 0.1 meter left of the world origin, 0.1 meter above the world origin, and 0.1 meter away from the world origin:

capsuleNode.position = SCNVector3(0.1, 0.1, -0.1)

Now, add the node to scene:

sceneView.scene.rootNode.addChildNode(capsuleNode)

The sceneView contains a scene which is responsible for holding all the 3D objects in SCNNode format that will form the 3D scene. We are adding the capsule to the root node of the scene. Root node’s position is exactly aligned to the position of the world origin. That means its position is (0,0,0).

Currently, our viewDidLoad method looks like this:

override func viewDidLoad() {

super.viewDidLoad()

sceneView.debugOptions = [ARSCNDebugOptions.showFeaturePoints, ARSCNDebugOptions.showWorldOrigin]

sceneView.session.run(config)

let capsuleNode = SCNNode(geometry: SCNCapsule(capRadius: 0.03, height: 0.1))

capsuleNode.position = SCNVector3(0.1, 0.1, -0.1)

sceneView.scene.rootNode.addChildNode(capsuleNode)

}

Now run the app.

Cool! We have just positioned a virtual object in the real world. You can play with different positions and different geometries to explore more. Now let’s rotate the capsule 90 degree around the Z axis so that it lies flat on the X axis and changes its colour to blue.

Euler Angles

Euler Angles are responsible for a SCNNode’s display angle. We will see how to use it to rotate the capsule.

Every SCNGeometry can have materials added to it, which defines the appearance of the geometry. Materials have a diffuse property which, when set, spreads its content all over the geometry.

In viewDidLoad, add the below lines after you set the position of the capsule.

capsuleNode.geometry?.firstMaterial?.diffuse.contents = UIColor.blue //1capsuleNode.eulerAngles = SCNVector3(0,0,Double.pi/2)//2

Here, in the first line, we are setting the blue colour to the very first material of the node which will spread across the capsule and will make it look blue. In line 2, we are setting the Z Euler angle to 90 degree radians. Finally, our view loads and looks like this:

override func viewDidLoad() {

super.viewDidLoad()

sceneView.debugOptions = [ARSCNDebugOptions.showFeaturePoints, ARSCNDebugOptions.showWorldOrigin]

sceneView.session.run(config)

let capsuleNode = SCNNode(geometry: SCNCapsule(capRadius: 0.03, height: 0.1))

capsuleNode.position = SCNVector3(0.1, 0.1, -0.1)

capsuleNode.geometry?.firstMaterial?.diffuse.contents = UIColor.blue //1

capsuleNode.eulerAngles = SCNVector3(0,0,Double.pi/2)//2

sceneView.scene.rootNode.addChildNode(capsuleNode)

}

Now run the app.

Great! A blue coloured sleeping capsule on the wall! You can even add textures as diffuse contents to make an object look more realistic. We will use that in the next section when we place the tiles’ textures on the floor.

Now that we have successfully placed virtual objects on the real world, it’s time to decorate our real floor with virtual floor-tiles. To achieve the floor effect, we will use a SCNPlane geometry. SCNPlane doesn’t have any depth like other 3D geometries, which makes it a perfect fit for our app.

ARSCENEView Delegates

Before starting the floor detection, we will explore some delegate methods of our sceneView to understand what capabilities we are equipped with to interact with an ongoing AR session.

func renderer(SCNSceneRenderer, didAdd: SCNNode, for: ARAnchor)

Whenever we move or tilt our device with an AR session on in it, the ARKit tries to find different ARAnchors in the surroundings. An ARAnchor contains information about a real world position and orientation that can be used to place an object.

Once a different anchor is found, a new node gets added to the scene with the same information to accommodate this newly found anchor. This delegate method will inform us about that. We will be using it to find all the positions on the floor to place the tiles.

func renderer(_ renderer: SCNSceneRenderer, didUpdate node: SCNNode, for anchor: ARAnchor)

Most of the time, all the nodes that get added from the anchors belong to the same object. Let’s say you are moving around the floor and the device finds a number of anchors at different positions. It tries to add all the nodes for those anchors, as it thinks that all these anchors belong to different objects.

But ARKit eventually recognises that all of them belong to the same floor, so it updates the very first floor node by appending dimensions of other duplicate nodes. This delegate method will inform us about that.

func renderer(SCNSceneRenderer, didRemove: SCNNode, for: ARAnchor)

After updating the first unique node with dimensions of all other duplicate nodes, ARKit removes all the duplicate nodes and the delegate method notifies us. We will be using all of the above delegate methods in our app (and their purpose will become clearer).

Plane detection

Presently, our scene is trying to gather all anchors that it comes across, as that is the default behaviour. But since a floor is a horizontal surface, we are only interested in anchors that are on horizontal planes. So, go back to our viewDidLoad method and write the below code before running the session (that is before sceneView.session.run(config)) line.

config.planeDetection = .horizontal

In the viewDidLoad method, you can remove everything after sceneView.session.run(config) as that was for placing the capsule on screen and we don’t need that anymore. Since we will be using all the above mentioned delegate methods, we need to make our viewController a delegate of the sceneView. Before the closing brace of viewDidLoad() method, add the below line.

sceneView.delegate = self

You should get an error now, as our view controller still is not conforming the sceneView delegate. To implement this, let’s create an extension of the view controller at the end of the ViewController.swift file.

extension ViewController:ARSCNViewDelegate{}

The didAdd SCNNode delegate method will get fired every time a part of the floor is discovered and a new node gets added to the scene based on the anchor. Within this method, we will create a floor node and will add it as a child of the recently added node in the position of the anchor.

ARArchor can be of four different types to solve four different purposes. Here we are only interested in ARPlaneAnchor which detects the horizontal or vertical planes.

Creating AR floor nodes

Let’s create a function that would receive an ARPlaneAnchor as a parameter, create a floor node at the anchor’s position, and return it.

func createFloorNode(anchor:ARPlaneAnchor) ->SCNNode{

let floorNode = SCNNode(geometry: SCNPlane(width: CGFloat(anchor.extent.x), height: CGFloat(anchor.extent.z))) //1

floorNode.position=SCNVector3(anchor.center.x,0,anchor.center.z)                                               //2

floorNode.geometry?.firstMaterial?.diffuse.contents = UIColor.blue                                             //3

floorNode.geometry?.firstMaterial?.isDoubleSided = true                                                        //4

floorNode.eulerAngles = SCNVector3(Double.pi/2,0,0)                                                    //5

return floorNode                                                                                               //6

}

Let’s go through the function line by line and discuss it in more detail. Please follow each line’s description, as it’s the trickiest part.

1. We are creating a node with a geometry of SCNPlane which has the size of the anchor. ARPlaneAnchor’s extent holds the position information. The fact that the extent.z has been used as height and not extent.y, might be a little confusing. If you visualise that a 3D cube is placed on a floor and you want to make it flat along a 2D surface, you would change the y to zero and it would go flat. Now, to get the length of this 2D surface, you would consider the z, wouldn’t you? Our floor is flat, so we need a flat node not a cube.

2. We are setting the position of the node. As we don’t need any elevation, we make y zero.

3. Set the floor colour to blue.

4. The material colour will be displayed only on one side unless we specifically mention it is double-sided.

5. By default, the plane will be placed vertically. To make it horizontal, we need to rotate it by 90 degrees.

Implementing the delegate methods

Now, let’s implement the didAdd SCNNode delegate method.

func renderer(_ renderer: SCNSceneRenderer, didAdd node: SCNNode, for anchor: ARAnchor) {

guard let planeAnchor = anchor as? ARPlaneAnchor else {return} //1

let planeNode = createFloorNode(anchor: planeAnchor) //2

node.addChildNode(planeNode) //3

}

In line 1, we are checking if the anchor is a ARPlaneAnchor, since we would only deal with this type of anchor.

In line 2, a new node is getting created based on the anchor. In line 3, it’s getting added to the node.

Now in the didUpdate SCNNode delegate, we will delete all our floor nodes. We’ll do this because the dimensions of the current node have been changed and the old floor nodes won’t match. Then we will again add a fresh floor node to this updated node.

func renderer(_ renderer: SCNSceneRenderer, didUpdate node: SCNNode, for anchor: ARAnchor) {

guard let planeAnchor = anchor as? ARPlaneAnchor else {return}

node.enumerateChildNodes { (node, _) in

node.removeFromParentNode()

}

let planeNode = createFloorNode(anchor: planeAnchor)

node.addChildNode(planeNode)

}

In didRemove SCNNode delegate method, we want to clean out all our junk nodes in a civilized manner.

func renderer(_ renderer: SCNSceneRenderer, didRemove node: SCNNode, for anchor: ARAnchor) {

guard let _ = anchor as? ARPlaneAnchor else {return}

node.enumerateChildNodes { (node, _) in

node.removeFromParentNode()

}

}

Phew! Thats it! Run the app.

Adding the tile effect

Wait, what? A blue floor? No, we are not completely done yet. Just a small change and we will have a stunning floor!

To change the blue floor to tiles, we need a texture. Let’s google for a floor tile texture. I searched for “wooden floor texture” and found some beautiful texture images. Save any of them on your Mac and drag it to the Assets.xcassets.

I named it WoodenFloorTile. You can name it whatever you want. Back to the ViewController.swift file again. In the createFloorNode function, instead of setting UIColor.blue as diffuse content, make it an UIImage with the name you have given to the image in the asset folder.

floorNode.geometry?.firstMaterial?.diffuse.contents = UIImage(named: "WoodenFloorTile")

Now run the app, and wait until the world origin loads. Once the floor is detected, move around to update the node information.

Wow, You really have a gorgeous floor! You can download multiple textures and place them in a listView. This allows you to change the floor based on selected texture, as it was shown in the first part.

Download the complete project from GitHub here.

Now that you have a nice floor, you must be missing some nice furnitures to give your room a great look! We will work on that later on.