Integrating multifactor authentication capabilities into applications is crucial for strengthening their integrity. In social apps, authentication mechanisms eliminate unwanted access to personal information between two parties. Facial authentication is not entirely new, as most devices have it built-in as security measure. It offers stronger protection compared to many traditional methods, especially against risks like phishing, brute-force attacks, and account hacking.

Outline

• What to expect

• Prerequisites

• A Brief Intro to the Face API tool

• Project Setup

• Demo Project: Integrating Facial Recognition and Authentication

• Additional Information and Tips

• Conclusion

What to Expect

In this article, I’ll walk you through creating a multi-factor authentication system for a chat application powered by Stream.io, and ensuring efficient user face ID authentication to allow only authorized access to your app. I will illustrate all these with relevant code examples.

Prerequisites

Here are the necessary prerequisites to follow along with this tutorial:

• Intermediate knowledge of Node.js/Express for the backend aspect

• Knowledge of React for the frontend aspect

• Stream.io API key

Before we get started, we’ll briefly highlight the facial authentication tool of choice: Face-Api.js.

A Brief Intro to the Face API tool

Face-Api.js is a facial recognition package designed for integration with JavaScript-powered applications. It was built on top of the Tensor flow library and provides extensive facial detection based on machine learning models and abstract calculations.

In addition to all these features, it's friendly to use and can also be used locally with its predefined models. Here is a link to its documentation page, which provides relevant code examples.

It provides features such as face detection, face capture, and face match, which use the Euclidean algorithm to make precise distinctions. We'll now set it up alongside our chat application project in the next section.

Project Setup

As mentioned earlier, this is a full-stack project containing both the frontend and the backend aspects. In this section, we’ll set up both code bases before proceeding to the demo project section.

Frontend

We will power the application using the Vite framework for the frontend.

npm create vite@latest

After creating the React application, install face-api.js with this command:

npm i face-api.js

This will install the face package and the required dependencies. You can then install Stream’s powered chat SDK, which will form the main crux of the project.

npm i stream-chat stream-chat-react

After successful completion, we are finally done with the project structure scaffold. To aid ease of local testing of our frontend application, we will have to host the face models needed by the Face package locally. Here is a link to the models. Kindly copy the model's folder and paste it into the public folder in the code project. Next, we’ll set up our backend project.

Backend

The backend is built to store user details and ensure user authentication before accessing the chat application. MongoDB will be the database of choice, and we will use the Express.js library as the backend API development environment of choice. For the ease of setup, kindly clone this code-base and install it on the local PC. It comes preloaded with the necessary installation files. To further enjoy a seamless backend experience, you can utilize the MongoDB Atlas option as the database for storing user details. With that, we will now begin the code project in the next section.

Demo Project: Integrating Facial Recognition and Authentication

In this section, we will walk through setting up an authentication page on the frontend where a user can register their details, username, email, and password on the registration page. They are also obliged to take a snapshot of their face, and the face API will be called to detect a face in the image. They won't be allowed to proceed beyond this until it is successful.

Thereafter, the image faceDescriptor function is called, which generates a unique face description vector value of the user’s face based on the machine learning models provided. These values are securely stored in the MongoDB database via the Express.js backend after successfully registering. The application is coupled to a multifactor authentication system, which has both the password based authentication and the facial authentication mechanisms.

When the first hurdle (password authentication) is completed, the user is then required to take a face match, comparing it with the user's face descriptor stored from the registration page. The comparison is achieved using the Euclidean algorithmic comparison based on the threshold we provide. If it meets the threshold, the face is said to be matched, and the user gets access to the chat application; else, the user is denied access to the Stream.io-powered chat application. Relevant source code snippets highlighting these steps will be provided concurrently with images.

We’ll begin by building a defunct registration page for our chat application using React, of course. We will begin by importing and initializing the necessary packages.

import React, {useState, useRef, useEffect} from 'react';
import * as faceapi from 'face-api.js'
import {useNavigate} from 'react-router-dom'
import axios from 'axios';

const Register =()=> {

    const navigate= useNavigate("/")
    const userRef = useRef();
    const passwordRef= useRef();
    const emailRef = useRef();
    const FullRef = useRef()

In the code snippet above, we imported useful React hooks and initialized our installed Face-api.js tool. Axios will serve as our API request tool of choice for this project. The useRef hook will be used to track the user inputs. We then defined the register function and initialized the various useRef hooks for the various input fields to be inputted.

    useEffect(()=> {
const loadModels =async() => {
await faceapi.nets.tinyFaceDetector.loadFromUri('/models');
await faceapi.nets.faceLandmark68Net.loadFromUri('/models');
await faceapi.nets.faceRecognitionNet.loadFromUri('/models');
await faceapi.nets.faceExpressionNet.loadFromUri('/models');
await faceapi.nets.tinyFaceDetector.loadFromUri('/models');
setModelIsLoaded(true);
                startVideo();
}
  loadModels()  }, [])

In the code above, the useEffect hook is called to ensure that the various locally stored face-api models are initialized and active in our application. The models are stored in the models sub-folder within the public folder. Going forward, after initializing our models, we will now set up our camcorder feature on our webpage.

  const [faceDetected, setFaceDetected] = useState(false);


        // Start video feed
        const startVideo = () => {
            navigator.mediaDevices
                .getUserMedia({ video: true })
                .then((stream) => {
                    videoRef.current.srcObject = stream;
                })
                .catch((err) => console.error("Error accessing webcam: ", err));
        };
        const captureSnapshot = async () => {
            const canvas = snapshotRef.current;
            const context = canvas.getContext('2d');
            context.drawImage(videoRef.current, 0, 0, canvas.width, canvas.height);
            const dataUrl = canvas.toDataURL('image/jpeg');
            setSnapshot(dataUrl);

            // Generate the face descriptor (128-dimensional vector)
            const detection = await faceapi
                .detectSingleFace(canvas, new faceapi.TinyFaceDetectorOptions())
                .withFaceLandmarks()
                .withFaceDescriptor();

            if (detection) {
                const newDescriptor = detection.descriptor;
                setDescriptionValue(newDescriptor)
                console.log( newDescriptor);
               setSubmitDisabled(false)
                stopVid()
            } else {
                console.error("No face detected in snapshot");
            }
        };
    const stopVid =() => {

        navigator.mediaDevices
                .getUserMedia({ video: false })
                const stream = videoRef?.current?.srcObject;
        if (stream) {
            stream.getTracks().forEach(track => {track.stop()})
            videoRef.current.srcObject = null;
            setCameraActive(false)
        }
    }
        // Detect face in the video stream
        const handleVideoPlay = async () => {
            const video = videoRef.current;
            const canvas = canvasRef.current;

            const displaySize = { width: video.width, height: video.height };
            faceapi.matchDimensions(canvas, displaySize);

            setInterval(async () => {
                if (!cameraActive) return ;
                const detections = await faceapi.detectAllFaces(
                    video,
                    new faceapi.TinyFaceDetectorOptions()
                );

                const resizedDetections = faceapi.resizeResults(detections, displaySize);

                canvas.getContext('2d').clearRect(0, 0, canvas.width, canvas.height);
                faceapi.draw.drawDetections(canvas, resizedDetections);
                    const detected = detections.length > 0;
                 if (detected && !faceDetected) {
                captureSnapshot();  // Capture the snapshot as soon as a face is detected
            }

                setFaceDetected(detections.length > 0);
            }, 100);
        };

In the code above, we begin by defining a useState array when the user’s face is detected during the sign-up process. Thereafter, the function to trigger the browser camcorder is then activated. With this on, we can then trigger the handlePlayFunction in the code. This function monitors facial detection as highlighted by the face models already initialized. The stopVid function is also triggered when the user’s facial detection has been successfully completed.

In this section, we also activated the browser camcorder tool in our application to provide us with real time video. The CaptureSnapshot function helps to obtain a snapshot from the current video being showcased.

const RegSubmit = async (e) => {
  e.preventDefault();
  console.log("hello");

  try {
    const res = await axios.post(BACKEND_URL, {
      username: userRef.current.value,
      email: emailRef.current.value,
      FullName: FullRef.current.value,
      password: passwordRef.current.value,
      faceDescriptor: descriptionValue,
    });

    console.log(res.data);
    setError(false);
    navigate("/login");
    console.log("help");
  } catch (err) {
    console.error(err);
    setError(true);
  }
};

With all the values obtained, the regSubmit function is then defined. When executed, it stores the provided user details with the face description object on our backend server which can then be accessed in the next section for authentication.

Below is the full registration code.

import React, { useState, useRef, useEffect } from 'react';
import * as faceapi from 'face-api.js';
import { useNavigate } from 'react-router-dom';
import axios from 'axios';

const Register = () => {
  const navigate = useNavigate("/");

  const userRef = useRef();
  const passwordRef = useRef();
  const emailRef = useRef();
  const FullRef = useRef();
  const snapshotRef = useRef(null);
  const videoRef = useRef(null);
  const canvasRef = useRef(null);

  const [modelIsLoaded, setModelIsLoaded] = useState(false);
  const [detections, setDetections] = useState([]);
  const [error, setError] = useState(false);
  const [snapshot, setSnapshot] = useState(null);
  const [cameraActive, setCameraActive] = useState(true);
  const [submitDisabled, setSubmitDisabled] = useState(true);
  const [descriptionValue, setDescriptionValue] = useState(null);
  const [faceDetected, setFaceDetected] = useState(false);

  useEffect(() => {
    const loadModels = async () => {
      await faceapi.nets.tinyFaceDetector.loadFromUri('/models');
      await faceapi.nets.faceLandmark68Net.loadFromUri('/models');
      await faceapi.nets.faceRecognitionNet.loadFromUri('/models');
      await faceapi.nets.faceExpressionNet.loadFromUri('/models');
      await faceapi.nets.tinyFaceDetector.loadFromUri('/models');
      setModelIsLoaded(true);
      startVideo();
    };

    loadModels();
  }, []);

  const RegSubmit = async (e) => {
    e.preventDefault();
    console.log("hello");

    try {
      const res = await axios.post('http://localhost:5000/v1/users', {
        username: userRef.current.value,
        email: emailRef.current.value,
        FullName: FullRef.current.value,
        password: passwordRef.current.value,
        faceDescriptor: descriptionValue
      });

      console.log(res.data);
      setError(false);
      navigate("/login");
      console.log("help");
    } catch (err) {
      console.log(err);
      setError(true);
    }
  };

  const startVideo = () => {
    navigator.mediaDevices
      .getUserMedia({ video: true })
      .then((stream) => {
        videoRef.current.srcObject = stream;
      })
      .catch((err) => console.error("Error accessing webcam: ", err));
  };

  const stopVid = () => {
    navigator.mediaDevices.getUserMedia({ video: false });
    const stream = videoRef?.current?.srcObject;
    if (stream) {
      stream.getTracks().forEach((track) => track.stop());
      videoRef.current.srcObject = null;
      setCameraActive(false);
    }
  };

  const captureSnapshot = async () => {
    const canvas = snapshotRef.current;
    const context = canvas.getContext('2d');
    context.drawImage(videoRef.current, 0, 0, canvas.width, canvas.height);
    const dataUrl = canvas.toDataURL('image/jpeg');
    setSnapshot(dataUrl);

    const detection = await faceapi
      .detectSingleFace(canvas, new faceapi.TinyFaceDetectorOptions())
      .withFaceLandmarks()
      .withFaceDescriptor();

    if (detection) {
      const newDescriptor = detection.descriptor;
      setDescriptionValue(newDescriptor);
      console.log(newDescriptor);
      setSubmitDisabled(false);
      stopVid();

      if (storedDescriptor && isMatchingFace(storedDescriptor, newDescriptor)) {
        setInterval(alert("face matched"), 100);
      } else {
        alert("No Match Found!");
      }
    } else {
      console.error("No face detected in snapshot");
    }
  };

  const handleVideoPlay = async () => {
    const video = videoRef.current;
    const canvas = canvasRef.current;
    const displaySize = { width: video.width, height: video.height };
    faceapi.matchDimensions(canvas, displaySize);

    setInterval(async () => {
      if (!cameraActive) return;

      const detections = await faceapi.detectAllFaces(
        video,
        new faceapi.TinyFaceDetectorOptions()
      );

      const resizedDetections = faceapi.resizeResults(detections, displaySize);
      canvas.getContext('2d').clearRect(0, 0, canvas.width, canvas.height);
      faceapi.draw.drawDetections(canvas, resizedDetections);

      const detected = detections.length > 0;
      if (detected && !faceDetected) {
        captureSnapshot();
      }

      setFaceDetected(detected);
    }, 100);
  };

  return (
    <div className="flex flex-col w-full h-screen justify-center">
      <div className="flex flex-col">
        <form className="flex flex-col mb-2 w-full" onSubmit={RegSubmit}>
          <h3 className="flex flex-col mx-auto mb-5">Registration Page</h3>

          <div className="flex flex-col mb-2 w-[50%] mx-auto items-center">
            <input
              type="text"
              placeholder="Email"
              className="w-full rounded-2xl h-[50px] border-2 p-2 mb-2 border-gray-900"
              required
              ref={emailRef}
            />
            <input
              type="text"
              placeholder="Username"
              className="w-full rounded-2xl h-[50px] border-2 p-2 mb-2 border-gray-900"
              required
              ref={userRef}
            />
            <input
              type="text"
              placeholder="Full Name"
              className="w-full rounded-2xl h-[50px] border-2 p-2 mb-2 border-gray-900"
              required
              ref={FullRef}
            />
            <input
              type="password"
              placeholder="Password"
              className="w-full rounded-2xl h-[50px] border-2 p-2 mb-2 border-gray-900"
              required
              ref={passwordRef}
            />

            <div>
              {!modelIsLoaded && cameraActive && !descriptionValue ? (
                <p>Loading</p>
              ) : (
                <>
                  {!descriptionValue && (
                    <>
                      <video
                        ref={videoRef}
                        width="200"
                        height="160"
                        onPlay={handleVideoPlay}
                        autoPlay
                        muted
                      />
                      <canvas
                        ref={canvasRef}
                        width="200"
                        height="160"
                        style={{ position: 'absolute', top: 0, left: 0 }}
                      />
                      <p>
                        {faceDetected ? (
                          <span style={{ color: 'green' }}>Face Detected</span>
                        ) : (
                          <span style={{ color: 'red' }}>No Face Detected</span>
                        )}
                      </p>
                      <canvas
                        ref={snapshotRef}
                        width="480"
                        height="360"
                        style={{ display: 'none' }}
                      />
                    </>
                  )}
                </>
              )}

              {snapshot && (
                <div style={{ marginTop: '20px' }}>
                  <h4>Face Snapshot:</h4>
                  <img
                    src={snapshot}
                    alt="Face Snapshot"
                    width="200"
                    height="160"
                  />
                </div>
              )}
            </div>

            <div className="mt-2">
              <button type="button" onClick={stopVid}>
                Stop Video
              </button>
            </div>

            <button
              disabled={submitDisabled}
              className="mx-auto mt-4 rounded-2xl cursor-pointer text-white bg-primary w-[80%] lg:w-[50%] h-[40px] text-center items-center justify-center"
              type="submit"
            >
              Register
            </button>
          </div>

          <div className="flex flex-col mt-1 w-full">
            <p className="flex justify-center">
              Registered previously?&nbsp;
              <a href="/login" className="text-blue-600 underline">
                Login
              </a>
            </p>
          </div>

          {error && (
            <p className="text-red-600 text-center mt-2">
              Error while registering, try again
            </p>
          )}
        </form>
      </div>
    </div>
  );
};

export default Register;

Going forward, we will be working on our multifactor authentication system. In the code below, we will be highlighting the loginSubmit function which will be triggered when the user email and password credentials are provided for logging in to our chat application. The useRef hook is initialized which ensures that the values passed in the input boxes are parsed to the backend via the Axios request tool.

import React, { useState, useRef, useEffect } from 'react';
import { Link, useNavigate } from 'react-router-dom';
import axios from 'axios';

function Login() {
  const navigate = useNavigate();
  const userRef = useRef();
  const passwordRef = useRef();

  const [error, setError] = useState(false);

  const LoginSubmit = async (e) => {
    e.preventDefault();
    try {
      const res = await axios.post(
        'http://localhost:5000/v1/auth/login',
        {
          email: userRef.current.value,
          password: passwordRef.current.value,
        },
        { withCredentials: true }
      );

      console.log(res?.data);
      setError(false);
      navigate('/confirm-auth');
      console.log(res);
    } catch (err) {
      setError(true);
      console.log(err);
    }
  };
}

The full login page code example will be provided here. After successfully confirming their identity via the use of the password authentication feature, we can then go on to confirm the user’s identity via the use of the face recognition system.

import axios from 'axios';
import React, { useRef, useEffect, useState } from 'react';
import * as faceapi from 'face-api.js';
import { useNavigate } from 'react-router-dom';

First of all, we will set up the app by importing the necessary dependencies as highlighted in the code snippet above.

  useEffect(() => {
    const loadModels = async () => {
      await faceapi.nets.tinyFaceDetector.loadFromUri('/models');
      await faceapi.nets.faceLandmark68Net.loadFromUri('/models');
      await faceapi.nets.faceRecognitionNet.loadFromUri('/models');
      await faceapi.nets.faceExpressionNet.loadFromUri('/models');
    };

    loadModels();
  }, []);

  const handleVideoPlay = async () => {
    const video = videoRef.current;
    const canvas = canvasRef.current;

    const displaySize = { width: video.width, height: video.height };
    faceapi.matchDimensions(canvas, displaySize);

    setInterval(async () => {
      if (!cameraActive) return;

      const detections = await faceapi.detectAllFaces(
        video,
        new faceapi.TinyFaceDetectorOptions()
      );

      const resizedDetections = faceapi.resizeResults(detections, displaySize);
      canvas.getContext('2d').clearRect(0, 0, canvas.width, canvas.height);
      faceapi.draw.drawDetections(canvas, resizedDetections);

      const detected = detections.length > 0;
      if (detected && !faceDetected) {
        captureSnapshot();
      }

      setFaceDetected(detected);
    }, 100);
  };

  const startVideo = () => {
    navigator.mediaDevices
      .getUserMedia({ video: true })
      .then((stream) => {
        videoRef.current.srcObject = stream;
      })
      .catch((err) => console.error("Error accessing webcam: ", err));
  };

  const stopVid = () => {
    const stream = videoRef.current.srcObject;
    if (stream) {
      stream.getTracks().forEach((track) => track.stop());
      videoRef.current.srcObject = null;
      setCameraActive(false);
    }
  };

  const deleteImage = () => {
    setSnapshot(null);
    setDescriptionValue(null);
    setFaceDetected(false);
    setCameraActive(true);
    startVideo();
  };

  const captureSnapshot = async () => {
    const canvas = snapshotRef.current;
    const context = canvas.getContext('2d');
    context.drawImage(videoRef.current, 0, 0, canvas.width, canvas.height);

    const dataUrl = canvas.toDataURL('image/jpeg');
    setSnapshot(dataUrl);
    stopVid();

    const detection = await faceapi
      .detectSingleFace(canvas, new faceapi.TinyFaceDetectorOptions())
      .withFaceLandmarks()
      .withFaceDescriptor();

    if (detection) {
      const newDescriptor = detection.descriptor;
      setDescriptionValue(newDescriptor);
      console.log(newDescriptor);
    }
  };

After initializing all the necessary dependencies, we also imported our models as we did in the registration page to detect the user’s face and then generate a face description. We also allowed for the user to delete the snapshot and retake the image as many times as possible.

  const FaceAuthenticate = async (e) => {
    e.preventDefault();

    try {
      const res = await axios.post(
        'http://localhost:5000/v1/auth/face-auth',
        { faceDescriptor: descriptionValue },
        { withCredentials: true }
      );

      console.log(res?.data);
      navigate('/chat');
    } catch (err) {
      console.log(err);
    }
  };

After the face descriptor object gets generated, we then sent it to the backend to compare it with the stored face descriptor obtained at the point of registration. If they match, we get redirected to the chat application. Otherwise, an appropriate error message denying us access to the chat application is displayed.

Here is the code to the FaceAuth page:

import axios from 'axios';
import React, { useRef, useEffect, useState } from 'react';
import * as faceapi from 'face-api.js';
import { useNavigate } from 'react-router-dom';

const FaceAuth = () => {
  const navigate = useNavigate("/");

  const videoRef = useRef(null);
  const canvasRef = useRef(null);
  const snapshotRef = useRef(null);

  const [cameraActive, setCameraActive] = useState(true);
  const [snapshot, setSnapshot] = useState(null);
  const [descriptionValue, setDescriptionValue] = useState(null);
  const [faceDetected, setFaceDetected] = useState(false);

  useEffect(() => {
    const loadModels = async () => {
      await faceapi.nets.tinyFaceDetector.loadFromUri('/models');
      await faceapi.nets.faceLandmark68Net.loadFromUri('/models');
      await faceapi.nets.faceRecognitionNet.loadFromUri('/models');
      await faceapi.nets.faceExpressionNet.loadFromUri('/models');
    };

    loadModels();
  }, []);

  const handleVideoPlay = async () => {
    const video = videoRef.current;
    const canvas = canvasRef.current;

    const displaySize = { width: video.width, height: video.height };
    faceapi.matchDimensions(canvas, displaySize);

    setInterval(async () => {
      if (!cameraActive) return;

      const detections = await faceapi.detectAllFaces(
        video,
        new faceapi.TinyFaceDetectorOptions()
      );

      const resizedDetections = faceapi.resizeResults(detections, displaySize);
      canvas.getContext('2d').clearRect(0, 0, canvas.width, canvas.height);
      faceapi.draw.drawDetections(canvas, resizedDetections);

      const detected = detections.length > 0;
      if (detected && !faceDetected) {
        captureSnapshot();
      }

      setFaceDetected(detected);
    }, 100);
  };

  const startVideo = () => {
    navigator.mediaDevices
      .getUserMedia({ video: true })
      .then((stream) => {
        videoRef.current.srcObject = stream;
      })
      .catch((err) => console.error("Error accessing webcam: ", err));
  };

  const stopVid = () => {
    const stream = videoRef.current.srcObject;
    if (stream) {
      stream.getTracks().forEach((track) => track.stop());
      videoRef.current.srcObject = null;
      setCameraActive(false);
    }
  };

  const deleteImage = () => {
    setSnapshot(null);
    setDescriptionValue(null);
    setFaceDetected(false);
    setCameraActive(true);
    startVideo();
  };

  const captureSnapshot = async () => {
    const canvas = snapshotRef.current;
    const context = canvas.getContext('2d');
    context.drawImage(videoRef.current, 0, 0, canvas.width, canvas.height);

    const dataUrl = canvas.toDataURL('image/jpeg');
    setSnapshot(dataUrl);
    stopVid();

    const detection = await faceapi
      .detectSingleFace(canvas, new faceapi.TinyFaceDetectorOptions())
      .withFaceLandmarks()
      .withFaceDescriptor();

    if (detection) {
      const newDescriptor = detection.descriptor;
      setDescriptionValue(newDescriptor);
      console.log(newDescriptor);
    }
  };

  const FaceAuthenticate = async (e) => {
    e.preventDefault();

    try {
      const res = await axios.post(
        'http://localhost:5000/v1/auth/face-auth',
        { faceDescriptor: descriptionValue },
        { withCredentials: true }
      );

      console.log(res?.data);
      navigate('/chat');
    } catch (err) {
      console.log(err);
    }
  };

  return (
    <>
      <div className="flex w-full h-screen flex-col justify-center">
        <p className="flex flex-col mx-auto items-center text-lg font-semibold mb-3">
          Take a snapshot to confirm your identity
        </p>
        <p className="text-center mb-4">Ensure that the picture is taken in a bright area</p>

        <button
          onClick={startVideo}
          className="flex w-[30%] mx-auto text-center items-center justify-center mb-5 h-[40px] bg-blue-600 rounded-md text-white"
        >
          Turn on Webcam
        </button>

        {!snapshot ? (
          <>
            <video
              className="flex mx-auto items-center rounded-md"
              ref={videoRef}
              width="240"
              height="180"
              onPlay={handleVideoPlay}
              autoPlay
              muted
            />
            <canvas
              ref={snapshotRef}
              width="240"
              height="180"
              style={{ position: 'absolute', top: 0, left: 0 }}
            />
            <button onClick={captureSnapshot} className="mt-4 mx-auto block text-sm text-blue-600 underline">
              Take a snapshot
            </button>
          </>
        ) : (
          <div className="flex w-full justify-center">
            <img
              src={snapshot}
              className="rounded-lg"
              width="240"
              height="180"
              alt="Face Snapshot"
            />
          </div>
        )}

        <div className="flex flex-row w-full justify-evenly mt-5">
          <button
            onClick={deleteImage}
            className="bg-purple-500 text-white p-2 h-[35px] rounded-lg"
          >
            Delete Image
          </button>
          <button
            onClick={FaceAuthenticate}
            className="bg-purple-500 text-white p-2 h-[35px] rounded-lg"
          >
            Upload Image
          </button>
        </div>
      </div>
    </>
  );
};

export default FaceAuth;

Displayed below is how the face authentication page should look like.

Having set up the frontend, let's head to the backend and configure the registration and login endpoint for our project. The entire code to the backend project can be gotten here. We will only be highlighting the faceAuth backend function in this article.

To verify authentication, we will be using the sessions option instead of the JWT option. Important user information will be stored and accessed in the session cookies attached to the requests and responses to the frontend. Here is the faceAuth function:

const faceAuth = async (req, res) => {
  try {
    console.log(req.session);

    const id = req.session.passport?.user;
    console.log(id);


    const user = await User.findById(id);
    console.log(user);

    if (user == null) {
      return res.status(400).json({ err: "User not found" });
    }


  } catch (err) {
    console.error(err);
    res.status(500).json({ err: "Internal Server Error" });
  }
};

First, we defined an asynchronous function named faceAuth. We then obtained the unique ID of the user who had successfully scaled over the initial login process from the request session.

To confirm the similarity of the user's stored face descriptor and the picture sent from the frontend, we utilized the matching face function based on the Euclidean algorithm to confirm the user's identity as done below.

const isMatchingFace = (descriptor1, descriptor2, threshold = 0.6) => {
  // Convert the stored descriptors to Float32Array if they aren't already
  if (!(descriptor1 instanceof Float32Array)) {
    descriptor1 = new Float32Array(Object.values(descriptor1));
  }

  if (!(descriptor2 instanceof Float32Array)) {
    descriptor2 = new Float32Array(Object.values(descriptor2));
  }

  const distance = faceapi.euclideanDistance(descriptor1, descriptor2);
  console.log("Euclidean Distance:", distance);

  return distance < threshold;
};

As stated in the code above, the threshold of similarity of comparison used was 0.6. This is flexible and can be modified to suit the user's preference, as a higher threshold will provide better accuracy overall.
If the function returns true, then the user has been successfully authenticated and can then have access to our chat application. Here is the full code snippet.

const faceAuth = async (req, res) => {
  try {
    console.log(req.session);

    const id = req.session.passport?.user;
    console.log(id);

    const { faceDescriptor } = req.body;
    const user = await User.findById(id);
    console.log(user);

    if (user == null) {
      return res.status(400).json({ err: "User not found" });
    }

    const isMatchingFace = (descriptor1, descriptor2, threshold = 0.6) => {
      // Convert the stored descriptor (object) to a Float32Array
      if (!(descriptor1 instanceof Float32Array)) {
        descriptor1 = new Float32Array(Object.values(descriptor1));
      }

      if (!(descriptor2 instanceof Float32Array)) {
        descriptor2 = new Float32Array(Object.values(descriptor2));
      }

      const distance = faceapi.euclideanDistance(descriptor1, descriptor2);
      console.log("Euclidean Distance:", distance);

      return distance < threshold;
    };

    if (isMatchingFace(faceDescriptor, user.faceDescriptor)) {
      console.log("Face match successful");
      req.session.mfa = true;

      return res.status(200).json({
        msg: "User authentication was successful. Proceed to the chat app.",
      });
    } else {
      return res.status(401).json({ msg: "Face does not match. Access denied." });
    }
  } catch (err) {
    console.log(err);
    res.status(500).json({
      err: "User face couldn't be authenticated. Please try again later",
    });
  }
};

With the main hurdle completed, we can then navigate to our application and have a seamless chat experience.

Additionally, as a safety measure, a rate limiter is also in place to minimize the use of brute-force techniques by malicious individuals to gain access to the chat application.

Additional Information and Tips

The overall aim of these efforts is to achieve a more scalable and secure method of user validation. The threshold can easily be modified and tweaked to improve application accuracy. Alternatively, the AWS Rekognition tool can sufficiently replace the Face API tool with efficient cloud-powered models. The limitations of facial recognition can also be overcome by exploring biometric authentication, as it’s a known fact that each individual's fingerprint is unique, greatly reducing the risk of user compromise.

Conclusion

So far, we have walked through the process of creating an efficient multi-factor facial authentication-based tool to prevent intruder access to our chat application, ensuring and prioritizing the highest level of user privacy. Need an SDK that assures you of a seamless and secure chat experience? Try Stream.io today.

With the advent of Web 2.0, authenticating users became a crucial task for developers.

Before Web 2.0, website visitors could only view the content of a web page – there was no interaction. This era of the internet was called Web 1.0.

But after Web 2.0, people gained the ability to post their own content on a website. And then content moderation became a never-ending task for website owners.

To reduce spam on these websites, developers introduced user authentication systems. Now website moderators can easily know the source of spam and can prevent those spammers from accessing the website further.

If you are want to know how to implement content moderation on your website, you can read my article on How to detect and blur faces in your web applications.

Now let's see what we'll be getting into in this tutorial.

What You'll Learn in This Tutorial

In this tutorial, we will discuss different authentication techniques you can use to authenticate users. These include email-password authentication, phone auth, OAuth, passwordless magic links, and at last facial authentication.

Our primary focus will be on authentication via face recognition techniques in this article.

We'll also build a project that teaches you how to integrate facial recognition-based authentication in your React web application.

In this project, we'll use the FaceIO SaaS (software as a service) platform to integrate facial recognition-based authentication. So, make sure you set up a free FaceIO account to follow along.

And finally, we'll take a look at the user privacy aspect and discuss how face recognition doesn't harm your privacy. We'll also talk about whether it's a reliable choice for developers in the future.

This article is packed with information, hands-on projects, and discussions. Grab a cup of coffee and a slice of pizza 🍕 and let's get started.

The final version of this project looks like this. Looks interesting? Let's do it then.

Different Types of User Authentication Systems

There are many user authentication systems out there right now that you can choose to implement in your websites. There are no real superior or inferior auth techniques. All of these auth systems depend on using the right tool for the job.

For example, if you are making a simple landing page to collect emails from users, there is no need to use OAuth. But if you are building a social platform, then using OAuth makes more sense than traditional authentication. You can pull the user's details and profile images directly from OAuth.

If your web application is built around any investment-related content or legally binding services, then using phone auth makes more sense. A user can create unlimited email accounts but they'll have limited phone numbers to use.

Let's take a look at some popular authentication systems so we can see their pros and cons.

Email-password based authentication

Email-password-based authentication is the oldest technique for verifying a user. The implementation is also very simple and easy to use.

The pro of this system is you don't need to have a third-party account to log in. If you have an email, whether it is self-hosted or from a service (like Gmail, Outlook, and so on), you are good to go.

The primary con of this system is you need to remember all of your passwords. As the number of websites is constantly growing and we need to log in to most sites to access our profiles, remembering passwords for every site becomes a daunting task for us humans.

Coming up with a unique and strong password is also a huge task. Our brains aren't typically capable of memorizing many random strings of letters and numbers. This is the biggest drawback of email-password-based authentication systems.

Phone authentication

Phone authentication is generally a very reliable auth technique to verify a user's identity. As a user typically doesn't have more than one phone number, this can be best suited for assets-related websites where user identity is very important.

But the drawback of this system is people don't want to reveal their phone numbers if they don't trust you. A phone number is much more personal than an email.

One more important factor of phone authentication is its cost. The cost of sending a text message to a user with an OTP is high compared to email. So website owners and developers often prefer to stick with email auth.

OAuth-based authentication

OAuth is a relatively new technique compared to the previous two. In this technique, OAuth providers user authentication and useful information on behalf of the user.

For example, if the user has an account with Google (for example), they can log in to other sites directly using their Google account. The website gets the user details details from Google itself. This means that there's no need to create multiple accounts and remember every password for those accounts.

The major drawback of this system is that you as a developer have to trust the OAuth providers and many people don't want to link all their accounts for privacy reasons. So you'll often see an email-password field in addition to OAuth on most websites.

Magic link authentication

Magic links solve most of the problems you face in email password-based authentication. Here you have to provide only your password and you will receive an email with an auth link. Then you have to open this link in your browser and you are done. No need to remember any passwords.

This type of authentication has gained in popularity these days. It saves a lot of time for the user, and it's also very cheap. And you don't have to trust a 3rd-party like in the case of OAuth.

Facial recognition authentication

Facial recognition is one of the latest authentication techniques, and many developers are adopting it these days. Facial recognition reduces the hassle of entering your email-password or any other user credentials to log in to a web application.

The most important thing is that this authentication system is fast and doesn't need any special hardware. You just need a webcam, which almost all devices have nowadays.

Facial recognition technology uses artificial intelligence to map out the unique facial details of a user and store them as a hash (some random numbers and text with no meaning) to reduce privacy-related issues.

Building and deploying an artificial intelligence-based face recognition model from scratch is not easy and can be very costly for indie developers and small startups. So you can use SaaS platforms to do all this heavy-lifting for you. FaceIO and AWS recognition are examples of these type of services you can use in your projects.

In this hands-on project, we are going to use FaceIO APIs to authenticate a user via facial recognition in a React web application. FaceIO gives you an easy way to integrate the authentication system with their fio.js JavaScript library.

Project Setup

Before starting, make sure to create a FaceIO account and create a new project. Save the public ID of your FaceIO project. We need this ID later in our project.

To make a React.js project, we will use Vite. To start a Vite project, navigate to your desired folder and execute the following command:

npm create vite@latest

Then follow the instructions and create a React app using Vite. Navigate inside the folder and run npm insall to install all the dependencies for your project.

After following all these steps, your project structure should look like this:

.
├── index.html
├── package.json
├── package-lock.json
├── public
│   └── vite.svg
├── src
│   ├── App.css
│   ├── App.jsx
│   ├── assets
│   │   └── react.svg
│   └── main.jsx
└── vite.config.js

How to Integrate FaceIO into Our React Rroject

To integrate FaceIO into our project, we need to add their CDN in the index.html file. Open the index.html file and add the faceIO CDN before the root component. To learn more, check out FaceIO's integration guide.

<body>    
    <script src="https://cdn.faceio.net/fio.js"></script>
    <div id="root"></div>
    <script type="module" src="/src/main.jsx"></script>
</body>

Now remove all the code from the App.jsx file to start from scratch. I've kept this tutorial as minimal as possible. So I've only added a heading and two buttons in the UI to demonstrate how the FaceIO facial authentication process works.

Here, one button works as a sign-in button, and the other one works as a log-in button.

The code inside the App.jsx file looks like this:

import "./App.css";
function App() {
  return (
    <section>
      <h1>Face Authentication by FaceIO</h1>
      <button>Sign-in</button>
      <button>Log-in</button>
    </section>
  );
}

export default App;

How to Register a User's Face using FaceIO

Working with FaceIO is very fast and easy. As we are using the fio.js library, we have to execute only one helper function to authenticate a user. This fio.js library will do most of the work for us.

To register a user, we initialize our FaceIO object inside a useEffect hook. Otherwise, every time a state changes, it re-runs the components and reinitializes the faceIO object.

let faceio;
useEffect(() => {
    faceio = new faceIO("Your Public ID goes here");
}, []);

Your FaceIO public ID is located on your FaceIO console. Copy the public ID and paste it here to initialize your FaceIO object.

Now, define a function named handleSignIn(). This function contains our user registration logic.

Inside the function call the enroll method of the faceIO object. This enroll method is equivalent to the sign-up function in a standard password backed registration system and accepts a payload argument. You can add any user-specific information (for example their name or email address) to this payload.

This payload information will be stored along with the facial authentication data for future reference. To learn about other optional arguments, check out their API docs.

In our sign-in button, on user click we invoke this handleSignIn() function. The code snippets for user sign-in look like this:

const handleSignIn = async () => {
    try {
      let response = await faceio.enroll({
        locale: "auto",
        payload: {
          email: "example@gmail.com",
          pin: "12345",
        },
      });

      console.log(` Unique Facial ID: ${response.facialId}
      Enrollment Date: ${response.timestamp}
      Gender: ${response.details.gender}
      Age Approximation: ${response.details.age}`);
    } catch (error) {
      console.log(error);
    }
  };

<button onClick={handleSignIn}>Sign-in</button>

FaceIO screen

How to Sign In using Face Recognition

After registering the user, then you'll need to get the user into the authentication or log-in/sign-in flow. Using the fio.js library also makes it very easy for us to set up a log-in flow using face authentication.

We have to invoke the authenticate method of the faceIO object which is equivalent to the sign-in function in a standard password backed registration system and all the critical work will be done by the fio.js package.

At first, define a new function named handleLogIn() to handle all the log-in logic in our React app. Inside this function, we invoke the authenticate method of the faceIO object as I mentioned earlier.

This method accepts a locale argument. This is the default language of the interaction of users with FaceIO widgets. If you are not sure, you can assign auto in this field.

The authenticate method also take more optional arguments like permissionTimeout, idleTimeout, replyTimeout and so on. You can check out their API documentation to know more about optional arguments.

We invoke this handleLogIn() function when someone clicks on the Log-in button:

const handleLogIn = async () => {
    try {
      let response = await faceio.authenticate({
        locale: "auto",
      });

      console.log(` Unique Facial ID: ${response.facialId}
          PayLoad: ${response.payload}
          `);
    } catch (error) {
      console.log(error);
    }
  };

<button onClick={handleLogIn}>Log-in</button>

Our user authentication project using FaceIO and React is now complete! You learned how to register and login a user. You can see the process is fairly simple compared to implementing an email-password based or some other authentication method we discussed earlier in this article.

Now you can style all the jsx elements using CSS. I didn't add CSS here to reduce complexity in this project. If you are curious, you can take a look at my GitHub gist.

If you want to host this React FaceIO project for free, you can check out this article on how to deploy your React and Nextjs project in Cloudflare pages.

How to Use the FaceIO REST API

Besides providing widgets via the fio.js library, FaceIO also provides REST APIs to streamline the authentication process.

Every application in the FaceIO console has an API key. You can use this API key to access the FaceIO REST API endpoints. The base URL for the REST API is https://api.faceio.net/.

The URL schema accepts URL parameters like this https://api.faceio.net/cmd?param=val&param2=val2. Here cmd is an API endpoint and param is an endpoint parameter if it accepts any.

Using the REST API endpoints, you can automate various tasks in your backend.

1. You can delete a face ID on a user's request.
2. You can attach a payload with a face ID.
3. You can change the PIN associated with a face ID.

This REST API is intended to be used purely on the server side. Make sure you don't expose it to clients. It's important that you read the following Privacy and Security sections to learn more about this.

How to Use FaceIO WebHooks

Webhooks are event-driven communication systems among servers. You can use this webhook feature of FaceIO to update and sync your backend with new events happening in your front-end web application.

The event of this webhook fires on new user enrollment, facial authentication success, facial ID deletion, and so on.

You can set up FaceIO webhooks in your project console. A typical webhook call from FaceIO lasts for 6 seconds. This contains all the information about a specific event in a JSON format and looks like this:

{
  "eventName":"String - Event Name",
  "facialId": "String - Unique Facial ID of the Target User",
  "appId":    "String - Application Public ID",
  "clientIp": "String - Public IP Address",
  "details": {
     "timestamp": "Optional String - Event Timestamp",
     "gender":    "Optional String - Gender of the Enrolled User",
     "age":       "Optional String - Age of the Enrolled User"
   }
}

Privacy and FaceIO

Privacy is the most important thing for a user nowadays. As big corporations use your data for their good, questions arise on whether the privacy of these face recognition techniques is valid and legitimate.

FaceIO as a service follows all the privacy guidelines and gets user consent before requesting their camera access. Even if the developer wanted, FaceIO doesn't scan faces without getting consent. Users can easily opt-out of the system and can delete their facial data from the server.

FaceIO is CCP and GDPR compliant. As a developer, you can release this facial authentication system anywhere in the world without facing privacy issues. You can read this article to know more about FaceIO privacy best practices.

FaceIO Security

The security of a web application is an important topic to discuss and consider. As a developer, you are responsible for the security of a site or application's users.

FaceIO follows some important and serious security guidelines for user data protection. FaceIO hashes all the unique facial data of the user along with the payload we specified earlier. So the stored information is nothing but some random strings which can't be reverse engineered.

FaceIO outlines some very important security guidelines for developers. Their security guide focuses on adding a strong pin code to protect user data. FaceIO also rejects covered faces so that no one can impersonate someone else.

Conclusion

If you've read this far, thank you for your time and effort. Make sure to follow along with the hands-on tutorial so you can fully grasp the topic.

The project should be approachable if you follow all the steps. If you make something out of it, show me on Twitter. If you have any questions, please ask. I will happy to help you. Till then, have a good day.

The camera is finally on stage to help solve user experience (UX) design, technology, and communication issues.

Years after the Kinect was trashed and Google Glass failed, there is now new hope. The impressive technological array that Apple minimized from a PrimeSense to the iPhone X is the beginning of emotion-dependent interactions.

It’s not new. It’s commercialized and gives developers access to indispensable information.

Recently, Mark Zuckerberg mentioned that much of Facebook’s focus will be on the camera and its surrounding environment. Snapchat has defined itself as a camera company. Apple and Google are also heavily investing in cameras. The camera has tremendous power that we have not yet tapped into. It has the power to detect emotions.

Inputs need to be easy, natural, and effortless

When Facebook first introduced emojis as an enhanced reaction to Like, I realized that they were onto something. Facebook recently added five emotions which helped Facebook to better understand its users’ emotional reactions to its content. I argue that the emojis are glorified form of the same thing, but one that works better than anything else.

In the past, Facebook only had the Like button while YouTube had the Like and Dislike buttons. But these are not enough to track emotions, and do not provide much value to researchers and advertisers. Most people express their emotions in comments, and yet there are more _Like_s than comments.

The comments are text based or even presented with an image, which is harder to analyze. That is because there are many contextual connections the algorithm needs to guess. For example, how familiar is the person who reacts to a post with the person who posted it, and vice versa? How is the person connected to the specified subject?

Is there subtext, slang, or anything related to the person’s experience? Is it a continued conversation from the past? Facebook did a wonderful job of keeping the conversation positive. Facebook prevented the Dislike button from pulling focus, which could have discouraged people from creating and sharing content. Facebook kept it positively pleasant.

Now I would compare Facebook to a glorified forum. Users can reply to comments or to the emojis. Users can Like a Like ?. Yet it is still very hard to know what people are feeling. Most people who read don’t leave a comment. What do readers feel when they read a post then?

The old user experience for cameras

What do you do with a camera? Take pictures and videos, and that’s about it. There has been huge development in camera apps. There are many features that are related to the surroundings of the main use case, for example high dynamic range (HDR), slow motion, portrait mode, etc.

_[image source](https://twitter.com/lukew/status/522056776477200384" rel="noopener" target="blank" title=")

Based on the enormous number of pictures users generate, a new wave of smart galleries, photo processing, and metadata apps has been created.

Photography from the Mac App Store

However, recently the focus has changed. It is now on the life-integrated camera, which is a combination of the strongest treats and best use cases for mobile phones. The next generation of cameras will be fully integrated into our lives and could replace all the input icons in a messaging app (microphone, camera, and location).

Cameras are one of the three items that has been consistently developed at a dizzying pace. The screen and the processor are the other two. Every phone that has come out has pushed the limits, and has done so year after year. The improvements made to the cameras are to their megapixels, movement stabilization, aperture, speed and, as mentioned above, the apps.

Let’s evaluate the evolution of a few products created by companies.

This is just a glimpse of the upgrade to megapixels (MP), not including double cameras, flash, etc. There have been many software changes.

Much of the development was focused on the camera at the back of the phone because, at least at first, the front camera was thought to be useful for video calls only. However, selfie culture and Snapchat changed that. Snapchat’s masks, which were later copied by everyone else, are a huge success. Face masks are not new. Google introduced them a while ago, but Snapchat was effective at growing the use of masks.

Highlights from memory lane

In December 2009, Google introduced Google Goggles. It was the first time that users could use their phone to get information about the environment around them. The information was mainly about landmarks initially.

In November 2011, Samsung Nexus introduced facial recognition as a way to unlock phones. Like many things done for the first time, it wasn’t very good and was later scrapped.

Samsung (Google) Nexus

In February 2013, Google released Google Glass. It had more use cases because it was able to receive input not just from the camera but also from other items like the voice. The Google Glass was always there, but it failed to gain traction because it was too expensive, looked unfashionable, and triggered a backlash from the public. It was just not ready for prime time.

Google Glass 1

So far, devices have only limited information at their disposal. The information they have are audio visual with GPS and historical data. But it limited Google Glass from displaying the information on the small screen that is positioned near the users’ eye. The screen blocked users from looking at anything else. Putting this technology on a phone for external use is not just a technological limitation but also a physical one.

When you focus on your phone, you cannot see anything else. Your field of view is limited. This is similar to the field of view in user experience principles for virtual reality (VR). That’s why there are cities that have created routes for people who use their phones while walking and have set up traffic lights that help people walk and text. A premise like Microsoft’s HoloLens is more aligned with the spatial environment and can actually help users while they move and use their phones, rather than absorb their attention and put them in danger.

_[Kinect](https://en.wikipedia.org/wiki/Kinect" rel="noopener" target="blank" title=") bought by Apple Inc.

Microsoft HoloLens

In July 2014, Amazon introduced the Fire Phone. It featured four cameras at the front of the phone. This was a breakthrough, even though it didn’t succeed. The four frontal cameras were used for scrolling and created 3D effects based on the accelerometer and users’ gaze. It was the first time that a phone used its front cameras as an input method from users.

The Fire Phone

On August 2016, Samsung’s Note 7 was launched. It allowed users to unlock their phones with iris scanning. Samsung resurrected a facial-recognition technology that had rested on the shelf for six years. Unfortunately, just looking at the tutorial can be vexing. Samsung didn’t do much user experience testing for that feature.

It is disturbing to hold this huge phone and put it at a 90° angle to your face. It is not something that anyone should be doing while they are walking on the street. It can work nicely for Saudi women who cover their faces, but because of manufacturing defects, many Note 7 phones overheated, combusted, or exploded. The concept of iris scanning was put on hold for another full year until the Note 8 came out.

From Samsung’s keynote

But by that time no one mentioned iris scanning. The Note 8 mentioned that iris scanning is an another way of unlocking a phone in conjunction with the fingerprint sensor. It’s probably because the phone was not good enough or Samsung wasn’t able to make a decision (similar to the release of the Galaxy 6 and 6 Edge). For a product to succeed, it needs to have multiple functions, otherwise it risks being forgotten.

Google took a break, and then in July 2017 released the second version of Google Glass as a business-to-business product. The use cases became more specific for some industries.

Google Glass 2

Now Google is about to release the Google Lens to bring the initial Goggles use case to the present. It’s Google’s effort to learn how to use visual information with additional context, and to figure out what product to develop next. It appears that Google is leaning towards a camera that users can wear.

Google Lens App

There are other companies that are exploring visual input as well. For example, Pinterest is seeing a huge demand for its visual search lens, which its users are using to search for items to buy and to help people curate products and services online.

Pinterest Visual Search

Snapchat’s spectacles allow users to record short videos easily (even though the upload process is cumbersome).

Snap’s Specs

Now facial recognition is also on the Note 8 and Galaxy 8, but this feature is not panning out as well as hoped.

Galaxy S8 facial recognition

To check out a facial recognition demonstration, click here.

Apple is slow to adopt new technologies relative to its competitors. But on the other hand, Apple commercializes its products, for example, the Apple Watch. The Watch was all about facial recognition and finite screen. There is no better way to make people use this feature than by removing all other options (like the Touch ID). It’s not surprising that Apple did this last year with wireless audio (Apple removed the headphone jack) and USB C on the MacBook Pro (by removing everything else).

There is a much bigger reason why Apple chose this technology at this time. It has to do with its augmented reality efforts.

Face ID has challenges that include recognizing users who wear the Niqāb (face covers), users who have had plastic surgery, and users who are changing physically because they are growing. But the bigger picture here is much more interesting. This is the first time that users can do something that they naturally do with no effort, while receiving data that is meaningful for the future of technology. I believe that a screen that can read fingerprints is better. It appears that Samsung is heading in that direction (although rumours has it that Apple tried and failed).

So where is this going? What’s the target?

In the past, companies used special glasses and devices to perform user testing. The only output they could give were heat maps. Yet they weren’t able to document what the users’ were focusing on or their emotions and reactions.

Tobii Pro glasses — is one example

Based on tech trends, it appears the future involves augmented reality and virtual reality. But in my opinion, it includes audio, 3D sound, and visual inputs combined. This would be a wonderful experience, which would allow users to look at anything, anywhere, and get information at the same time.

What if we are able to know where the users are looking at, and what they are focusing on? For years this is something that marketing and design professionals have tried to capture and analyze. What can do that better than the set of arrays a device like the iPhone X has as a starting point? Later on this evolved into glasses that can see what the user is focused on.

Reactions are powerful and addictive

Reactions help people converse and increase retention and engagement. Some apps offer reactions to posts as messages that can be sent to friends. There are funny videos on YouTube that show the reactions of people who watch videos. There is even a TV show, called Gogglebox, that is dedicated to showing people watching TV.

In IO Google, the annual developer festival, Google opened the option to pay creators for their platform. Its like what the brilliant Patreon site is doing but in a much more dominant way. A way that helps you to stand out from the crowd and grab the creator’s attention is SuperChat.

In Chris Harrison’s student project from 2009, Harrison created a keyboard that had pressure sensing keys. Depending on the force with which users type, the keyboard reads the users’ emotions and determine if they are angry or excited. The letters too get bigger as a result. Now imagine combining it with a camera that sees the users’ facial expressions while they are typing, as people tend to express their emotions while they’re typing a message.

How would such a UX look like

Consider the pairing of a remote and center point in virtual reality. The center is our focus, but we also have a secondary focus point, which is where the remote points are. However, this type of user experience cannot work in augmented reality. To take advantage of augmented reality, which is a new focus for Apple, the user’s focus must be known.

_[illustration source](https://blog.kickpush.co/beyond-reality-first-steps-into-the-unknown-cbb19f039e51" rel="noopener" target="blank" title=")

What started as ARKit and Google’s ARCore SDK, both will be the future of development. This is because of the amazing output and input both can get from the front and back cameras combined. This will allow for a greater focus on the input.

A more critical view on future developments

While Apple opened the way for facial recognition and triggered reactionary Animoji, it is going to get interesting when other organizations start implementing Face ID. Currently, it is manifested in a basic and harmless way, but the goal remains to get more information. Information that will be used to keep track of and sell products to us. It will also allow us to learn more about us and gather our emotional data.

Animoji

It is important to say that the front camera doesn’t come alone. It’s the expected result of Apple buying PrimeSense. The array of front-facing technology includes an infrared camera, depth sensor, etc. (I think they could do well with a heat sensor too.) It’s not that someone will keep videos of our faces using the phone, but rather there will be a scraper that will document all the information about our emotions.

Can’t be fooled by a mask — from Apple’s keynote

Or funny enough

Summary

It’s exciting that augmented reality have algorithms that can read faces. There are many books that talks about how to identify facial reactions, but now it’s time for technology to do this. It will be wonderful for many reasons. For example, robots can now see how we feel and react, or with glasses, we can get more context about what we need them to do. Relating to computers, it’s better to look at a combination of elements because that helps the machine to understand you better.

The things that you can do if you have the information about what the user is focusing on are endless. It’s the dream of every person who works with technology.

This blog post was originally showed here.