Chapters
Chapters

Chapters

Hide chapters

Apple Augmented Reality by Tutorials

Before You Begin

Section 0: 5 chapters
Show chapters Hide chapters

Section I: Reality Composer

Section 1: 5 chapters
Show chapters Hide chapters

Section II: Reality Composer & Image Tracking

Section 2: 2 chapters
Show chapters Hide chapters

Section III: Reality Composer & Object Tracking

Section 3: 1 chapter
Show chapters Hide chapters

Section IV: RealityKit & Face Tracking

Section 4: 3 chapters
Show chapters Hide chapters

Section V: ARKit & SpriteKit

Section 5: 3 chapters
Show chapters Hide chapters

Section VI: ARKit & SceneKit

Section 6: 2 chapters
Show chapters Hide chapters

Section VII: ECS & Collaborative Experiences (Bonus Section)

Section 7: 1 chapter
Show chapters Hide chapters

Section VIII: Creating 3D Models With Object Capture (Bonus Section))

Section 8: 2 chapters
Show chapters Hide chapters

10. Face Anchors
Written by Chris Language

Heads up... You're reading this book for free, with parts of this chapter shown beyond this point as scrambled text.

You can unlock the rest of this book, and our entire catalogue of books and videos, with a kodeco.com Professional subscription.

Unlock now

This chapter continues from the point where the previous one left off. Thanks to SwiftUI, the AR Funny Face app now sports a very basic UI. In this chapter, you’ll continue to focus on that facial recognition component by using face anchors in RealityKit.

You’ll also get to build some funny scenes with these crazy props:

A humongous pair of sunglasses, a glass eyeball and one epic mustache. With these cool props at one’s disposal, the AR Funny Face app is off to a great start.

Note: Feel free to continue with your own final project from the previous chapter. If you skipped a few things, load the starter project from starter/ARFunnyFace/ARFunnyFace.xcodeproj before you continue.

What Are Face Anchors?

Hanging an anchor from one’s face sounds extremely painful. Luckily, the type of anchor we’re referring to here is an AR Face Anchor — undeniably one of the coolest types of anchors available. Thanks to Reality Composer, creating face anchors is super easy.

By using the TrueDepth front-facing camera, face anchors provide information about the user’s facial position, orientation, topology and facial expression.

Unfortunately, you can only use face tracking if you have a device equipped with a TrueDepth front-facing camera. If the device is Face-Id capable, you’re good to go.

When the camera detects a face, an anchor gets added slightly behind the nose in the middle of the head.

Here, the cute monkey head demonstrates how a face anchor is created after a face is detected.

It’s also important to know that the anchor uses a right-handed coordinate system measured in meters.

Here’s a breakdown of each axis:

  • X-Axis: The red arrow pointing right represents this axis.
  • Y-Axis: The green arrow pointing up represents this axis.
  • Z-Axis: The blue arrow pointing forward represents this axis.

Creating Face Anchors

It’s time to dive into the action and build some face anchor scenes with the provided props.

Creating Multiple Scenes

To add more props, you have to create multiple scenes within the Reality Composer project. Each scene will house a single facial prop. To switch between the different props, you simply need to switch the different scenes.

Code Generation

Reality Composer is tightly integrated into Xcode. When you build the project, Xcode will inspect all the associated Reality files within the project and generate Swift code.

Fixing the Project

You’ll look at the coding side of the project next. When you recompile your project, it generates an error.

arView = ARView(frame: .zero)
return arView

Switching to the Front-Facing Camera

When the app starts, you need to manually switch to the front-facing camera. To do that, you’ll need a little help from ARKit.

import ARKit

AR Session

Before moving on, you need to learn about the AR session, which you can access via ARView.session.

AR Configuration

Before starting an AR session, you have to create an AR session configuration. You use this configuration to establish the connection between the real world, where your device is, and the virtual 3D world, where your virtual content is.

// 1
let arConfiguration = ARFaceTrackingConfiguration()
// 2
uiView.session.run(arConfiguration, 
  options:[.resetTracking, .removeExistingAnchors])

Switching Between Multiple Scenes

Finally, you need to switch between the three scenes when the user presses the Previous or Next buttons.

switch(propId) {

  case 0: // Eyes
    let arAnchor = try! Experience.loadEyes()
    uiView.scene.anchors.append(arAnchor)
    break
            
  case 1: // Glasses
    let arAnchor = try! Experience.loadGlasses()
    uiView.scene.anchors.append(arAnchor)
    break
            
  case 2: // Mustache
    let arAnchor = try! Experience.loadMustache()
    uiView.scene.anchors.append(arAnchor)
    break
  
  default:
    break
}

Testing the App

Finally, you’re ready to do your very first build and run. Before you do, connect your physical device to your machine and select it in Xcode.

Manually Removing Anchors

Every time you switch from one scene to another, you load an anchor that appends to ARView.Scene.anchors. If you continue adding multiple anchors, you’ll end up with multiple props stacked on top of each other.

arView.scene.anchors.removeAll()

Key Points

Congratulations, you’ve reached the end of this chapter and your AR Funny Face app looks great. Take some selfies of yourself and some friends and try the funny props.

9. RealityKit
11. Facial Blend Shapes
Have a technical question? Want to report a bug? You can ask questions and report bugs to the book authors in our official book forum here.
© 2023 Kodeco Inc.

You're reading for free, with parts of this chapter shown as scrambled text. Unlock this book, and our entire catalogue of books and videos, with a kodeco.com Professional subscription.

Unlock now

Access this book

To highlight or take notes, you'll need to own this book in a subscription or purchased by itself.

Read for Free with a Subscription

Buy this book for $59.99