Chapters
Chapters

Chapters

Hide chapters

Apple Augmented Reality by Tutorials

Before You Begin

Section 0: 5 chapters
Show chapters Hide chapters

Section I: Reality Composer

Section 1: 5 chapters
Show chapters Hide chapters

Section II: Reality Composer & Image Tracking

Section 2: 2 chapters
Show chapters Hide chapters

Section III: Reality Composer & Object Tracking

Section 3: 1 chapter
Show chapters Hide chapters

Section IV: RealityKit & Face Tracking

Section 4: 3 chapters
Show chapters Hide chapters

Section V: ARKit & SpriteKit

Section 5: 3 chapters
Show chapters Hide chapters

Section VI: ARKit & SceneKit

Section 6: 2 chapters
Show chapters Hide chapters

Section VII: ECS & Collaborative Experiences (Bonus Section)

Section 7: 1 chapter
Show chapters Hide chapters

Section VIII: Creating 3D Models With Object Capture (Bonus Section))

Section 8: 2 chapters
Show chapters Hide chapters

9. RealityKit
Written by Chris Language

Heads up... You're reading this book for free, with parts of this chapter shown beyond this point as scrambled text.

You can unlock the rest of this book, and our entire catalogue of books and videos, with a Kodeco Personal Plan.

Unlock now

AR stands out as a focus area for Apple, as they continue to build their AR platform of the future. Thanks to AR Quick Look, AR has become extremely accessible and is now deeply integrated into iOS, macOS and tvOS.

Creating immersive AR experiences has historically been difficult, requiring a vast amount of skill and knowledge. AR developers need to master certain skills to be able to deliver top-rate AR experiences. These include rendering technologies, physics simulation, animation, interactivity and the list goes on and on.

Thankfully, that all changed with the introduction of RealityKit.

With RealityKit in your toolbox, creating AR experiences has never been easier.

In this section, you’ll learn all about RealityKit and face tracking. You’ll create a SnapChat-like face filter app with SwiftUI called AR Funny Face, where you get to mock up your face with funny props. You’ll also create an animated mask that you can control with your eyes, brows and mouth.

What is RealityKit?

RealityKit is a new Swift framework that Apple introduced at WWDC 2019. Apple designed it from the ground up with AR development in mind. Its main purpose is to help you build AR apps and experiences more easily. Thanks to the awesome power of Swift, RealityKit delivers a high-quality framework with a super simple API.

RealityKit is a high-quality rendering technology capable of delivering hyper-realistic, physically-based graphics with precise physics simulation and collisions against the real-world environment. It does all of the heavy lifting for you, right out of the box. It makes your content look as good as possible while fitting seamlessly into the real world. It’s impressive feature list includes skeletal animations, realistic shadows, lights, reflections and post-processing effects.

Are you ready to give it a try? Open RealityKit and take a look at what’s inside.

At its core, you’ll find many of Apple’s other frameworks, but the ones doing most of the work are ARKit and Metal.

Here’s a breakdown of RealityKit’s coolest features:

  • Rendering: RealityKit offers a powerful new physically-based renderer built on top of Metal, which is fully optimized for all Apple devices.

  • Animation: It has built-in support for Skeletal animation and Transform-based animation. So, if you want, you can animate a zombie or you can move, scale and rotate objects with various easing functions.

  • Physics: With a powerful physics engine, RealityKit lets you throw anything at it — pun intended! You can adjust real-world physics properties like mass, drag and restitution, allowing you to fine-tune collisions.

  • Audio: Spacial audio understanding and automatic listener configuration let you attach sound effects to 3D objects. You can then track those sounds, making them sound realistic based on their position in the real world.

  • ECS: From a coding perspective, RealityKit enforces the Entity Component System design pattern to build objects within the world.

  • Synchronization: The framework has built-in support for networking, designed for collaborative experiences. It even offers automatic synchronization of entities between multiple clients.

Enough talk, it’s time to dive into some code!

Creating a RealityKit Project

Now that you have some understand about RealityKit’s features, you’ll create your first RealityKit project. Launch Xcode and get ready to create a new Augmented Reality App project from scratch.

Reviewing the Project

At first glance within the project, you’ll notice the usual suspects — but there are a few new things, too:

RealityKit API Components

Now, take a look at a few main components that form parts of the RealityKit API.

Building the UI with SwiftUI

When you created the app, you selected SwiftUI for the user interface. Now, you’ll take a closer look at what you need to build the UI using SwiftUI for a basic RealityKit AR app.

Tracking the Active Prop

Your AR experience is going to contain multiple scenes with various props to make your pictures funnier. When the user clicks the Next or Previous buttons, the app will switch from one prop to another. You’ll implement that functionality now.

@State var propId: Int = 0

@Binding var propId: Int

return ARViewContainer().edgesIgnoringSafeArea(.all)

// 1
ZStack(alignment: .bottom) {
  // 2
  ARViewContainer(propId: $propId).edgesIgnoringSafeArea(.all)
  // 3
  HStack {
  }
}

Adding Buttons

The buttons all use images. So next, you’ll add the required images to the project by dragging and dropping all the image files from starter/resources/images into Assets.xcassets.

Button(action: {
  self.propId = self.propId <= 0 ? 0 : self.propId - 1
}) {
  Image("PreviousButton").clipShape(Circle())
}

Button(action: {
  //self.TakeSnapshot()
}) {
  Image("ShutterButton").clipShape(Circle())
}

Button(action: {
  self.propId = self.propId >= 2 ? 2 : self.propId + 1
}) {
  Image("NextButton").clipShape(Circle())
}


Spacer()

Taking Selfies

What good is an AR face mockup app if you can’t use it to take selfies?

var arView: ARView!

let arView = ARView(frame: .zero)

arView = ARView(frame: .zero)

func TakeSnapshot() {
  // 1
  arView.snapshot(saveToHDR: false) { (image) in
    // 2
    let compressedImage = UIImage(
      data: (image?.pngData())!)
    // 3
    UIImageWriteToSavedPhotosAlbum(
      compressedImage!, nil, nil, nil)
  }
}

self.TakeSnapshot()

Requesting Access to the Camera and Photos

You’re not done just yet. You still need to make sure the app asks for access to the camera and the photo’s album.

Key Points

You’ve reached the end of this chapter. To recap some of the key takeaways:

Where To Go From Here?

There’s much more content about RealityKit waiting for you from WWDC 2019. I highly recommend you check out the following:

8. Object Anchors
10. Face Anchors
Have a technical question? Want to report a bug? You can ask questions and report bugs to the book authors in our official book forum here.
© 2023 Kodeco Inc.

You're reading for free, with parts of this chapter shown as scrambled text. Unlock this book, and our entire catalogue of books and videos, with a Kodeco Personal Plan.

Unlock now

Access this book

To highlight or take notes, you'll need to own this book in a subscription or purchased by itself.

Read for Free with a Subscription

Buy this book for $59.99