12.
ARKit
Written by Chris Language
Heads up... You're reading this book for free, with parts of this chapter shown beyond this point as text.
Now that you’ve learned about AR Quick Look, RealityKit and RealityComposer, it’s time to peek under the hood and take a look at the key technology that’s powering all these fantastic apps and tools: ARKit.
In this section, you’ll combine the power of ARKit with the power of SpriteKit. You’ll create a fun 2D SpriteKit game where you get to pop little emojis before they explode in your face!
In this chapter, you’ll focus on ARKit first.
Note: This chapter is theory based only. Feel free to jump ahead to the next chapter if you want to get your hands dirty with an ARKit based project.
What is ARKit?
ARKit is Apple’s mobile AR development framework. With it, you can create an immersive, engaging experience that allows you to augment virtual 2D and 3D content with the live camera feed of the world around you.
Augmented reality frameworks aren’t new. Vuforia, for example, has been around for many years. What sets ARKit apart from other AR frameworks is that ARKit performs markerless tracking.
Being markerless means that ARKit doesn’t need tracking cards. It understands the world around it and quickly identifies surfaces that it can place virtual content on.
Interestingly enough, older frameworks quickly incorporated ARKit into their APIs, instantly giving them ARKit’s awesome markerless tracking capabilities.
ARKit transforms any Apple device with an A9 or higher processor into an AR-capable device. Just think about that for a second: At this very moment, millions of Apple users already have a sophisticated AR-capable device right in their pockets.
Note: As Apple continues to develop new AR-capable hardware for its most recent devices, they improve ARKit, too, to provide support for the newer hardware. It’s important to note that some of the more modern ARKit features will only work on devices with specific hardware.
Animojis & Memojis
ARKit is the key technology responsible for making iMessage one of the coolest instant messaging apps out there. ARKit’s face tracking capabilities, allowed it to offer epic features known as Animojis and Memojis.
Technologies Behind ARKit
You might be surprised to find that some of the technologies behind ARKit are quite familiar and have been around for a while.
Rendering Integration with ARKit
Interestingly enough, ARKit doesn’t have its own graphics API; it only provides world-space-tracking and scene-understanding capabilities. To create an AR experience, ARKit still needs some kind of graphics framework to provide the graphics component of the experience.
ARKit-powered App Examples
To demonstrate ARKit’s capabilities, here are some examples of interesting ARKit-powered apps you can find for free on the Apple AppStore. Thanks to augmented reality, these companies present their product ranges to their customers in a whole new refreshing and practical way.
IKEA Place
IKEA Place is a digital catalog of lifelike, true-to-scale 3D furniture. Thanks to the power of ARKit, customers can preview IKEA’s furniture in their own homes, giving them a real sense of the furniture’s scale and ascetics.
Warby Parker
Warby Parker is a digital eyewear catalog allowing customers to browse through a vast collection of eyeglasses, sunglasses and contact lenses. With ARKit’s face detection and tracking capabilities, customers can virtually try out and preview frames.
Minecraft Earth
Get to experience Minecraft in a whole new way with Minecraft Earth. With ARKit’s help, players can now enjoy this classic game in augmented reality. They can build amazing creations in tabletop mode or in life-size mode, and even collaborate and build together.
ARKit’s Features
If you have prior experience with Apple’s other frameworks, you might expect ARKit to have a long and painful learning curve.
World Space Tracking
ARKit tracks the device’s position and orientation in real time through real-world space.
Scene Understanding & Persistence
To achieve markerless tracking, ARKit creates and manages its own map of the surfaces and feature points it detects, then stores that map in a point cloud. Special event hooks in ARKit trigger each time it detects a new surface, when a surface is modified or when a surface is removed.
Light Estimation
ARKit can provide accurate real-world lighting conditions. It analyzes the active video feed frame data to calculate the estimated environmental lighting conditions, and then provides a light source to light your virtual scene.
Real-time Environment Map
To push the visual fidelity of your virtual content even further, ARKit generates a real-time environment map by sampling your immediate space. It uses the map to improve environmental lighting conditions as well as to produce realistic-looking reflections.
Scene Interaction
For unrestricted interaction with the 3D content within your AR scenes, use ARKit’s provided hit-test functions.
Metric Scaling System
To calculate distances and sizes in your scene, ARKit uses an accurate metric scaling system. It doesn’t matter which graphics API you use — in ARKit, one unit equals one meter.
2D Image Detection & Tracking
ARKit can detect and track up to 100 images and rectangular shapes at a time. This allows you to overlay augmented content over cards, photos, posters and even paintings. ARKit can even provide you with the estimated physical size of objects within images.
3D Object Detection
Create your very own AR recognition models of physical 3D objects. Once you give it an AR model, ARKit can detect that physical 3D object, even in fairly complex environments. How about blowing some smoke out of a toy car exhaust?
Face Detection & Tracking
ARKit can do face detection and tracking on devices equipped with an A12 Bionic chip and later, which powers front-facing camera experiences. Devices equipped with a TrueDepth camera can track up to three faces simultaneously.
Multi-Camera Tracking
You can simultaneously use face and world tracking on the front and back cameras, opening up new possibilities.
Motion Capture
Real-time motion capture allows you to track a person’s body position, movement and pose and produces a 3D skeleton consisting of joints and bones.
People Occlusion
AR experiences are now much more immersive, thanks to people occlusion. This is a green-screen-style effect made possible by machine learning.
Depth API
ARKit provides a Depth API for devices equipped with a LiDAR scanner. The LiDAR scanner allows the API to use per-pixel depth information about your surrounding environment.
Scene Geometry
With devices equipped with a LiDAR scanner, ARKit is able to create a topological map of your space. Thanks to machine learning, ARKit has a deep understanding of your environment, too. It can place labels next to identified content like floors, walls, ceilings, windows, doors and even seats. It can then use all this information to provide object occlusion and real-world physics for your virtual content.
Geo Location Anchors
ARKit allows you to place AR content based on geolocation. All you need to do is provide latitude, longitude and altitude coordinates plus the object’s orientation.
Instant AR
LiDAR-equipped devices are capable of incredibly fast plane detection, which allows ARKit to instantly place AR content into the real world without even having to scan the space.
Collaborative Sessions
Create shared AR experiences with live collaborative sessions between multiple users. ARKit builds a collaborative world map between multiple users, allowing them to get into a shared AR experience much faster — and, ultimately, making the development of shared AR experiences easier and faster, too.
ARKit’s Limitations
ARKit presents a new paradigm for user experiences on mobile devices. It does, however, have some limiting qualities that impact the user’s AR experience. As a developer, you should be aware of these limitations.
Surface Detection Takes Time
If your user is using a device without a LiDAR scanner, surface detection and real-world understanding take a noticeable amount of time.
Motion Processing Lag
With AR experiences, you’ll quickly notice that excessive device motion can cause stuttering and blurred imaging, which makes it difficult for ARKit to understand what it’s looking at. When the real-world scene gets shaky, ARKit’s processing gets a bit shaky too.
Low-Light Conditions
Low-light conditions are problematic. The on-device camera cannot discern low-light scenes as well as the human eye can, which means ARKit has trouble with scene detection when the lights are dim. It may even fail to recognize any surfaces at all.
Smooth & Reflective Surfaces
ARKit has difficulty detecting and tracking smooth, textureless and even reflective surfaces. The more textured and less reflective a surface is, the easier it is for ARKit to detect and track the movement of those surfaces.
Ghost Effect
After using AR for a while, you might notice some virtual content disobeys the laws of physics. For example, the virtual content might pass through real-world objects, much like a ghost would pass through a closed door.
Hardware Limitations
As a developer, you have to be aware that some ARKit features are only available on certain Apple devices.
ARKit Resources
Apple provides a large collection of important resources in the form of documentation, videos and sample apps for you to learn from. Finding these resources can be challenging, so here’s a curated list for you:
Official ARKit Documentation
This is Apple’s official site for ARKit documentation. Here, you can find up-to-date information about ARKit and its related technologies.
Human Interface Guidelines for Augmented Reality
Apple also provides a set of Human Interface Guidelines to coach you on the best practices when designing your ARKit apps. This guide shows you exactly what to do — and what not to do. Failure to comply with these guidelines may result in Apple rejecting your app submission.
Official WWDC Videos on ARKit
Apple always packs a good deal of information into its WWDC presentations. Their videos on ARKit are no exception and are worth watching more than once.
Interactive Content with ARKit
This is one of Apple’s official ARKit demos, and it’s a fun way to get some hands-on experience with ARKit.
Key Points
You’ve reached the end of the chapter, but your journey of creating AR-enabled games and apps with ARKit has just begun.