Chapters
Chapters

Chapters

Hide chapters

Apple Augmented Reality by Tutorials

Before You Begin

Section 0: 5 chapters
Show chapters Hide chapters

Section I: Reality Composer

Section 1: 5 chapters
Show chapters Hide chapters

Section II: Reality Composer & Image Tracking

Section 2: 2 chapters
Show chapters Hide chapters

Section III: Reality Composer & Object Tracking

Section 3: 1 chapter
Show chapters Hide chapters

Section IV: RealityKit & Face Tracking

Section 4: 3 chapters
Show chapters Hide chapters

Section V: ARKit & SpriteKit

Section 5: 3 chapters
Show chapters Hide chapters

Section VI: ARKit & SceneKit

Section 6: 2 chapters
Show chapters Hide chapters

Section VII: ECS & Collaborative Experiences (Bonus Section)

Section 7: 1 chapter
Show chapters Hide chapters

Section VIII: Creating 3D Models With Object Capture (Bonus Section))

Section 8: 2 chapters
Show chapters Hide chapters

17. ECS & Collaborative Experiences
Written by Chris Language

Heads up... You're reading this book for free, with parts of this chapter shown beyond this point as scrambled text.

You can unlock the rest of this book, and our entire catalogue of books and videos, with a kodeco.com Professional subscription.

Unlock now

In the early days of ARKit, it quickly became apparent that something important was missing: the ability to share augmented reality experiences among multiple users.

Later versions of ARKit addressed the issue by introducing ARWorldMap. The map contains a space-mapping state along with a set of anchors from a world-tracking AR session. This map can be shared, allowing multiple users to experience persistent AR anchors within the same space.

With the assistance of a peer-to-peer network, multiple users can share an ARWorldMap in real time, creating a collaborative experience. Using ARKit, the process is somewhat painful, requiring vast amounts of manual labor from a coding perspective.

Apple created a fantastic ARKit example project that you can explore. Find the project here: Creating a Multiuser AR Experience

However, since iOS 13, you’ve been able to pair RealityKit with ARKit to automate most of the manual effort that ARKit-based apps typically require.

In this chapter, you’ll create a modern take on the classic Tic-Tac-Toe game and deliver a RealityKit-based collaborative experience. The new project will borrow from Apple’s ARKit example project, but will mainly focus on the RealityKit side of things.

It’s time to get going!

Exploring the Project

There’s a starter project waiting for you in the starter/XOXO folder. The project is a basic Swift-based app that uses a classic style storyboard UI.

Load the project in Xcode so you can take a quick tour of the important components within.

ViewController.swift

Open ViewController.swift. By now, you’re very familiar with the inner workings of the ViewController.

MultipeerController.swift

Open MultipeerController.swift.

Main.storyboard

Open Main.storyboard and flip the orientation to landscape.

Info.plist

Open Info.plist.

Creating the AR View

Now that you’ve gotten the basics out of the way, you’ll start filling in the missing pieces, beginning with the AR View.

Setting the Player’s Color

Add the following variable to the Properties section:

var playerColor = UIColor.blue

playerColor = UIColor.blue

playerColor = UIColor.red

Sending Messages

Add the following helper function to Helper Functions:

func sendMessage(_ message: String) {
  DispatchQueue.main.async {
    self.message.text = message
  }
}

Creating the AR Configuration

You start all AR experiences by creating an ARConfiguration and running the AR session. Well, this AR experience is no different.

initARView()

func initARView() {
  arView.session.delegate = self
  arView.automaticallyConfigureSession = false
  let arConfiguration = ARWorldTrackingConfiguration()
  arConfiguration.planeDetection = [.horizontal]
  arConfiguration.environmentTexturing = .automatic
  arView.session.run(arConfiguration)
}

What is ECS?

When using the RealityKit framework to create content for your AR experiences, it’s important to note that the framework runs a CPU-based entity-component system (ECS) to manage physics, animations, audio processing and network synchronization. The framework then relies on Metal for GPU-based multithreaded rendering.

Predefined Entities

With RealityKit, you can easily create your own custom entities with custom behaviors based on the various components you add to them.

Creating the Game Board

Now that you have some background, it’s time to create some of your very own entities, starting with the game board.

var gridModelEntityX:ModelEntity?
var gridModelEntityY:ModelEntity?
var tileModelEntity:ModelEntity?

Creating the Model Entities

Look carefully at the game board and you can see that the entire board is constructed out of just three distinct shapes: the two grid bars and the square tiles. You’ll create those shapes next.

initModelEntities()

func initModelEntities() {
  // 1
  gridModelEntityX = ModelEntity(
    mesh: .generateBox(size: SIMD3(x: 0.3, y: 0.01, z: 0.01)),
    materials: [SimpleMaterial(color: .white, isMetallic: false)]
  )
  // 2  
  gridModelEntityY = ModelEntity(
    mesh: .generateBox(size: SIMD3(x: 0.01, y: 0.01, z: 0.3)),
    materials: [SimpleMaterial(color: .white, isMetallic: false)]
  )
  // 3  
  tileModelEntity = ModelEntity(
    mesh: .generateBox(size: SIMD3(x: 0.07, y: 0.01, z: 0.07)),
    materials: [SimpleMaterial(color: .gray, isMetallic: true)]
  )
  // 4  
  tileModelEntity!.generateCollisionShapes(recursive: false)
}

Cloning Model Entities

Now that you’ve created the three main shapes, you’ll use them to construct the game board. Instead of re-creating each element from scratch, you’ll clone the original entities.

func cloneModelEntity(_ modelEntity: ModelEntity,
  position: SIMD3<Float>) -> ModelEntity {
  let newModelEntity = modelEntity.clone(recursive: false)
  newModelEntity.position = position
  return newModelEntity
}

Adding the Grid

Now, you’re going to use the helper function above to create the grid. Add the following function to Model Entity Functions:

func addGameBoardAnchor(transform: simd_float4x4) {
  // 1
  let arAnchor = ARAnchor(name: "XOXO Grid", transform: transform)
  let anchorEntity = AnchorEntity(anchor: arAnchor)
  // 2
  anchorEntity.addChild(cloneModelEntity(gridModelEntityY!,
    position: SIMD3(x: 0.05, y: 0, z: 0)))
  anchorEntity.addChild(cloneModelEntity(gridModelEntityY!,
    position: SIMD3(x: -0.05, y: 0, z: 0)))
  anchorEntity.addChild(cloneModelEntity(gridModelEntityX!,
    position: SIMD3(x: 0.0, y: 0, z: 0.05)))
  anchorEntity.addChild(cloneModelEntity(gridModelEntityX!,
    position: SIMD3(x: 0.0, y: 0, z: -0.05)))
}

Adding the Tiles

With the grid out of the way, you need to add the tiles to the game board. There are nine slots to fill.

anchorEntity.addChild(cloneModelEntity(tileModelEntity!, 
  position: SIMD3(x: -0.1, y: 0, z: -0.1)))
anchorEntity.addChild(cloneModelEntity(tileModelEntity!, 
  position: SIMD3(x: 0, y: 0, z: -0.1)))
anchorEntity.addChild(cloneModelEntity(tileModelEntity!, 
  position: SIMD3(x: 0.1, y: 0, z: -0.1)))
anchorEntity.addChild(cloneModelEntity(tileModelEntity!, 
  position: SIMD3(x: -0.1, y: 0, z: 0)))
anchorEntity.addChild(cloneModelEntity(tileModelEntity!, 
  position: SIMD3(x: 0, y: 0, z: 0)))
anchorEntity.addChild(cloneModelEntity(tileModelEntity!, 
  position: SIMD3(x: 0.1, y: 0, z: 0)))
anchorEntity.addChild(cloneModelEntity(tileModelEntity!,
  position: SIMD3(x: -0.1, y: 0, z: 0.1)))
anchorEntity.addChild(cloneModelEntity(tileModelEntity!, 
  position: SIMD3(x: 0, y: 0, z: 0.1)))
anchorEntity.addChild(cloneModelEntity(tileModelEntity!, 
  position: SIMD3(x: 0.1, y: 0, z: 0.1)))

Adding the Anchor

Now that you’ve now completed the grid and all the tiles, your next step is to add the game board to the AR scene.

// 1
anchorEntity.anchoring = AnchoringComponent(arAnchor)
// 2
arView.scene.addAnchor(anchorEntity)
// 3
arView.session.add(anchor: arAnchor)

Placing Content

Now that your game board is ready to place in the scene, you need some user input to know where to place it. All the user needs to do is tap the horizontal surface and the game board should appear in that position. Your next step is to ensure the app recognizes the user’s tap.

Creating a Tap Gesture

You’ll start by creating a basic tap gesture to handle user touch input.

initGestures()

func initGestures() {
  // 1
  let tap = UITapGestureRecognizer(
    target: self, 
    action: #selector(handleTap))
  // 2
  self.arView.addGestureRecognizer(tap)
}

Handling Tap Gestures

But hang, on there’s an error. You still need to define handleTap().

@objc func handleTap(recognizer: UITapGestureRecognizer?) {
}

Getting the Touch Location

After the user taps the screen, you’ll cast a ray into the scene to see where on the surface the tap actually occurred. This lets you position the game board just where they want it.

guard let touchLocation = 
  recognizer?.location(in: self.arView) else { return }

Tapping a Surface

Now, to perform the actual ray-cast into the AR scene.

let results = self.arView.raycast(
  from: touchLocation,
  allowing: .estimatedPlane,
  alignment: .horizontal)
    
if let firstResult = results.first {
  self.addGameBoardAnchor(transform: firstResult.worldTransform)
} else {
  self.message.text = "[WARNING] No surface detected!"
}

Tapping a Tile

OK, now that the game board is visible in the scene, what’s next? Well, when the user touches a tile, that tile should change to the player’s color.

if let hitEntity = self.arView.entity(at: touchLocation) {
  let modelEntity = hitEntity as! ModelEntity
    modelEntity.model?.materials = [
      SimpleMaterial(color: self.playerColor,
      isMetallic: true)]
  return
}

Collaborative Experiences

When multiple people share an augmented reality experience from their own personal viewpoints on separate devices, it’s known as a collaborative experience. To achieve such an experience, all the devices should be connected to one another via a local network or Bluetooth. The devices share an AR world map, which localizes each device within the same space. During an active collaborative session, entities in the augmented space synchronize across all the other devices.

Creating a Multi-Peer Network with MCSession

Thankfully, all the hard work is already done, thanks to MultipeerSession, which is part of your project. It acts as a basic wrapper class for MCSession, which is the network session class that connects multiple peers.

Adding Multi-Peer Connectivity

Now that the network is ready, you’ll create the multi-peer session.

var multipeerSession: MultipeerSession?
var peerSessionIDs = [MCPeerID: String]()
var sessionIDObservation: NSKeyValueObservation?

initMultipeerSession()

func initMultipeerSession()
{
    multipeerSession = MultipeerSession(
      receivedDataHandler: receivedData,
      peerJoinedHandler: peerJoined,
      peerLeftHandler: peerLeft,
      peerDiscoveredHandler: peerDiscovered)
}

func receivedData(_ data: Data, from peer: MCPeerID) {
}
  
func peerDiscovered(_ peer: MCPeerID) -> Bool {
}
  
func peerJoined(_ peer: MCPeerID) {
}
  
func peerLeft(_ peer: MCPeerID) {
}

Handling Session ID Changes

When a peer connects or when your session ID changes, you need to inform the connected peers of your current peer ID.

private func sendARSessionIDTo(peers: [MCPeerID]) {
  guard let multipeerSession = multipeerSession else { return }
  let idString = arView.session.identifier.uuidString
  let command = "SessionID:" + idString
  if let commandData = command.data(using: .utf8) {
    multipeerSession.sendToPeers(commandData,
      reliably: true,
      peers: peers)
  }
}

sessionIDObservation = observe(\.arView.session.identifier,
  options: [.new]) { object, change in
    print("Current SessionID: \(change.newValue!)")
    guard let multipeerSession = self.multipeerSession else 
    { return }
    self.sendARSessionIDTo(peers: multipeerSession.connectedPeers)
}

Handling the “Peer” Discovered Event

When the network session discovers a new peer, it triggers peerDiscovered(_:), asking it for permission to allow the new peer to connect.

guard let multipeerSession = multipeerSession else 
{ return false }

sendMessage("Peer discovered!")

if multipeerSession.connectedPeers.count > 2 {
  sendMessage("[WARNING] Max connections reached!")
  return false
} else {
  return true
}

Handling the “Peer Joined” Event

When the peer is allowed to connect, the network session will trigger peerJoined(_:).

sendMessage("Hold phones together...")
sendARSessionIDTo(peers: [peer])

Handling the “Peer Left” Event

When a peer leaves, you need to update peerSessionIDs. To do this, add the following to peerLeft(_:):

sendMessage("Peer left!")
peerSessionIDs.removeValue(forKey: peer)

Configuring RealityKit for Collaboration

Well, that’s all you need to do to create a multi-peer network, but you’re not quite done yet. You still need to configure RealityKit for collaboration.

Enabling Collaboration

To use collaboration, you need to enable it when you create the AR configuration. Do this by adding the following line of code to initARView(), just before running the AR session:

arConfiguration.isCollaborationEnabled = true

Setting the Synchronization Service

When you use RealityKit, you have to synchronize all of its entities and their components with all the connected peers.

extension MultipeerSession {
  public var multipeerConnectivityService:
    MultipeerConnectivityService? {
      return try? MultipeerConnectivityService(
        session: self.session)
  }
}

// 1
guard let multipeerConnectivityService =
  multipeerSession!.multipeerConnectivityService else {
    fatalError("[FATAL ERROR] Unable to create Sync Service!")
  }
// 2    
arView.scene.synchronizationService = multipeerConnectivityService
self.message.text = "Waiting for peers..."

Handling a Successful Connection

Now that everything’s in place, once a new peer successfully joins, RealityKit will create an ARParticipationAnchor for that peer.

func session(_ session: ARSession, didAdd anchors: [ARAnchor]) {
  for anchor in anchors {
    if let participantAnchor = anchor as? ARParticipantAnchor {
      self.message.text = "Peer connected!"
      let anchorEntity = AnchorEntity(anchor: participantAnchor)
      arView.scene.addAnchor(anchorEntity)
    }
  }
}

Requesting Network Permissions

Oh, you’re not quite done yet. There’s one last thing that you have to do and that’s to request network permissions.

Managing Ownership

During a collaborative experience, when you create an entity, you become the owner of that entity. Should another peer attempt to modify an entity that belongs to you, they’ll be blocked.

Enabling Automatic Ownership

To keep things simple, when another peer requests ownership of an entity that belongs to you, you’ll simply transfer ownership to that peer automatically.

anchorEntity.synchronization?.ownershipTransferMode = .autoAccept

Requesting Ownership

Now, you need to make sure you request ownership when you tap on a tile.

if let hitEntity = self.arView.entity(at: touchLocation) {
  if hitEntity.isOwner {
    let modelEntity = hitEntity as! ModelEntity
    modelEntity.model?.materials = [
      SimpleMaterial(color: self.playerColor,
        isMetallic: true)]
  } else {
    hitEntity.requestOwnership { result in
      if result == .granted {
        let modelEntity = hitEntity as! ModelEntity
        modelEntity.model?.materials = [
          SimpleMaterial(color: self.playerColor, 
            isMetallic: true)]
      }
    }
  }  
  return
}

Removing Anchors

As a final touch, when you’ve played a few games with a friend, it would be nice to clear out the playing field so that you can play some more.

func removeAnchors() {
  guard let frame = arView.session.currentFrame else { return }
  for anchor in frame.anchors {
    arView.session.remove(anchor: anchor)
  }
  sendMessage("All anchors removed!")
}

removeAnchors()

Key Points

Congratulations, you’ve reached the end of this chapter and section. You can find a copy of the project in its final state under final/XOXO.

16. Focus Nodes & Billboards
18. Creating 3D Models With Object Capture
Have a technical question? Want to report a bug? You can ask questions and report bugs to the book authors in our official book forum here.
© 2023 Kodeco Inc.

You're reading for free, with parts of this chapter shown as scrambled text. Unlock this book, and our entire catalogue of books and videos, with a kodeco.com Professional subscription.

Unlock now

Access this book

To highlight or take notes, you'll need to own this book in a subscription or purchased by itself.

Read for Free with a Subscription

Buy this book for $59.99