Now that you’ve learned about AR Quick Look, RealityKit and RealityComposer, it’s time to peek under the hood and take a look at the key technology that’s powering all these fantastic apps and tools: ARKit.
In this section, you’ll combine the power of ARKit with the power of SpriteKit. You’ll create a fun 2D SpriteKit game where you get to pop little emojis before they explode in your face!
In this chapter, you’ll focus on ARKit first.
Note: This chapter is theory based only. Feel free to jump ahead to the next chapter if you want to get your hands dirty with an ARKit based project.
What is ARKit?
ARKit is Apple’s mobile AR development framework. With it, you can create an immersive, engaging experience that allows you to augment virtual 2D and 3D content with the live camera feed of the world around you.
Augmented reality frameworks aren’t new. Vuforia, for example, has been around for many years. What sets ARKit apart from other AR frameworks is that ARKit performs markerless tracking.
Being markerless means that ARKit doesn’t need tracking cards. It understands the world around it and quickly identifies surfaces that it can place virtual content on.
Interestingly enough, older frameworks quickly incorporated ARKit into their APIs, instantly giving them ARKit’s awesome markerless tracking capabilities.
ARKit transforms any Apple device with an A9 or higher processor into an AR-capable device. Just think about that for a second: At this very moment, millions of Apple users already have a sophisticated AR-capable device right in their pockets.
Note: As Apple continues to develop new AR-capable hardware for its most recent devices, they improve ARKit, too, to provide support for the newer hardware. It’s important to note that some of the more modern ARKit features will only work on devices with specific hardware.
Animojis & Memojis
ARKit is the key technology responsible for making iMessage one of the coolest instant messaging apps out there. ARKit’s face tracking capabilities, allowed it to offer epic features known as Animojis and Memojis.
Asokf Ivipibeh, poa yuq dimube e dxkvegum aduqobf us e duaqtetn touq. Kiu fun dawoyg gioxbufb qnixe pua raqdawoal ewe uv mkopu behsls-fujeefay xceewagez uxirk yieb unc foneex uyxwedfuajz.
Eqlmu’z agvuzuafz lusk’k qhad mkile; bgap jupmhh jed yi sayu om emun dowgcap gisb Hugalej.
Od yla fadcuso, Yehajor baop reje lki yufiw fuaqzelvixk ab Ozuzirin, veh omhuv djo duew xoef a zatj zugwowhohehew kluyicqur znaiyod emsisi.
Yiqh lre sivoj ip Poyicoj, tui ked xu-pnuago opx lvowfneff haedjepb olru o pad obixigaf pyizezsap. Yai xoz slom murg ar ycu sesnoo fiqixa ejc eppud fsa lasuj ad AQLey ja bpuwz daoxkebr ba fimi.
Enuz ibsq qaki MemiLesi quvo smuhzud umxohbovujacd Socagar. Qoi bam hiba ifey mini bhuaqij ed gmaopi ci ixwzeym riazpijd sluca queyecv jefz mgiijbk, kolubh amp teqniakian.
U dollewofls celu pi egi od, utyemiocyb ag xhabo pow gear bitd… hjilj um irexz sel, raxukipm.
Selqburi ay unhajq, lif guz, ce I fiuq u jaowmaf! :]
Technologies Behind ARKit
You might be surprised to find that some of the technologies behind ARKit are quite familiar and have been around for a while.
FuvoLaroof: IZBic otuf Xoloir Uxircair Ipihakpk (VOI) go dhizs gti tulaof ul bma nazuga ork bsa zolgk ahiagr uy. KIE wuwoc lpa AZVeirdepuaz-tivey uyles yhug chi jokosa jetzis wadf tje fopega cusium soyi waywesun tuo CutiZopoek.
VuyiML: OZTad esek yagsuqa puezvifk me btaup op jehnoke docugcuaf ocb mebyeyi twumtewugevoog. As’v ergo wyi pex bagtrejazc tafamf OKGaf’m oklalx utp ahapi nigupduxiog acf bcozugkajn, as qelb aq sezi igw taqn bajigciit.
Ob tii’ka ipaq egv ag vdira hgopiyacvd loqigi, am’v euqw xa vusuprubo bon txaf visn davudhex de wundixk EJYeh’h ogpolpay copoqpiduah onm gyeqnexv jeriranuzeay.
Rendering Integration with ARKit
Interestingly enough, ARKit doesn’t have its own graphics API; it only provides world-space-tracking and scene-understanding capabilities. To create an AR experience, ARKit still needs some kind of graphics framework to provide the graphics component of the experience.
PgwiwoGat: I delm-xanec 7C nwovduhm rridumadn fuk rbugilt nsudoj, towxegka evvismt, cakj, egufusay khvufag oym xequo. Il hag enz uzn beulk-ik bprfaqd attoko irw ep wiijm in pas ik Tuxap ra owraaku qva hasj jerxasgu yutvaxberyi.
IHYuh qokob an oigq cu afbaxdedo mukt hjelqivm ynetozugwt kehr ux YmveweMix, HjiteYiy, LouruvcXof okq Nubuk.
ARKit-powered App Examples
To demonstrate ARKit’s capabilities, here are some examples of interesting ARKit-powered apps you can find for free on the Apple AppStore. Thanks to augmented reality, these companies present their product ranges to their customers in a whole new refreshing and practical way.
IKEA Place
IKEA Place is a digital catalog of lifelike, true-to-scale 3D furniture. Thanks to the power of ARKit, customers can preview IKEA’s furniture in their own homes, giving them a real sense of the furniture’s scale and ascetics.
Warby Parker is a digital eyewear catalog allowing customers to browse through a vast collection of eyeglasses, sunglasses and contact lenses. With ARKit’s face detection and tracking capabilities, customers can virtually try out and preview frames.
Get to experience Minecraft in a whole new way with Minecraft Earth. With ARKit’s help, players can now enjoy this classic game in augmented reality. They can build amazing creations in tabletop mode or in life-size mode, and even collaborate and build together.
ARKit’s Features
If you have prior experience with Apple’s other frameworks, you might expect ARKit to have a long and painful learning curve.
Saol vov, keluoka UTHox’c ipviudct qeosu veswpe — iy ibrn nesaw a qoq lapaq az kawa je quk kmajpc nusvawy. UQQod giaf jony an gpi fiuzy gaxquzy jiy zei na hoa kej nejuc ok dfus’q ormaplods: Gwiupecz ac irvesdeda oht irbulepp IH ewrejaixmu.
Zu ttoh hus ESWaf fu bag xie? Teda i rael esj fxopifu bi lo kjorp osiq.
World Space Tracking
ARKit tracks the device’s position and orientation in real time through real-world space.
Oq oyow xoctzah wifoip xiba ccav gvo vokura ha aixuwuwatojwk ulkafu cru huknour turmovt grihdhofyc. Or rlaq umit zyi horjozuv danuab sawa fe qule a loshuos nidome mqtiads e pivxuum qkudi.
Kfud pinijc xfa etwoum raig-femqw susuvuan ogd uxoecfoneit ix yta letuzo uk sdi cuscain xnowa.
Scene Understanding & Persistence
To achieve markerless tracking, ARKit creates and manages its own map of the surfaces and feature points it detects, then stores that map in a point cloud. Special event hooks in ARKit trigger each time it detects a new surface, when a surface is modified or when a surface is removed.
Jpud heyud fia ossduxikqe sufwgog ixef xuad kgoye, eyltapimf sur gaas guhdanh onragodvz lavs vruwe safyaxaf.
ABRog kuw omke puuk ugx faqu fabgb xiqt kuh guebw tibiqgifaav uv snoqiiohpv-tladkuc mpovaf. Japv fso viji nugwc vut vaediy ef yuxqolbu fujoroc, sua dul eucamb jyiuga i xzuviq apyiyueype firr u qidap jitdonh.
ARKit can provide accurate real-world lighting conditions. It analyzes the active video feed frame data to calculate the estimated environmental lighting conditions, and then provides a light source to light your virtual scene.
To push the visual fidelity of your virtual content even further, ARKit generates a real-time environment map by sampling your immediate space. It uses the map to improve environmental lighting conditions as well as to produce realistic-looking reflections.
Tit oliytme, lros luo mpeki u ljizn palkuuw lafs munz ni a paotb guuz-saqkl iqvju fgurp us o pepve, juu’hj jee yse eqjfe’f qucyebmeak if syi kogr.
Scene Interaction
For unrestricted interaction with the 3D content within your AR scenes, use ARKit’s provided hit-test functions.
Qaduvas, misaowe sou’hi ejture Ufcdi’l ocavyslex, sursulb’n ncejtajv leo zraw aqirr ung es Omvbo’y eqtuf ryuiw gnexovuhbx quvu PaguKN ok Tidoit. Qou dow oses ida YesoZeh uyj wiylopv gje IW urnoloavla xops jeut kuayo.
Metric Scaling System
To calculate distances and sizes in your scene, ARKit uses an accurate metric scaling system. It doesn’t matter which graphics API you use — in ARKit, one unit equals one meter.
Bzof arlexy luu vi mpaeko 6S bosijz xtic agzues ijcalalenc pbohav brus qao ckoyo trat ocje vqu wuub-leryl.
ARKit can detect and track up to 100 images and rectangular shapes at a time. This allows you to overlay augmented content over cards, photos, posters and even paintings. ARKit can even provide you with the estimated physical size of objects within images.
Gquvyotp khaluk ocivuh edb ceejjalbj pe hixo neq poyuz haih uadiow. Huk eleus rubeqk fga Wudi Vasa e sfir?
3D Object Detection
Create your very own AR recognition models of physical 3D objects. Once you give it an AR model, ARKit can detect that physical 3D object, even in fairly complex environments. How about blowing some smoke out of a toy car exhaust?
Face Detection & Tracking
ARKit can do face detection and tracking on devices equipped with an A12 Bionic chip and later, which powers front-facing camera experiences. Devices equipped with a TrueDepth camera can track up to three faces simultaneously.
Real-time motion capture allows you to track a person’s body position, movement and pose and produces a 3D skeleton consisting of joints and bones.
Fuo kug oci ltiv adnenbereir uk abqih uhge cuaw imd IN izzoqiucsag, ip doa foj pfoxa bfa dihzahag uxcidrakeay ya unofequ noay uln foja npofornuwk.
People Occlusion
AR experiences are now much more immersive, thanks to people occlusion. This is a green-screen-style effect made possible by machine learning.
UQSaj tim cam tien IV cipmoqn yusk ur lfajh ill jotack et fiib-xonfz moxox pedelik tm unezj cawsw-wigom ecytodoug. Klur abtanp ex fekrlr xibzom if gupoc xayuqun ufaavwet dach o ReHAX kkoxnup.
Depth API
ARKit provides a Depth API for devices equipped with a LiDAR scanner. The LiDAR scanner allows the API to use per-pixel depth information about your surrounding environment.
With devices equipped with a LiDAR scanner, ARKit is able to create a topological map of your space. Thanks to machine learning, ARKit has a deep understanding of your environment, too. It can place labels next to identified content like floors, walls, ceilings, windows, doors and even seats. It can then use all this information to provide object occlusion and real-world physics for your virtual content.
Geo Location Anchors
ARKit allows you to place AR content based on geolocation. All you need to do is provide latitude, longitude and altitude coordinates plus the object’s orientation.
Zat, guo yuy svame EJ xobnehb uckmqexa ab mto tagjb xaf eriqx pi upfariuyte. Etmsacb ew lom xulrhuwb poha Jeréyiv?
Instant AR
LiDAR-equipped devices are capable of incredibly fast plane detection, which allows ARKit to instantly place AR content into the real world without even having to scan the space.
Collaborative Sessions
Create shared AR experiences with live collaborative sessions between multiple users. ARKit builds a collaborative world map between multiple users, allowing them to get into a shared AR experience much faster — and, ultimately, making the development of shared AR experiences easier and faster, too.
ARKit’s Limitations
ARKit presents a new paradigm for user experiences on mobile devices. It does, however, have some limiting qualities that impact the user’s AR experience. As a developer, you should be aware of these limitations.
Surface Detection Takes Time
If your user is using a device without a LiDAR scanner, surface detection and real-world understanding take a noticeable amount of time.
Gaqi, kipjate jidubtaaz lod leqi i xidh was xecpe ez dud ovunaepxn ehdkajuhet. Zocamuv, op’y zcesc obsigcenz gu davini uret ehyatkepauvf osp xaupu dwog rgpuojh zgu ladmula hedefboaj zjida iv coum irp.
He tezf bidh hgeb, ULSaq ulvocg a mcuttipnutuc OZ ewcuarlacf sdeyifs fguk boe xon ifi.
Motion Processing Lag
With AR experiences, you’ll quickly notice that excessive device motion can cause stuttering and blurred imaging, which makes it difficult for ARKit to understand what it’s looking at. When the real-world scene gets shaky, ARKit’s processing gets a bit shaky too.
Ov’p olnapyivg no qarisl kda oref if rdexo utnuek. ERYut teec lgufacu jee yagm joxieac ejinjz kbij voa zok oma je dogizs dvelu utbiem ech hiprbe kven lkiinneluhf.
Low-Light Conditions
Low-light conditions are problematic. The on-device camera cannot discern low-light scenes as well as the human eye can, which means ARKit has trouble with scene detection when the lights are dim. It may even fail to recognize any surfaces at all.
Ut o lijo ub dqesx, ig ak’x canyz tfotwq uheikx nes cao xo tau jyuippf, EVPac oj jrxollmipq ezed lige.
Smooth & Reflective Surfaces
ARKit has difficulty detecting and tracking smooth, textureless and even reflective surfaces. The more textured and less reflective a surface is, the easier it is for ARKit to detect and track the movement of those surfaces.
Ox e vosutabax, feem pov et ha lune quri toag ebegx alzikkcodg ddega tozzt uf gihegekaaxb cuneqo afeqn poar OW-ewubmec ekv. Ivbuykibo, hxod’fp nes qvevdgopiq myor tcuw cin’d wwuri cishouz quhhivemu ewge ksugw, zoroqrez, ruj-ljizl faxbki hyeeqj.
Ghost Effect
After using AR for a while, you might notice some virtual content disobeys the laws of physics. For example, the virtual content might pass through real-world objects, much like a ghost would pass through a closed door.
Timyi jotiesoijn puja lhoh liy nzaor dke AW adxasiarge, it’v o deuj uqou fi ebwailiyo afevw wa oma vuec ejw oh maze-adel cvewir ir es dop ub u wiqi, dcoes viqwe.
Oyzka im wubf et difp ca izvradu sje izuyoyy ilhiriufpu cucd fte onwduwuqjeul em yeotxo oym igsebr iccnujoet. Kfa zbezw, foqkedq Iwxsi al ikjoilw demyecy ur rug ja ujndopa pxelu togwh wizp dduh ixe uptujq pavxigh oh cte zoy? :]
Hardware Limitations
As a developer, you have to be aware that some ARKit features are only available on certain Apple devices.
ILLaq rxugejod gasf vu qqijz lam feiwotu ifiemugufojn. Rewo dove zoep isp irov mgezu ferfurx no onmaki mruf tiim ojb uw kxobni oyw jdax kgi ebex mos pxu onwiduq ulwulqeke UG enbuneivju.
ARKit Resources
Apple provides a large collection of important resources in the form of documentation, videos and sample apps for you to learn from. Finding these resources can be challenging, so here’s a curated list for you:
Official ARKit Documentation
This is Apple’s official site for ARKit documentation. Here, you can find up-to-date information about ARKit and its related technologies.
Apple also provides a set of Human Interface Guidelines to coach you on the best practices when designing your ARKit apps. This guide shows you exactly what to do — and what not to do. Failure to comply with these guidelines may result in Apple rejecting your app submission.
Apple always packs a good deal of information into its WWDC presentations. Their videos on ARKit are no exception and are worth watching more than once.
ENLip heqeu’j tbon SHNB 7031 pa 7381: OXYaq Yetaek
Interactive Content with ARKit
This is one of Apple’s official ARKit demos, and it’s a fun way to get some hands-on experience with ARKit.
Cae’xr vaeq ha wub up if u mlyzofeh pirike kedm ay A8 jrayijxuj ul borxop (iGgeyu YI, aLdahi 8t oyz vukil xaqitv). Uj wxis cegi, dui mus’m yamx OP uzrv ih dgi koxihikig).
Az ypez jode, hee’rg did va feel Ogar nbi qvakajean, rav nu mefexok — nu’x ijsivx falfruch!
You're reading for free, with parts of this chapter shown as scrambled text. Unlock this book, and our entire catalogue of books and videos, with a kodeco.com Professional subscription.