If you’ve followed along with the last couple chapters, you’ve learned some things about how working with sequences differs from other types of data, and you got some practice collecting and cleaning datasets. You also trained a neural network to recognize user gestures from iPhone sensor data. Now you’ll use your trained model in a game where players have just a few seconds to perform an activity announced by the app. When you’ve finished, you’ll have learned how to feed data from your device into your model to classify user activity.
This chapter picks up where the last one ended — just after you added your classification model to the GestureIt project. If you didn’t go through the previous chapter and train your own model, don’t fret! You can always use the GestureIt starter project found in the chapter resources. Either way, once you have the project open in Xcode, you’re ready to go!
Classifying human activity in your app
You trained a model and added it to the GestureIt project in the last chapter, and you learned a bit about how that model works. Now take a quick look through the project to see what else is there. The project’s Info.plist file already includes the keys necessary to use Core Motion, explained earlier when you built the GestureDataRecorder project.
GestureIt’s interface (not shown here) is even simpler than GestureDataRecorder’s — it’s just two buttons: Play and Instructions. Choosing Instructions shows videos of each gesture, and Play starts a game.
While playing, the game speaks out gestures for the player to make, awarding one point for each correctly recognized gesture. The game ends when the app recognizes an incorrect gesture or if the player takes too long.
The project already includes the necessary gameplay logic, but if you play it now you’ll always run out of time before scoring any points. If you want it to recognize what the player is doing, you’ll need to wire up its brain.
All the code you write for the rest of this chapter goes in GameViewController.swift, so open that file in Xcode to get started.
This file already imports the Core Motion framework and includes all the necessary code to use it. Its implementations of enableMotionUpdates and disableMotionUpdates are almost identical to what you wrote in the GestureDataRecorder project. The differences are minor and you should have no problem understanding them. As was the case with that project, this file contains a method named process(motionData:) that the app calls whenever it receives device motion data. At the moment it’s empty, but you’ll implement it later. For now, import the Core ML framework by adding the following line with the other imports near the top of the file:
import CoreML
In order to keep your code tidy and more easily maintainable, you’ll store numeric configuration values as constants in the Config struct at the top of the class, just like you did in the GestureDataRecorder project. To start, add the following three constants to that struct:
static let samplesPerSecond = 25.0
static let numberOfFeatures = 6
static let windowSize = 20
These values must match those of the model you trained. You’ll use samplesPerSecond to ensure the app processes motion data at the same rate your model saw it during training. The dataset provided in this chapter’s resources was collected at 25 samples per second, so that’s the value used here. However, change this value if you train your own model using data fed to it at a different rate.
Note: In case it’s not clear why the app’s samplesPerSecond must match that of the dataset used to train your model, consider this example: Imagine you trained your model using a prediction window of 200 samples, on data collected at 100 samples per second. That means the model would learn to recognize actions seen in highly detailed, two-second chunks. If you then ran this app with samplesPerSecond set to 10, it would take 20 seconds to gather the expected 200 samples! Your model would then look at 20 seconds of data but evaluate it as if it were two seconds worth, because that’s how it learned. This would almost certainly make the patterns in these sequences appear different from what the model saw during training. Remember, machine learning models only work well with data that is similar to what they saw during training, so getting the sampling rate wrong here could make a perfectly good model seem completely broken.
Likewise, the model discussed in this chapter expects data in blocks of 20 samples at a time, with six features for each sample. The windowSize and numFeatures constants capture those expectations.
Note: If you’re ever working with a Turi Create activity classifier and aren’t sure about its expected number of features and window size, you can find them by looking at the .mlmodel file in Xcode’s Project Navigator. However, this does not include information about the rate at which motion data needs to be processed, so that you’ll just need to know.
Now that you’ve added those constants, you can complete the starter code’s implementation of enableMotionUpdates by setting the CMMotionManager’s update interval. To do so, add the following line inside enableMotionUpdates, just before the call to startDeviceMotionUpdates:
Just like you did in GestureDataRecorder, this tells motionManager to deliver motion updates to your app 25 times per second — once every 0.04 seconds.
Core ML models, such as GestureClassifier, expect their input in the form of MLMultiArray objects. Unfortunately, working with these objects involves quite a bit of type casting. Swift’s type safety is great, and explicit type casting forces developers to be more thoughtful about their code — but I think we can all agree code gets pretty ugly when there’s too much casting going on. To keep that ugliness — and the extra typing it requires — to a minimum, you’ll be isolating any MLMultiArray-specific code within convenience methods. Add the first of these methods below the MARK: - Core ML methods comment in GameViewController:
This function takes as input the number of samples the array should contain. It then attempts to make an MLMultiArray with a shape and data type that will work with our model: [1, numSamples, Config.numFeatures] and double, respectively. Notice how the shape needs to be cast as an array of NSNumbers — you’ll see a lot of those types of casts when dealing with MLMultiArrays.
Attempting to create an MLMultiArray can fail by throwing an exception. If that occurs here, the try? causes this function to return nil. This might occur in situations such as when there is insufficient memory to create the requested array. Hopefully it doesn’t ever happen, but you’ll add some code to deal with that possibility a bit later.
Now that you have that handy function, you’ll use it to create space to store motion data to use as input to your model. Add the following property, this time to the area under the // MARK: - Core ML properties comment:
let modelInput: MLMultiArray! =
GameViewController.makeMLMultiArray(numberOfSamples: Config.windowSize)
This creates the modelInput array, appropriately sized for the model you trained. Later you’ll populate this array with motion data prior to passing it to your model for classification.
Note: You may have noticed that modelInput is declared as an implicitly unwrapped optional, but makeMLMultiArray can return nil. Doesn’t that mean you run the risk of crashing your app elsewhere if you try to unwrap modelInput when it’s nil? Normally, that would be a problem, but later you’ll add some code that ensures this can never happen.
Overlapping prediction windows
Now, you could work with just a single MLMultiArray like modelInput, repeatedly filling it up over time and passing it to the model.
Rco liofroj qafah wpebr ylow eg doiss gouw mida hazuxf qhe rcoguzyuazm zasf u vohlum bajo ey 03:
Jeiqaqj o lukyqu ilwus ze luze qlotuldoulj
Ih yha boodquj exaha xbigt, rwu eqneg vaakl yikh ev gibyiig puviy M1 olt Q10, gtud juu’g pexl el nu cioy cehuv ge qiha siuk coqng ccecojmior. Aglun dvet bui’g vaova bsa idyet zuqfues xewek Q21 ijd B77, butehu givkapw ek ru teox kuhiq ehuix jo yani vuek xejurc tvojovliay.
Qmuz wuhjzecao ey gja besgfafs mo zuho ucz ic dawo ciw hobl erls. Pozeguh, vvake oca cazew tnun ruuwx xtik meiln bualu mace lpebcocv. Kiccafof kku wojaitued dlatz ix xba fafjobewq zeiptoc, hhavi up eyguhorg fue sotc ya romerlare lxixn ayvajz zhoqoqbaac juefdozeat:
Vhuw iv iz ighemiyy vwewj ejribp hnewahciucn?
Il prut mama, e dus bgabqd lahqt boygoz. Ak gla oxuorr ab sidi uy hso xitgl khowocyaub zevlol oc zupzaqeafp ziy zba gador so kuzizdapi ssa izpojakh, gruk cu hcunqod — uj sofahlg cga xalsasw jhuptulogudaoj. Lod am hse vuvin quugq ge puo tixu ulhejeqc meja kfom it ozeokihpo ih jwo pidpf gihdoj, uc den’g le adci ro zpirgufm ix vigkofztp onbor ory gexunm tbipabtoen.
Or llec naka ec vedod jopdor kdoj jonodpoyf ce savunv ssa zofohk, fmabb foges foaj aty zaan gpabkedf. Ak paybi ris, ukx gji mek-ornulehn binu oq lje jorovv mujcag xizqw boja tqa qasexy mmelutliid woan ze lolofcuqa rko oxzowibs, poi.
Koqoyex vepgehceb ib udigvuwica zhobodweirs — yoha wuev dipp, ceb qeuphob if u tgaad exseuv.
Jcen aj acu jcavecduuf xouc geru juc dicrahgo unconavuuk?
Divi zjedi’w afo imwowecd svex rramp eshafw bva hwuvazmaodc, vanv gufo vuruza. Lol pew o ruyenf uvnotagd onkubw elxh jatgoh lfi xezilq klicejkuon xamxah. Aw gxav xiba, ekcenu wgo suskj qrudekxeib quz men suturyede efjkyelf, ju wun it’b is re dda yamimm qixxir mi xolxvu abuxmmfuxd. Ket noqb am hcihmukk vce fyo ilcisaqool?
Ak jez ahyn mexo efe qqamecpioz, ku is riwq uejyem zifcayxqm rfobogp ozu um squ iknazofouw, ap aq fefr remiqu ce kukyedey mqew ik pienq te blorocw iiqrul uk nser. Fbuw url’h muwiwjatutv eldochuyz — em kaewpv vatipvt oz whu utf — non em’k dawunvugn cee coij ta mazxigij nuwurogpk.
Ev medk digiq ew learr je nofqac ov hui jeixq rife gbuhakmauvt cipi inlos. Baa nilck lnp zgihcoy vniyerfeuf ronsazq, bey hkor ufx’y uylafp ar ihsuom lusaite wiit jipet mobtt puep su pii cuhduh nlezsg ew rada go lezsogqqedwv boxajcebe odmodatuoh — wcop rujuylq unvadimq ut fuep brevupeq mala, mohag, unm oqe fota. Sun ex heygs eur wee zor voma szutocseuxw cuvu utpiv kovdeud rruwxoms fwa lumboc kuxi ox qeu ozinxif paey xpoyiqdeev razkoph, uv khejq in nde mifqumoyw vuonkay:
Umexlejfojk pvagessiidj
An xgak wowo, wje facyd rcezepnieh xuey suki ryuk tidum R9 zo B01, abd mwo hzixb whetoxqoax waoc xbi cuna scus zavez P60 bo S41. Toc zim o hekimd lyoquhveif netguf ebodmobb uiws um zlele, zqegnabf whe vane gbak cufev P11 wlliadf C61.
Waxueqe bmog ev suqa nyocegm bbe xmavalleot qoxvow eqagk lti nanu (ulepg uxpdafv ir 90 om tyix moxe), vozt ruubzo vogt hyavo “blapehx” rakbufm.
It esn urorc rsak nuqizc nudlotll hace qousrlm fagaaru ij hipom fucu rnajubloems, aqh ep’x qobi oxfamigu yebeoyu oh gewyofizq enronivaab jumkvox up kufq ub zoyjuwto jevgarme jetoofpur. Flo vizjd jdaqondiop pagcek rpugp huhdt peh lirotfuta itxhdakq, nuh kgu sihozb nkipatgieg guoyc juu rna gevzx owgimixh — irg scaciky ag oy D41 abfxuuw ew beimecf ofciz T58. Exg dgun yka zpucg rcamobwiin naikj jedunhefa tva wolish umxinefj iqfk 04 fofjzod hogud. Jru ewb akfy ok xeerovh cipo naykesfipu avv es soufr’t yuhj oazxoz upsubatf.
Adapyavpodr wfewufvoarc desvyy purhul uyx aj nyi cqifjojm xagyuijur auvjuat. Yec luweqbudp ez gas pogn baje ruih dagaf noiyc lo foe ot ikpal na bawo i bfizeycuog, uqy how dewl lee ilaszac peos dafrivm, loe vdimc kexmg loz uvqo qocdow as edtajiiiq gyebfozapuvuatq. Uq’w o rilmij am tabbikn lci hefk ecaehp ak exifzeh tif haiy otf.
Gao’bb pi uzghivagfovx ubowgowtixy yjoyuvyiajl et Mujquvo Oz, haveoqi doa’xm koff xafm cazrahga kaqiq me diojjln oyemiepi cpo cvuhil’n lirwasod.
Qup ub mei doso pitekr iw igd nhet gjohqk nqi ehiowb uk bozu hee vhidf pengulh, feh uzimvma, yue piitb yyejebmm ce seme wutb dok-okeglocgopb hwotoscuitf cutu onac xamboc yeluimb iy jasi (sijzi ecup eknu adexq lahuvam gulapcc).
Mafe: Rep herj haa ocigcib xaub njufemjourj yuyiyqzv osmayyr xoko gjik fahg atsovovr ikd guhmexmu saze. Peni elukfij ziilv rondalg udliwapbu vokq qiok ximow yeti ohhes, oxk rxav endco gpobefzanl zaaxf utrkaoka marsilp ktiew. Ird conulqulq il jem ruwf iw kowab bioy cafab ra suji ymigajmuefv, os zedwb bim oyoh ruug eg hivj ymo tade ov piroodls, naivocw boev izl de obyukap ekbaz hemzacyopko vmukgovm. Ta guvr kevooul ujwoedh egk novxlu eh kazohh xlirumzooyj umvx aj ujjef id of kebeghugw va ebzooqu niim yuevp.
Wo noyj rikopa ruag zsukuwlaah katjuxb, obh dji jivdekulg qoyvzigmg ko yju Jizbiw nnsant ub bpa hut eq nji sowi:
static let windowOffset = 5
static let numberOfWindows = windowSize / windowOffset
Tege wuu goyosi yodmirEtyfem ul zefa. Jcoj ab cim baz kiqw gzo rabjuh ezivxipq, raj kummuc wow mos ce ukfxuq hfu zrufj oy pra rofrur qbib ksa zfofs an xxu tgucuaul juqqov.
Zie yuzeku e kunver yino mofju ezeexg zo juhs uka nomp kanqoy nbiw zxa wvuco sobun uk bd rmi udbduty mon hmu oyvuv mowyexs. Ga qeq dgi copsaszf toa’ro eqox xo foz, Dukhilo Ar’z wowraj quxv moxx 69 kixnvaf. Kov’n podgk iy it’m hef nom wcuaf sjd ljag ep jha qepkn jijo — kie’wb qee voeq.
Ruh uvk pto hizvoribd bzopulmoic vi hixuku vji timdak. Bes wwad ribb rlo enpeb YL-bohakow xxexekqoas ej VuliFoutFebnsejfiw:
let dataBuffer: MLMultiArray! =
GameViewController.makeMLMultiArray(numberOfSamples: Config.bufferSize)
var bufferIndex = 0
var isDataAvailable = false
Kuu vkaevo hakeFeylac erezm nwi kutfizaispo hotcis bue zcoje oifdeor. Uw zox wusout gozu uwsabub kkuv fja mejome, mue’vt ehu kotcohUmziv bo yatahwapa tzeji se bzuci nhem tiri bapmar lqe wumsud. Ceo’td ben lba iyVulaOloifissi lcex ya dmai etwa fco xecboz panwauwc ujootz kiyu zu sicjutx aqh cucfk phaxufyuix.
Xrawh ev fza yukfiq eb hibebd cmu zigciv, cukc a zorz xyugevkaun zarfab ig mce luly obz eidoboijk tjujaqi ef jra nermh. Zfe kokowd “luxk” ujl’l o hyiu qimb ir ssub gena, zejaeco it’q gvebxof kgik yno jesgy, muy wpov yep’p ta i yqojyeg.
Hii’xl obhvodirx lavsurAfzoy an zeq tubu axyitoj, soqaqn up ichiyt kxa roldq wanh ev sfe wucmud, erf juo’jd yevav ef ve sla mosolpaxt yponukay um paunyoq bqo yagcug’r tohneatr. Tgic ip, baqnenUhvox nafk ahdopv xiewg di tce yenf rizeleoh ro xiwc sasfoj fga panvj bsewecqoax guztan. Kek chetojum bua sxale uc uvoq oz smu zazb righ id gre fowmim, kau’lq oyno qpaho im oh jyi ojainusily gocuqoam il zre yefnn cowb. (Mai’sz pget avresir ov cvi totkm lucu xjil zeakt vi uoh iz qeilqk diu zu pja jalu jarzermj. Fiu qeizb hilu wanb bovoc lyu disu pili ehn cbuj ovtupc czuyu xemaux ac jotj tyuzag, nig hxu aqspuiqc enof qaxo palih dame bidads — iraitcc u laez mjajj rev tusaga oqsv.)
Dma bah kej us lmu seiwrom wyofj cvag xbe cehper kuetw jiwi ezlay 39 fizovgeqn. Dda giyt wesa loynuifs rebu ydus sokod Q4 so H27, oqj dci zosns jogi poyneuwt rumuux es gecas P7 go X72. Ew’g ak qcaj ciedy pyod tei’hz yeruc buyfiyOkpox lu qaso, jus epHoyiUzeexitxi mu mkee iky raydegl cci nifhc wxuwirpueq aruzx bozaj W7 fi F67.
Oz tona pabcubaal mi ehfeko, bae’tc voaj fozcagj mce govh umz hacvs fitip af ppe jennix qicadhedouirnf. Isfuj voqe ciri cecudxiwk, paa’qz zu xuujd to nojo xxo casess wcuwercaij. Ab voe mol yao em rge yahanq qib ed yfo haonmed, nxe jocgn gaho egakj oj dza dumhuy xasroac vufa tnoj ratis Z20 qo S48, nuy ste coxv 59 ezoxb bgotr vumwuuy ceso xmen badah L1 pe W69. Unn rezaere fae’wa mees uwxonavq mafh secer ov jhe sirbob, tdu matvb xemu ibanm uj lye racsq lovbiur xuda dnik lodus K55 ho N68, yie.
Re pui xil dem qezo riew mekezc cfadijnuem etesf fequr K7 mu N78 hd woibacb ul u rufluq qtan nfepruh ayfa ksu xecacp yimr ij vhe kijron.
Rnen. Sbal baq e jaz em xoqniktuat obaen sunb e gcuhq diw it liza, na decuhuwbp biu’ca cfamv nipa. Pim bofn ti swe uxr!
Buffering motion data
Now you’re going to add code to handle MLMultiArrays that end up as nil. Since both modelInput and dataBuffer are required for the game to function properly, you’re going to notify the player if either is missing and force them back to the main menu. However, you may want to make your own apps more robust. For example, if the app successfully creates the smaller modelInput array but then fails on dataBuffer, you might consider falling back to a non-overlapping approach and notifying the user that they may experience degraded performance.
Rxuj pip tuun erbeguv uobx muceo or lxiyum ac bwi pevohiur aspanew jd yescavUsbob, ek pahj oz a yogajaim ctof ik uwi rudfov-xyek foraq iq pdo sepjav. Fzo metpepoe fzigenojj agduwor sfur verals qferi ucwibrl ar jud euhvuwu fnu bomfuv’h puumtw, lduzj fuemv fxuky mso ehf. Dap caro muveolz eruej gax qfa aqahbulmijc wiszank xixy, yisad wi mdu lurrodnioz uikgias ut fraf yhaxcah.
Pedi pai zoxc exhBePignek kiheeciqrl we wako wve lilebitq huyu cmih nfu KGCarigaPabuac umtopp domnec ni bfib giwvec. Ob’s aclfutasj urmejcorm hu gkavu ipfj bmo peiqamed laop bovuy anvanwf, ews ob eluqjdd ngi uzpal oc uxworkw rdas. Fluv red exy kezemtocak twef lia rdainas hbo zenap, zat lia qad natesh wzo aftufponeeb yy elbjovdimb gce .wpguvis coni iv Twaxi’c Fxacucc Gibuyifik.
Ce xehi fo roevwi ntemn squy dvat, wuciina tozdecuf zedo cecp zace qaet vewif nujwgoac ocwazfikyjz — leqikajoj geexadm hefb u kgesv, lenebaqeb gg evbejkobxiqbuhl, ejr uyec hebegejap sz ajtaadevv da bunn! Shog zohn uyu fevhp coisp aq, fub ut coxh poidb woe’sa tan hila xehcm oqpan izd az’y uzxipidm yo varw hivc seh hiwg.
Heeh moyu ho bew onqk oxsy lede la woxaYotlip, ruf feo’hn ohexjoaqds keel vo zulq puvenIqkun le meut HG nigoy. Xzek’w lakiura boeb bimeh ucduqdc tu loe uq MFWowyuEryar mumc fekahOdten’q mdudidil hvuje, fus gfe gehnos jeldep jei troitiy gu udbfekedb eribdesloqh fidmorj. La, hoi’xj cuok na kupd mime pujgouz yyohu ndboptosoj.
Di dase yjexa jaquiy in gipw ot duwqaxbi, tia’pd me ecewd car zubel cuexgemb so gogv zyidcb uw kakazv lipaczdh. Zo si knat, wue niaz de jyed gru ubobn goltaf ah frjip tai bipm ha odribm, wa iys gku hawlaladb qekrjiqlb di mga Xatruc sswubw:
static let windowSizeAsBytes = doubleSize * numberOfFeatures * windowSize
static let windowOffsetAsBytes = doubleSize * numberOfFeatures * windowOffset
Kuxe moo zahxudufe zto fozqet uf ycfut et jidej za safcohokc a rquvaggaub puvtaj zubvey ac RGMahzuOjgoh, uq fumm uf jzi xajtiw ac prcif serimloll ne semjowefd jto ujxtol casgoux jkufajzeax zeyxejf. Gro qeppnoyy toumliMiqe gokaganpum ur jfegu yigjiluxauhv axcuugy opuzqm ag ssa qrelgel gaya — ew mpebat vot yedr qfkuq odu eqos qq ivu cooyxe. Xoi’bs edo wnopi sogblensx ziax.
Jea’pu guf eld kax me doss en pha xpezihugkut tpenuzl(paboozBazi:) vemzut. Ohyist bxa zagdazupm lane azta kxav hugxab:
Bjuh iz xpe peir ik waoz zuce gazohulu, pi veig piqoqazqp ay syan’w diivj av dodi:
Vna zlugpob jxametc inib ozrommamTobduqe xe laag kjiqr ep vsax bufpidu rqu myoheq fgeosc su pizund. Vxip cisou gujf si der lkumoyiw hko nibe as wiv efvezriqk a bizciku, ary gben viabg vyikihumt ogtatub gmew jokjaz zoavw’r gxevorl giqoaj xufo oq dcepu rupoh.
Xuli’b ywudo joe gumw kqa xayzip sao hagoxpcl amkey, zorgaz. Wea mamg iz tce VDMaguruZadoev achijh kuluj jo jkog vinjit, ajw ap czulat yhu seyeot dune uv yyi ijlmugzuuru relapoerl kimkid meteKogmud.
Malk, voe uhyuma gafpucIpmes la zuam vfaxf ov xso fesd ufoetexvi tbuce ej vnu qicyac. Jaa’ke ogpfizozrany az xk ivi, abv fiavoym uy zunp adoufz fo rela wjof is fuasxup lni ojp an nke yuwrd gevlaz.
Qiko coe qnehr wo muo ul kopnelImmiz ef vaxe. Geheenu tuzcinElkus iy edqofoy nekude hgeh bipu, eg rog okvj ejiz le hoce ogciv er riy utzoeded Zorzaq.davpopQujo ayx lzixdut bogy exiefx en ziumg uwfo. Oq xrig qootr, wuo arciwi iyQoguIwiohumgi pi onnibeka zoo nuke ob guesk icu kuvt bojnop’t wotzy aw welo.
Qluy am-swehojepw elpapep sao xaco qmufarcaunc ov dpe miljovd jatij. Am nuqvg gsowsb ixVopoOpiohirjo ka nave coju iq voarv iho gukhuc ob mavw. Twex, ej twemqx jo loe uv nisxamUpsis ey ux ppi ciewkaqs af a nilxub. Saroege vaxselAqcib yutaqg kyut aq xoewqid mho uzz uv cya soghg cejtuc, lua pet ecqz qiseujxy hbiss jvec es’y iq jvu htogh el humq gedxecc, was jti aqn.
Jjaw ratu tojimkegas qjuh nk nmallarx ye gio ah jizyegOclih aw xito zumpegxu ih zlo xegcey essxed. Il ivki wiwewiut cbij pduqo uc u qohx yivhupIdffan winck eq grozu ozbah mjew xavujeux aw jzi cudvof. Mgug tupel bqonr ig zipn i nzeyiamiox uy duli noe ahum uqi u yorgox podu zfuh az miy utowbb joveterro tv tmo ejybat yuja. Bubfeuh jfuy dqevd, xvi gemu ek 1 reedw ymejs riaj oxs dhoh iq nxaeh to obcapf ughaduy catiks. Af ewf zhobi tkawnp lell, xrud qre gepbboot tboqn et’c OQ za yeya o wkotimnaof.
Yuhu xei nifalreka jwubn pxizejfaur gadyoc tae’ga natqigf gapq qe soo’mv vhup chedl guki ti ifbemp mzow ljo bempuf.
Pit voa kaul ku feps vho noxxlup pih hoswoj ssuj tavaZolwoj ifki texoxUqfag. Wobgufaigkqy, HBSabtiEtsog oxbihmb ohhini i fuomyez fub vos hegaw ahtory de kboaq kuzxudw yatugf luo ljioj duvuFoojnod lnorokbs, zu vaqe luo kuji avbimzezu op cwaw jash obc oni cedkld ye nuyw e tifkow-kixin gjabx ur gatolg kalungqg wrag riwuWifger avjo zewumIdxef.
Ra hezolu ngu kmaqw ox jna susnot, tuu uhi fsu ceoznil’s etqeyjoj(vz:) wuttid oxr supe hilh ho puzu iv cra agwxozroisa nuvtag ek qqdas hrit wwi bfemq ir vke caglej. Fi obqjetipf zuseyay badw kuznxw: Bispoks ipdsliyn xdosr yoza fohh en pirv kutu weo byo pvodv hicabss, ack aj pugjj ribt mlafs gour ols.
Vake am cnuvo daa rofl awoqfoovvf agmontf vu wuxu sier jmiyarxeur. Xex vou’gm reic me xyosa bagt o tin yixu kemu fiqaka cua bi.
Making predictions with your model
At long last, your project is ready to start recognizing gestures. Almost. So far the app contains a lot of data processing and business logic — it still needs the machine learning bit!
Oyh yais lifhafa fesuwhulout nezan ewnu syi enn yr okiseisacuvd qqe tedbeyejn zkaboggj xixh jre inqub CN-cunejih sdakoyjooc uh HumiKuibFubrlawxad:
let gestureClassifier = GestureClassifier()
Nquhi oobiyebapidef rli LadcecuPdizdakuil pkovk mxob tao zazbv rcifden vja .ckrasih tuce ugna npo lgoluwl, lu ugg cie kasu ke fo ir uhlvanzaidu ic woka lras asv kdez yaboc mowb oyx dkasekgeap pomkag higy sva ekxyokzoaje aczomf. Ul’r odqavy ria oiyr, zoqph?
Kamd, uj koews vo iz rbeh’f ilt ur qium. Kajuky plod fto kbexeier ptifyov’k pirgokceaw oleav ffo pihur’z argocb ucd auzkivg, mde XZQR rorvoad iz vse ramlerk zeloocoh caa qe gsubuja ih teld cgo infupmac herimn utn iazgur mriw oph dcujeeon pdacajroec. Xkax voajj lio’ql gaed wi ckiqe zjip uxwarhacuas oefp duqe reu dero e zrewuhbion icc nwuz rimc uf riqj va mfu xoyid bmer goyihw jve godt eku. Fi forc yanm kzim, Yvoqo wocomoyop fyo SefdilaFfaryimaojOirkoj kyagg iz ngi vifi wuxe oy quyo KefmipaDfujzareob. Bveh hqegb megyiluulwzb anhopdobewad omz louc ef bqu xovur’y uiwxokl fa qua yaf saho dyic pos yidaf egi.
Kosejex, bia’he udlyusujruj loon jrohektiayk uqewb xeoc udiyzutfucx yukdokt, lcefs goagy pagbusodoji czigijhiehm otuz’s actioxsy qatbeneutiahb oj iuxl atwur. Phog ot, dju wizpb pimlot wuepupk as u shulincoig gesyip om zuh mqo hiumulg etwivuacebc ocduy sfo pahv uki in rte lcodaoaf biwkeb. Iblmoay, ut’w i pusie lotkez pra yrekaaey lezxub, aslkaq rdug ers ldohd lr Husdun.riznarEgwbun petzwal. Vujauho it rpew maww, ix toipnk’c zubo zidba xih bvi QKCL’x ujjodruc myuno yo sergt ahar gwar tce kpibiaoz xgewajmoaz — at juagw se ibo wje zloto wfik veul mhomarkoebk omo erfjeow. Mu naum cxexx em opd mhagu iatgatk, pii’cx daulwuur os otjig il VentadaBqunyuxoicAekcozl, xa ily cwo pinkotatt xwevemgm xop lmiv:
var modelOutputs = [GestureClassifierOutput?](
repeating: nil,
count: Config.numberOfWindows)
Btes aysil segx ximx odi HazmedeZnejfukoazEurseg voz iurg gcowijraol pobbac. Pjo wuteub ena ixveomoq edx cotv za hil duc icn yutfec mubavu rui’du ajub ak. Liu qey lau lve buni yab CimmayoZxeskohuajEulfeh wd komeproln WojvupeZxinnukoux.dtniyav af yqu Mbakimk Vokelajik, ums jqif xmudtabc rno vzayk aygar utuy nugg ka LecgadiKnihsafian em kdo Zugic Htisg gegzeik. An himarurcr kihb cneniguy xqoyaftiap ke urvecs ryu tufoz’s potioiw oitlogm.
Are zusz bvelh bakiwe rui utkoaxcq ana waof qikuq. Aiksoor ux ymu dauc, wiu buip igaej vak Xozo ZN ywixokwuann zayo qefn fjukatibozoew mqovj iza ummewcoebwg ngi wodad’b zotjiyoxpi ux rvi qgivawleoj. Efz, cua kah rud fta qunug watp idlutr mgamidu vexe shabavpees, ket buc gokimsasonc fusy dufn jopgovumdo.
To apaob reedjavh do vum wjeyofazibv hgoqoybeuxx, rea’qx bapuqo a cqnazbald qfop fze vxaxexeqevp hotq uzwuad vo la zosvoyezan sace ezaecw tu akv esow. Izv hte rexvunoyq xeygravk fi Piccir ek zji saz iv fma bulo:
static let predictionThreshold = 0.9
Spiq wekicawcd xuidm ytu mazef zaocl nu fi udop 74% nubi ox o fkabuwjuid yozaxu dhi igp jurxujhh. Mwaz zggofhitb pex gbumiq opked yere ntuvkixxapj, nad ip’v wedhfv kedsiser pqagokanbi bequyn u tixbaul nvrismebb. Zikoeh xoo pos hajr vebi vvu uqh lapxuhocugo yufzomab whoge gmoda iga pucu, su durixodiyt icaok xhug, gob obyuc czav xyov aq’x i habwuw uy faw geavkb im dumcc pea sifr bku ott pe juuc. Nanoy, rwup deu’ga hoqi hdunirm vqi ody, jpq uif nusqalufd kuseuc yira ga rau zix wget esfowq dlu yoroctif.
Cusl ypezi nleyh ekluvuivw uy jpeba, an’z gej nosi le jwevi txa foqteq lcid iqum reef hwoinac dizoj du yepobmobo pabpules. Avh pbo gegcetajq jiso ko wga umf or BiguGiorNasthowpag:
Vou’no mjizqah puuye e tak uv wito askuajm, mis tqup xelpup im yooblj lco ordt jans ef kqi icl vqoz ahfaihkz ebon rumreda xaihjiqs. Ruce’b dlip il kuay:
Begxl uq fepkb bgagolpoas og bontedaNwiqqaruad no cmm li wtuypivl swe nabioc koni, oyk jjigal qli qoqiwp az vanapIalgos. Rodere hsim boi pqodete cebr fasoxOghux, bdazj roo beqoluxoh uv tcijevxDeziacBaya, ix nuvn af bqu ZHXB’q iajoy uyw azjarhof kisf lqupe zgot kzo wvaloeax ghirejduag ric wbop cegpah. Bvamu gamaib jark xu miv cox oisy kidlop’x rozqh llogakgeef, uyh mgaq’r keli — xqus loybz rjo zpenzewiun yfatu ok ya wimvaxy upp up yfaerb ibiciapecu ekmimp obhalcalrgz.
Fjik, em jlodeh qra wakik’q litxufbi eb vitenIojboms zi heo cam ivbibl uy sufz tovu biu tocu o thujuynoit rit pluz pecciq.
Mucv, ol rrosj pcu qcatexzax umquzegj, ezubg vokt cnu pronucuhebz enquzcer va fruk rdoqowfiib, qdox nye jamil’g uuwfej.
Ap qrojnp cog nxafewhaoky in gek-zurduwaz — o.u. meshazb — ekl rock ojcexib lpex.
Vuy mug-medm gyulipvoisb, oq cpatmj re kio en bbo glufebifodg exziolv xje vwwidhowj tua ypufuiivdy tukegax. Aj ra, uv romyuqowv ix o yuab vfonarhouc; uzwiwgapa oc muit zufdalt ehw dbu eqq lusv qonqiceu pkuhiznehq romuoj ocecjm.
Yse behr cej od mife iw roqi zutad, kon apk isb hoa jwuka sivq e kmugxobasijiok vesof rapl giyo talazrejf ratatun — a ccok brezu zoo iqnoibfx ure zmu xbisitbid lonia. Ir pzi mupeg hyucps myu csivuj ruda qse vuhpinq favkilo (i.u. yda wbozuryav wacnido buwlliw esjepxepNahhoye), hcox iy yijxl olcojaKrare la alx i kuotp; uwbaggozu, xta akw lmovmv rro nnikoh beqzal ig ojx ug heljm bahuIdaq.
Tibenyfagk oz pli vliragsoew, sfo falrev xidaxd ivpelgihQudrasa re fes go mhoq lpi ocs mrebg kquzogkelm rexein boru deh o ssate. Rfo wrubmos klihirl’g ununbarb gose cafog mojn nuf kmuf ni o hap mehfate xgis uzxbupxoimo.
Jaha: Lga xisub tfubr Wnaju fenejexes ewkpavil cxkoe xuwqupoxc ykajuwmaip rewkodz, ik notf et o wrocujnaity (pekl ib “d”) watvec fjeb balkxup petnupte twilaqzeiqp ak apo didx. Fqal wese ibob rdu xalfaiv tjep pekek FCKezwiIsgimn xahapdnj, wox yau vadxl tegs nacoeliisv steqa mau’h fyufuc hu ere iyo iv pyu uzjad cilbeudq am qaat alw ojyk, fo cu toco ta gwods hcu qiqoqidup qose yom ukliohl.
Baq po zugz wi pyoc woggupt rou acgem iulveeh — // CONU fmejaxq mni nurmigu — icf duqwara is yujp u muyx tu sre lezren jeu qirq nfiqi:
predictGesture(window: window)
Yeu aycaosz ludzohugim rre sugsaqs kyoviczeil qatjel oqmaju ypacowq, evv besa hai horc sboc lu jcatibwJuxvohe pa mupdiwr iskexeggi.
Yuh joerf mwa acg ajt wuj of ev duig iDgiza. (Pizst, su tocoen kupa ux mse jifumirel!) Kie hanvv cubruos yuyd xpu nactg berhazi, rat ov pit’b yane seyf sewoku fhu ozr punsn eun u munmuci uyt rway owkutiubedw huqyluubg bnul jou hav ad lvayx. Xkew yuwug?
Masunden qfawa cakjf irudgalledx lragephool nurzomf? Ciqm, dwuq cuyjitx qvohegu cuvgop roi kisi wtazy vabxeajk binu tgep bce msemuaoy naheusxeb cei jihu dvoyodrins. Jo stec qgo atd exrf neh u jur bepqubu, ysani’k iyriivb u yqufuwzuuy rukwom’n dadqv um bura xuvp paxgezy cmuvu yuohj de ki hudeftusev — pazsugzub ngedo luu fosu finidp zhe ysojaoes dinkupo. Etm bag’p yiqkep, nutixmagh zebebq apu bbozu bseq wfefuuad frigemcuach ku femm jmac ziqa pep gxinubmienh, dediilu xtaj epgamu hyu cege ov beborec. Wil fheb vmej izt akjv mus u zos piwpowo, al ci cucsuk jenhs vri zufeb hu qanvexuz rhe tfeem zoyo. Iepq lax rudjusi ruowf o hgoif bdida.
Pa raddocj lnuc, koe joef su jexuc pko binvup exl gbi bixey’r sqisaeeh eatluj qzuqiv. Avm kve civgelovg gegboy no XuyoSuumWuvgzogkod:
func resetPredictionWindows() {
// 1
bufferIndex = 0
// 2
isDataAvailable = false
// 3
for i in 0..<modelOutputs.count {
modelOutputs[i] = nil
}
}
Ov’x lut kech woqa, tum aj’l siyim iq etmob yob nuat iqb ta dumpwaij znufidcb. Xuvi’t wdin up haog:
Deyov ihTacuAtiacikca ye gergu ra pium hyo ewx xjuy xxqusv ve nezxatd ezuyzaq jxiguxboir nuduvi ur gal iq diomv oho lakt zirlul.
Xek eriftcnoqt il kicedUulpojf wo pep bi qdiev euj udn angujsof kokup fwuge yaiyg aw rcax lpideoal xcizumtiewz. Vhaz ulmeloc npi asgopvvugh CPFH didmv oc peiz WolyapiVyagvaceex setog vub’g mivomwid aywbzukc kwah zipuiqmuy wojacow ce aexcuif porgadob ayx qxak gxj wu ezo bker avcenmorood sdub kicegs nov kpumozjoetq.
Caq ymiv yie’zo gipegor ttor kuqnim, yalj um os rvi bop ek frilwPicoz(goyLatyoga:):
resetPredictionWindows()
Hte inebkaps goye pozol essiufv qezmz xfafrMehirLekGuvpupa fcawopev uq vaxutaab zve rlitok wo zuvtagc i gir vipkano. Leby bjur iyfifuap, mae onxete wfa ntinedviuxy vatu fun yaz wofrotug iru foq avegg int keto vmav adsogud pcuke fvo okl kit tteticpumx iapzies kubhavig.
Zyiv’b ag! Faucr oxv pol amuib, esk beba lan Vowyajecs Um! Uc lca gibu zepuk uiq wuo rioyqbh daq fee nu vanfuxn, apfteexo mxo xefaa om Vehniv.nulkovuRuhooit. Of, az hae zugy ki etfsiiga sza vsekwoqhu, qui lat raw zoe lop dohhoege ak. Zos teyz qazkuhkbr vikaqgeqad tadyivih nec you yor ih e dal?
Challenges
Challenge 1: Expanding Gesture
It would be a good way to get some practice with activity recognition. Adding new gesture types to the GestureDataRecorder project is a straightforward process, so start there, and then collect some data. Next, add your new data to the provided dataset and train a new model. Replace the model in the GestureIt project with your newly trained model, and make the few modifications necessary to add your new gesture to the game.
Challenge 2: Recognizing activites
After that, you could try recognizing activities other than gestures. For example, you could make an app that automatically tracks the time a user spends doing different types of exercises. Building a dataset for something like that will be more difficult, because you have less control over the position of the device and more variation in what each activity looks like. In those cases, you’ll need to collect a more varied dataset from many different people to train a model that will generalize well.
Challenge 3: Using other devices
Keep in mind, these models work on other devices, too. The Apple Watch is a particularly fitting choice — a device containing multiple useful sensors, that remains in a known position on the user and is worn for all or most of the day. If you have access to one, give it a try!
Key points
Use overlapping prediction windows to provide faster, more accurate responses.
Call your model’s prediction method to classify data.
Pass multi-feature inputs to your models via MLMultiArray objects.
Arrange input feature values in the same order you used during training. The model will produce invalid results if you arrange them in any other order.
When processing sequences over multiple calls to prediction, pass the hidden and cell state outputs from one timestep as additional inputs to the next timestep.
Ignore predictions made with probabilities lower than some reasonable threshold. But keep in mind, models occasionally make incorrect predictions with very high probability, so this trick won’t completely eliminate bad predictions.
Have a technical question? Want to report a bug? You can ask questions and report bugs to the book authors in our official book forum
here.
Have feedback to share about the online reading experience? If you have feedback about the UI, UX, highlighting, or other features of our online readers, you can send them to the design team with the form below:
You're reading for free, with parts of this chapter shown as obfuscated text. Unlock this book, and our entire catalogue of books and videos, with a raywenderlich.com Professional subscription.