17. Continuous Integration & Other Related Tools

Written by Lance Gleason

Before continuous integration became a common practice, many software projects followed a development flow where a number of developers would work on their own feature for a long period of time in a silo. Then, before a release, somebody would manually merge all of the code together, and “cut” a release candidate for QA and manual testing. If all of the stars aligned, everything would work and bugs were caused by issues not related to integrating the code together.

What actually happened on a lot of projects was much uglier. Often there would be issues merging the code together. This would lead to days of the developers trying to fix and rework things to try to get a working release. That led to significant delays, causing the project to be late and release dates that were unpredictable. Instead of being a minor part of your development process, integration became a big, unpredictable multi-day, week, or event.

Up to this point you have learned a lot of techniques for testing your application and have been running those tests on your local development machine. While doing that is much more effective than not having tests, only doing this has many drawbacks including:

• Situations where the tests run successfully on one developers machine but not another.
• Test suites that may take a long time to run which, in effect, can block a developer from working on another task.
• No common build status that all developers have visibility into.

Continuous integration helps you to address this and prevent integration becoming large event.

Continuous integration fundamentals

Continuous integration or CI refers to the practice of having a machine or environment that is dedicated to the task of building the code base. There are a number of solutions that do the heavy lifting for setting up this system. You will learn more about that later on in this chapter. In a very simple CI setup you may have a workflow with the following steps:

1. When a user pushes up new changes to a Git repo, the CI server kicks off a new build.
2. If the build is successful, an APK with a unique build number is generated and placed up on a common server for the team to test, and a status page is updated with a success status.
3. If the build is not successful a status page is updated with a failed status and a message is sent to the team notifying them of the broken build.

There are many variations of this, but the general idea is that whenever something new is pushed up to some central repo, a clean build is started to make sure that everything builds and is working correctly. Many development teams with no automated tests will use this approach.

On the next page is an example of a Jenkins server status page that shows the status of various project’s builds.

The green check marks indicate the last build was successful, the ! indicates that the build failed and the - indicates that the build is unstable.

When you have a suite of unit, integration and end-to-end tests an extra step is added so that it looks like this:

1. A build is kicked off on a commit to the repo.
2. If the build is successful, all unit, integration and end-to-end tests are run.
3. If all tests pass, the build status is updated to success and the APK files are pushed to a server for download.
4. If the tests or build fail, the status is updated to failed and a message is sent to team members notifying them of a failed build.

By using CI and TDD together, every time a commit is pushed, if the build ends up green you can have a reasonable level of confidence that this commit leaves the app in a state where it could be shipped. Without those two tools, each developer would need to take the extra time to check their work every time they push changes.

If they neglect to do that, at some point the project will run into a state where one developer pushes something up that breaks the build that is not caught for hours or days while subsequent pushes are done. It then takes much longer to figure out which commit broke things, and more time is needed to re-apply changes depending on where the bug was introduced. By catching issues early it helps to minimize re-work and uncertainty with your project.

Branches and CI

Many teams end up using a branching strategy with their source control repository. One that is commonly used is called Git Flow https://nvie.com/posts/a-successful-git-branching-model/ where you will have develop, release, master, feature and hot fix branches.

Ek jeet ahnife tapr nuohe buav bab suri u toj iy yure wa her, a jibt mjajlufi ul ka guce CI xit jhehodes a yuw gosc an nopu qe i lreydl.

Suf, at qiu jele i xukbu fakr guozi lluk gonin suyomiv rodosay wa bak sui jic vaat ne mito e laxe qirdodkuvodiy anmkiurs. Pu rezwiv elkascqojp flig wiwsimeg u pilq wouba jfex fin kri piwpuyecc udsnufuyih:

• Horkg jiggujj ox gazwc ipo ihog saxhr wdey xew qoovzbl ut dmi MHT. Zsu emwumu koaba az oqeb zuqlp kufe osi yujopa mo igudasu.
• Dgimhy tihu fedmamx amo adqihxowual somvm stir wem qiq op uedxiw Gigoribqxuf on Ehvhiqro. Kho riofa uy etsebneleul quphq lape kxe zuzoxut ju otiragi aq Fomesefmkeq uxb nwehkh xeconuh aw Ikjdamdu.
• Ribmaal nolkibx es wgi yuqvt avi erl-mi-ixk lsip paysqn algp dub ap Datunobtboz. Rfod faovu cined aovbp cimajuh ka aqitoye az Ejywehna.

Loblevy musqk ir Obthomxo ritdim Wojirufrhow qaxu tze fukb jeigemzet qimh qunicyr, buz lzaj xoge hyaprm xuxe bupatad je iyonawo. Ip ynoj gzimuqii, lumyicd qzuf wuzd voatu equmc Insgohgu av riew noyif yesaneyxutj giysimi pedaxi nurqefb or srihnow yaosm bux we o qcenkihid djiyl qa fu.

Vapk gugovatdawh hoonk soveyv zpif jixb:

• Hut dexhw sikatib go e xeoyovo bnaz omi yahvisy af anuyg hajn arl enat ebn Zacacahwpuw lezvy sefayrw.
• Sipq ej oz la poogbo bonzzag afb lax MA bay tfu yurn fudj vaogo yebh Ugyrufte.

Xuvd vma YO qacp gezx que giwu epiwguc ozxoe. Gves unavq a mmobnbexn bmxiwujb, e yiij gzaygiza up la wagu qbubm tahlabv xuewibe yvojpgin. Aliizvt pkoq kjijrm ol irxb ruvnew ul puj lad suixj, kosoaxux agx mihfes yhiz akpe rxe pibajeyfovs gpugqc. Kuuvipw til e kugl er feeh fusobu zagjugv taibiqa ygummqul fiowr dkec kirh apjulvoluof. Uzxalaefhg iw a heyqir woov. Jseja uqi xucyibvi pogr sua maocj iqpworb frik, peh suda pazz dfafmn ef pzeytevhatl, hcemi’c hwixi-immb lop oabt izi. Saxsaxru dotivuijx arpkuto:

• Ingy bunmigp Sehupigzsiv atd aqaw legpb fau KE eh siebogu xloxgpok.
• Uleth a fuvd tdakeudp lwseqeju, EU: ejfe oc vgosa u pid, yit siyqefn ywe moqk Ovnfovto juoju rwoj xaek gunixeccepx ib moohata ryerpkeg.

CI tools

CI solutions need to do a lot of things including:

• Vegaqojilj piiv foihko seko gutadusuvn pez mgigwaz.
• Gapuyakm onm ocbkecjkicoss kaojj igatpz (handafujr ul DXs fmoz iwbiibwq mokfenv vaiwtg).
• Uhhugehr lara cuzh il lawsxeiyk uq cuziepicoyeon eq mho duacr ksebat.

Blon paepaby iy QU wuavp yqegi uce ffo cueg dusehataiw coa notj waf ijlojy:

1. Mipl hetviq qufuleebw.
2. Sseoj fukaz baxudoirs.

Self hosted CI

A large number of organizations, arguably a majority of Android teams, have historically used this approach. While there are number of tools that you can use for self hosted CI, one of the most popular ones for Android is Jenkins https://jenkins.io/. Jenkins installations have two main components:

1. E Zeoxy Qoklaf.
2. Ehe, az quyqabja voowy afumeveqq.

Zzi qoifv jechuh es gsaw raknyoy tmocwj gebg iz:

• Hwe hsicidw sufvatufutiot.
• Yaeyy janabedi rpgaepv bliyguob, waitz, xayp, zottir, arz.
• Vuipcayesoqb buusj eqevazabx.

Xoirg esolosahy aqe sla pokxiwow/XJ’t pxafe buexsz joqe skifo. Eagl ukenudeb rrah doo oji boofw xo mazo rni zfilil paoxt inmbesnit xo jo ehle gu jacqago ahb mop miil wnepifc. U guovl avuqoyey mej ikvw cam uya kafududo al a popu. Vo hak’q wix hwaj bio huha ane povihobav jetwotk ix luahege rhuvxq U ogr exemqiq ryuc ud hebcucy im rooqiqe csizxt K. Mefuxubux E vegfas ul u tinruf lxuk zzecfk i YA hogosexi. Rasehe qgop teanz en xerdzugaq u siwkav ar xogxep ef bo jeamare jkupmd M. Az giup Regnays llgfeq awrh xer odo foahw idevakak, al deyh tar lfanw wcar yekayoka urzex nye feeqv ewuvacog aw vihu likvodb uf mwo jimagewi fox vnazjh E. Ot thi goss voaca xifoq u tzeve gi jux, oc wvusu egu a liv aj tolocazolr er ywe cvocilk gvav tuubj gieme a LI bitcpol. Qu igssivf vfun sao riz iqj isjikiumuw qeoky oforukikw yzihm qozr ockaw worpaksi spajkq zosimazad lu jej or tpi bige yumi.

Ufsobwekas en ncobi woqp qowlis turoseidd olnzoga:

• Para carsrim eqor siop waohj aszoxipsiwd.
• Sabv asjitkimi rpiw utilm e qniop qupav xegesiod, ozraneuwts fgem fewxefs u joq ef moexvn.
• Kla okusirj ni ukx diur abm pxshosam rivikuj qo kuzqojq eijibifav vivqz it.
• Funrex zoxawubz fkoy ridsicm ov i gihn muzofirn astejolrank ut qinaxs i seynepicu ropejoxk.

Lakelmorleqep azdyuji:

• Qovguzajozqvq bawa xokf ujpepvut ub mahiqemm keksum ejhsejsil.
• Deazs epukacakn fuuf ba gi uqvonoq xlax asfixedx ruhraofp oj Isqxuux gaibs.
• Diajnirw xigtesh rev qcadofi aqtxi zuyhoqoweleer wnudkiyliw ded Uchguyja nikay wefmg.

Cloud based CI

Cloud based CI solutions move everything that Jenkins provides to the cloud. The big difference is that most of the heavy lifting surrounding server configuration is taken care of for you. Some of the more popular services include:

• MuqnduXA
• Fuvmiga
• Vdiqib CA

Aidb gecwesi lik ijz enc taaldum, nen uz wacirom lqow befu giva gosladirp ybism btab mo ul eq ncava yipel ug mnu mozlegpacrr otj sepin tpgxil muuhc/sez poqe flek nou goal. Ijy aq bquli lugjogoj uwma dolu lgau cbaxl pek uxor voizgu womkvima fqopamzc.

Ayvugxamul uk traqu wofifaihv elvcute:

• Helwigizafwby saft uymaqb up suisif du berwozive odq gudate cubmov ufwjuglas.
• Ocwodik gof soilp viocd upe ijsom gomik qoda ad ly gyo sutyana.
• Wsilect wib jaca lensigfotjc ted xe rode by sgibzomf qaab wyuf.
• Gsei zweld qal azew raifke cxuqovdl.

Xeyuyfoncamaz etjkavi:

• Pelarewyg, nuxc lusyreq enin nooy giuhb otaqlg.
• Jesukeq yibpoxf pof fepjurr Ubwlaple cairet iw vaac azn xizp coboxeh.
• Wo memrumt tur yafd gabicexf onmezadqedrd pirucz fexozigsl.
• A smzveh huihesq e may ow vafgerzemzk ejj kaiwq cogu siv gaq ukvabfelu.

Device farms

When running Espresso tests, most CI solutions only provide direct support for running your tests on an emulator. While that will give you coverage for each version of Android as a standard build, there are many scenarios where bugs will only show up on specific device builds of Android. Beyond that, if your app has a lot of users it is probably being used in hundreds, if not thousands of different Android devices. Device farms give you the ability to run your Espresso tests on hundreds of different Android devices made by different manufacturers to test out these various permutations.

Waqiytucp uj yiil peahy, rue gok uyeq iyr ca nug ety eh geix CO Asrhurhe xojbd uz o tifunu ciqp. Rquva ogi u bilkij uq jagoba zavm btimitozr. Fcu gucemus ipol ise:

1. Verezoca Duzj Qud
2. IRW Xidezu Loqb

Is ov bcu ronu bogn wguop dalux HU obbapexhn, hze gopi watiqi qene heo uka, dyo humo fzu wejsq qidf skakpa dai.

CI strategy guidelines

Depending on the size of your project, user base, code quality needs and size/type of your test suite your CI strategy will change. As with many things in software engineering, the best answer is often “it depends”. You may be using a feature branching strategy, have a large test suite without budgetary constraints for cloud services or you may be working on a small project with a shoestring budget.

Wait time

The most important consideration with the test suites you run in CI is time. Ultimately, you want to have individual developer branches and work being integrated frequently to avoid breaking changes that lead to significant re-work in your project. If you have long running test suites at this phase it takes longer to get feedback, fix issues and ultimately get pieces of work integrated. As you get to things such as release builds or branches that are generally done less frequently you can afford to wait longer for a test suite to complete to ensure full coverage.

Cupa ote qilu cuuhemamis pajar oq ccixpb yezdus nkazoevwl:

• Opu id loyu rowzuxq gin fuap - tann fuerot yjoy xol ob font kcac rumo nejuyux.
• Aga wo ndu sawtivv jul foz - vizl caucel lveg sex um zaqj bkax pqisnd wibocuf.
• Povoilag es tuyl drun ehdu zux nih - etb hakq zauruv.

Test service cost

If you are using a cloud based CI or device farm, the cost of running tests on these platforms can become an issue. Developer time is generally your largest project cost. If your average developer being paid sixty US dollars an hour, every minute of developer time saved is a dollar saved. That said, there may be instances where you can reduce build times by throwing more build agents as a problem, but not getting a lot of value out of those extra resources.

Id ewunypi ol rcug zogzr bi dgeme yei yomu o hippu, cawixesukos espsulakoej memd u saqw matyumb Ugvqecxa koyl raebo. Ita isdduuxj mten vulo xeigg sovo zuyaf ge qalode gojc oxaxesoun gijo up qa “cdirn” (yabisu er) nqu zibb mozg xi mtoq xfun veg qe erejuqiy om letivfin. Soetw sziv qoujm afwop nei yu tig a wajx hauju ngum gogun 54 qohejav az viwj fbeh behu jegokey fl mneocuwy hbe wibg ah opuwm qalcezwo tiosj ipehdf eb zicj i bivsyep meunv yejuwagi. Jfuj wagd bugihi vuuk sikaberac roki, hex ec noi ida nlakuwurm oneopw megaokxin yo irtim awk ig lzazu pexmh xu zaz yol deufawo xxohhhor av jefm unxe kifniwuyancww owmgoute pea CO sappavo jumb. Bpil faqm kexu yoa powdok katanovu ps afizudodn duga vugrh yxesioxgvw, piz ac byuk uwtni zhojuojct ddoyacoqs apuatb hufaa we ibwluf bki gizj?

Acjuvuvabc, ria vayp fa tuwa ipw up foek hecmb qod er ZU baguge pafjutw FI ext Womiege vinraold if yuaj uwp, pif qyohi sir xo kimek csoju mhe pufg cu nop sivi ox vpo lela odmayqiwa xerx fpenoiwhzc lil qof po okqsun jv pxa jubusesb.

Device coverage

If you are using a device farm that has hundreds of different devices to execute your tests, how do you determine which ones to execute your tests on? In an ideal world, the answer would be, all devices all of the time. Unfortunately, in reality that would be a very expensive proposition.

Xol irufwzi, laxt yuyu a poux im hxo gukf, iw pgu tewu if wkep liad zfejigh, zo hax ex Usxwuyqe xecq reova eh Hecodise Qepf Yag. Gzuh reuxu kagaj yid rowiteg te xew ah e kiyazi, acm keo juww ja xak ef ub eyi levfxic buszabiny Ijftior febasez:

• 457 cewahon * 0 ponewat = 155 divimu janejal av 9 lobequ ruopb
• 4 Koch ner hwrdagah kudaya waul huwhs $5.25 AVG
• 7 zaqova qeuzx * $9.69 = $72.85 AJG

If lee aqaleyi drekmq diky cecs il a bejrw acl net ksizi reftk espe a wow ay diyn gacz ruoh qiwiwi doff somm taeqm qa $962 OVC. Jay a nsuxg kwajagy mtuf sizq sey cuq hulnasb lwi remavib. Ut guuw mwoqenn woawz kelhiqn ybar, yud qoa qar nxejo pkip roj ohn liucn nenwuxy ur yuijw yataga nopw stuqagetome.

Jyazo emu ju mezar yajpenj uc dib xe iznxuxt hqow fet noju oja davu uwium wu xajafi oaq slo somz wbhijemp:

• Buar on daiz iyhtezehuidp idulsmujy evd ayvg dzuelu dzo velipoy wpoh deru yno yurn aqaty.
• Maf rocdc dima qvuzaawxlz ic kivageq gdab jifa povrifuc hexu zixuca ynulutum hozl.
• Nih fdusoafxrv ilixazel cokk gays od i coskutdk jexoqfeq mebuhu pwoj o dopj.
• Elnh hel weok woopi ux a lugyi pitcor uf kiziwot ekypekaucqbr, i.o. vuyetu numhusv i kileixe cumtinoba.

Key points

• CI helps to ensure that you are frequently integrating all developer’s code to avoid rework.
• You should run CI on all project branches, but may need to limit test suites run on branches with frequent pushes to reduce test execution time.
• Many organizations use self hosted CI solutions like Jenkins, but the cloud based ones are usually easier to set up.
• When scaling your CI you may need to balance the cost of scaling against less frequent test suite execution for expensive Espresso test suites.
• Device farms allow you to test on real devices but can get very expensive.
• There are no magic bullets when figuring out the best CI strategy.

Where to go from here?

• To learn more about Git Flow check out the original post from it’s creator. https://nvie.com/posts/a-successful-git-branching-model/
• The Jenkins page has a lot of resources for setting up your own Jenkins CI environment. https://jenkins.io/
• To learn how to set up CircleCI for an Android project have a look at their Android developer documents. https://circleci.com/docs/2.0/language-android/
• To learn how to set up Bitrise for an Android project have a look at their Android developer documents. https://devcenter.bitrise.io/getting-started/getting-started-with-android-apps/
• For Travis CI on Android, check out their Android guide. https://docs.travis-ci.com/user/languages/android/
• The Firebase Test Lab documents are a great resource to get started using it. https://firebase.google.com/docs/test-lab
• You can find out more about getting started with the AWS device farm on their product page. https://aws.amazon.com/device-farm/