Putting together the best CI workflow for Swift Apps

Dennis Pilarinos at Playgrounds Conference, 2017


Video transcript:

Thank you so much for having me. I appreciate the opportunity. It was a 16 and a half hour non -stop direct flight from Vancouver to Sydney, so I might be slightly out of phase or a little bit tired, but I'm excited to be here for sure. If you're not familiar with it, BuddyBuild is a continuous integration and deployment service that's built specifically for mobile teams. At BuddyBuild, we actually have a fairly unique perspective on the industry. We get to see how thousands of teams work every single day. My goal today is ultimately to show you the data that we've gathered to help inform decisions as you go and build applications and experiences that your users love. We really want to make sure that as you're thinking about how you build your apps, that you take some of this kind of information into consideration.

First, what I thought we would do was actually completely forget about developing iOS apps, forget about Swift, forget about mobile. I actually just want to talk a little bit about how software is built. How is software just generally built? In my time at Microsoft and Amazon, these are some of the obvious lessons that I learned and I suspect it's pretty obvious to you as well. At the end of the day, the creation of software is really this interplay between two groups of people who create the software, and the people who they're building it for. At some point, what ends up happening is developers like ourselves, designers, product managers, marketing managers, they all get together to create a piece of software. At some point, they want to share it with their end users, and then they ask those users to provide feedback to go and test it.

Sometimes they'll ask testers, they'll actually ask for explicit feedback, so they might say like, "Hey, what do you think of this font, or what do you think of this color palette?" Sometimes you might be participating as part of a test unbeknownst to you. You might not even know. Amazon is a very classic example. The ‘Buy It Now’ button on any of the product detail pages, so if you're looking to buy an item, they'll often test various versions of that button. What color, where should it be located, all those types of things. A/B testing. That's a kind of an implicit piece of feedback or a test that you might be participating day to day as we browse the web in fact.

At some point, you're actually going to get feedback from those users. You'll get a piece of feedback that says, "Hey, that color palette is actually really hard for me to read or to see because I'm red/green color blind." Or, "I really don't like that font. It's too old school." Or you can get a bunch of information that says like, "Hey, the way that we were using the color palette for the ‘Buy It Now’ button, that actually converts a lot better." You get feedback from users. You obviously go and fix those bugs or address those issues. You create the next version of the product. You share it out. You test it again, get more feedback, and you basically have this iteration workflow, does that make sense? People's heads are nodding. Yeah. Pretty straight forward.


Now, in order to build software in that way, you're likely going to end up relying on a kind of core set of infrastructure and a set of tools and systems. Specifically, what those are, are continuous integration solutions, a continuous deployment solution, some means of getting feedback, and then collecting analytics and crash reporting. How many folks here at some point built web apps? Yeah, I'd say about half to three quarters of folks. If you're a web developer you're like, "Yeah. Thanks, Dennis. This is super obvious." This is a well understood and well solved problem if you're a web developer.

However, if you're a mobile developer, it actually turns out to be pretty broken. It can actually be quite complex and cumbersome setting up your CI system. It's often very difficult to deploy builds to users. Getting feedback from them is often very rare and more often actually quite ambiguous. Then, getting crash reports and analytics from people often lacks context. You don't actually know necessarily what might have caused that crash. We found out how painful it was two years ago. BuddyBuild was actually an accident. We came upon this problem because we were building a little side project, and this graphic is actually one of the motivations behind it. The side project was effectively Instagram for bad drivers.

You can imagine, sounds like people already get where this is going, you can imagine where you'd launch an application. There'd be a photo ... sorry, the camera view would kind of pop up. You could take a picture of someone doing something silly, like parking a giant vehicle in two compact spots. You can choose from a list of infraction types, like they did this, or they ran a red light. Things of that nature. You could type in their license plate and hit send. Basically, an opportunity for us to kind of capture and share the frustrations we felt of just being in traffic. That was our little toy app, and in that process we recognized that it was actually really hard to work in that iteration work flow, to actually unlock those cycles.


Now, you know that app was actually pretty straight forward. I would argue it's probably a lot simpler than a lot of the apps that you're probably working on today, but it faces the same challenges that we face even to this day, specifically you end up having to make thousands, and thousands of decisions over the life cycle of an application. Things like, from a process perspective, who here uses GitFlow, as an example? Who uses a Variant of GitFlow? Who doesn't know what GitFlow is? Yeah. Perfect. There's actually some people who don't. Where should you start your source code? Which tools should you be using? Should you be using CocoaPods or Carthage? When do you move between one version of Swift to the other? These are all decisions that you have to make, and if you get them wrong it can actually be quite costly.

As developers, we want to make choices that make us as effective as possible in building these applications. I'm going to be showing you a lot of data in a few minutes, and that data is derived from teams who are currently building using BuddyBuild. It's important that we actually provide context for how we capture this data. If I gave you data without this context, it could be a little bit disingenuous or kind of confusing. I'm going to quickly walk through some of the features of BuddyBuild as a means to demonstrate how we collect his data. One of the things that is really important is that we're able to get applications onto BuddyBuild very, very simply. There is very little effort and energy you have to actually expend in order to get BuddyBuild set up for your project.


Here's an example of what it would look like in the past. I can't remember the last time we did the search, but if you look for how you want to set up Jenkins with TestFlight, and Jenkins is a very popular and originally predominantly in a web CI solution with TestFlight. I think everyone here knows what Test Flight is. The number one Google Search Result at the time was actually a 19 page tutorial. In that tutorial, the author actually says, "Wow. That was quite the journey." This is the person who wrote the blog post, and then if you read the comments you can see stuff at the bottom like, hey it took me ten hours of Google and cryptic error messages to kind of get this set up, or it sucks, or I had to write off the rest of my week.

If you make kind of a bad decision, you end up with an experience like this. You end up spending more time setting up and operating these solutions where you could be spending that time actually building your app. I mentioned this a little bit earlier, one of the ways that BuddyBuild was able to get the data that I'm going to show you is that we automate that entire experience. You can log in with GitHub, Bit Bucket, or Git Lab. You choose the app or the repository that you want to build, and BuddyBuild literally automates every aspect beyond that. There's no scripts. There's no YAML files. If you can clone and build locally, BuddyBuild will do it for you within three button clicks. What happens behind the scenes is we actually provisioned the physical infrastructure that's required, so if you think deeply about it, you actually have build on Macs so we have fleets of Macs across many data centers. We pull up a virtual machine to pull the source code down. We do an analysis to go and figure out what is this project? Is it Swift? Is it Objective-C? Which version of Xcode is it using?

We do all this conferencing by reflecting on and introspecting on the product itself. As part of that process, we're able to accumulate a ton of data around these types of applications. Does that make sense so far? Yeah. Great. Next, once you've got a build up and running, inevitably the thing you're going to want to do is actually go and deploy that build. You're going to want people to actually go and try it and give you that feedback. You want a reliable kind of straight forward process by which that build gets deployed. Here are your kind of standard two options.

TestFlight, which I think we're super familiar with, is fantastic in many ways. There are some problems with it though. You often have to wait for review times as your app needs to be reviewed. Switching between different versions of your app, so if you have a QA team that needs to go between a couple of different versions as your kind of testing features, that's actually not possible with TestFlight. You can only have one active app at any given time. It's also, for non-technical users, it's a bit of a cognitive load, so when we were testing my Instagram for bad drivers, I wanted to send it literally to my mom, and I had to explain to her that she had to install an app, TestFlight, so that she could install my app. She didn't understand why that was the case. That's one option.


The other option is the process of managing provisioning profiles. Marin's talk was incredibly accurate and detailed for how that works. How many people consider themselves experts in code signing and provisioning profiles? Awesome. Cool. There's like three of four. I'd love to chat with you afterwards as would Marin I'd suspect, but really what that process looks like is, as you want to deploy a build, if you're not familiar with it, if you want to deploy a build to someone, you have to get their device UDID. You have to navigate the Apple developer portal, add the device, regenerate the provisioning profile, resign the binary, and then resend it to that user. There's a lot of moving parts there. It's often quite laggy between getting all the data and getting the UDID, and getting the build out to the user.

What you find is a significant drop off in terms of people who will actually go and beta test your app. Is this familiar with folks? Yeah. Perfect. One of the things we wanted to do, and one of the things people love about BuddyBuild, is we automatically manage UDIDs on your behalf. BuddyBuild will actually, when you send a build to someone you just send it to their email address so you get the benefit of the speed, the ease of use and familiarity of email. What happens when they click install, is BuddyBuild will actually go and do all of those manual steps for you. It goes to the Apple developer portal. It adds the device. It regenerates the provisioning profile. It resigns the binary, and then make it available to them, all while they see a little spinner.

One of the benefits is that you could just automatically, you'd basically send it to them just through emails, and BuddyBuild automates that entire flow. Of course, when you're ready you can deploy to the AppStore as well. Now, devices, if you think about it, you've got a build up and running pretty quickly. You've hopefully removed a lot of the friction for how you get that build out to users, but devices really aren't meant for the creation of content. Their form factor doesn't lend itself well to long form feedback. There are a handful of users who will peck away at that device for a while. In general, you don't often get lots of feedback, and when you do get feedback, it's often ambiguous.


Here's a classic example. When I was at Amazon, I was building the Silk web browser for the Kindle devices. At one point, I got an email from Jeff that says, "I don't like this menu." Excuse me. You can imagine, A, it's the CEO of Amazon, and B, I have to ask him which menu he's referring to. If you're building the web browser, he could have browsed Amazon.com and been talking about one of the menus on the actual website, or he could be talking about the browser menu itself. This is the beginning of a long and arguably painful thread. Instead, what I really want to do is, kind of one of the features that BuddyBuild provides, is the ability for him to take a screenshot, for him to select a region of the screen, type the feedback that he wants to say like, I don't like this menu, and I get a bunch of the device meta data to help me debug that issue as well, so I can see things like which version of the app he was running. If it was a performance issue or if he was on WiFi or if he was on LTE or network, which version of the OS, things of that nature.

We can make it really, really easy to get feedback from users. 

Next, screenshots if you think about them are that explicit type of feedback. Crash reports are actually a form of explicit feedback. They're using your app. Something bad happens and you get a crash report. It's really a consequence of using your app. I'm sure crash reporting is very familiar to everyone here. Generally, what you see is the call stack, which shows you all the functions or methods that you were using that led up to the crash. One of the things that BuddyBuild does to help developers is actually show you the specific line of code that caused the crash. Because we build the code, we can actually go completely round trip. When a user crashes, when the app crashes for your user, we can show you what the reason was for that.

Better still, we can actually show you exactly what that user was doing that was leading up to the crash. This is an example of a game called 2048, and you can actually see, this is one of the crash reports. It's a feature that we call instant replay. You can actually see the touch targets of the person using the application leading up to the very second and what they did to create the crash. In this case, hitting the ‘settings’ button crashes the app and you can see them actually do that. Does that make sense? Again, getting feedback from users and making it easy for them to give you feedback is super important.


The last thing that we do to help gather this data is keep up with Xcode. Here's probably the experience you're used to seeing, if you have a decent internet connection you have to pull down this giant file. You have to jeopardize the sanity or cleanliness of your machine or you build system. You have to install it and help that nothing breaks. Instead, what BuddyBuild will do is actually take the last known green version, the last known green build of your application, run it against a version of Xcode, and email you with the results. Every beta that comes out every four to six-ish weeks, you get an email from BuddyBuild that says, "Hey, things are going to be great." Or, "You know what, that Swift API actually went from deprecated to completely obsolete, and your app is effectively broken." That is a feature that we called Xcode Preview.

That's a little bit of the context for how BuddyBuild works and how we're able to gather the data. Now, we're actually going to talk a little bit about the process of gathering the data. It's kind of important to note that we have thousands and thousands of companies that use BuddyBuild and they're anywhere from solo developers in incredible parts of the world, all the way to Fortune 100 companies. The data set that I'm going to present to you is a sampling of 1,000 companies. We rotate this data set, so it's not the same 1,000 companies, and this is just a subset of the companies who do use it, but it gives you hopefully some context by which to think through who are the teams that are using this? How reliable or interesting actually is this data?


First question, this is the interactive part, or more interactive part. Where do people host their source code? What I mean by that is do they host in it the cloud or do they host it on premise? 95% of apps, of that sample that we did, actually hosts their source code in the cloud. Now something a little bit more kind of, I figured actually that would be largely consistent with this audience. How many people here use BitBucket? Cool. I'm going to say like 45-50%. How many people here use GitHub? A little bit more than that. How many people use something else? Wonderful. Here's what the data says. Just over half is GitHub. Just under is BitBucket, and 5% are others. GitLab, basically anything that connects with SSH. This is kind of interesting to give you a sense for like where do you sit on that spectrum?

Now, if you go into and figure out, of the self-hosted providers, who are the most popular? How many people are familiar with GitLab? Yeah. Even with their outage, I felt really bad for those folks that day. 79% of the on premise solutions are GitLab based. 17% are actually using BitBuckets server, which is also affectionately known as Stash. 3% of people are actually using GitHub Enterprise.

Now, the next kind of interesting data set is which platforms that people are using. 73% of people have already moved to Swift 3.0.x. 27% of people are moving from Swift 2.3. The next version of this talk, what I'm going to do is actually animate these graphics so that you can see them over time, like over a month. We pull this data fairly frequently and we'll show you how this thing is trending. Suffice it to say, the adoption of Swift 3.0 is breath-taking. What's the version, the minimum supported iOS version? There are still 1% of apps who require iOS 6 or support iOS 6. I feel really bad for them. 7 is at 6%. iOS 8 is at 33%. iOS 9 is at 48, and iOS 10 is at 12%. I gave this talk a couple of months ago, iOS 10 was at less than 1%. You can see people quickly moving their apps through the platform requirements.


Watch. How many people have an Apple Watch? Wow. That's awesome. Makes sense given this community. How many apps do you have on your Apple Watch that are non-standard? A handful of them? Like the non-built in ones? Yeah.  You end up getting basically 97% of people, or only 3% include a watch app. I'm getting the five minute slide, so I'm going to quickly go through these. Of tvOS apps, of the apps, only .3% of people actually build a tvOS app as well. This one is always fun. Who likes Carthage? Who likes CocoaPods? Yeah. That's right.


CocoaPods, 66% of people. 8% of people use Carthage. 23% of people actually don't use either one, and 2% of people use both, which hurts my head. Which version of CocoaPods do you use? This is incredible to see over time. You can see 1.1 is really kind of the standard, but I suspect in a month 1.2 would easily surpass 1.1.1. How many people check in their pods folder? This is religious at this point. How many people do not check in their pods folder? It's about 50/50. I've seen wars break out for less as it turns out in talking to teams. If you get a larger team, when you have five or more people, that actually changes substantially. Larger teams tend not to check in their pods folder. With merge conflicts and things in CocoaPods it gets kind of tricky. There's over 2,699 CocoaPods that are referenced. These are the popular ones. Fabric, Crashlytics, Alamofire ... obviously you'll get a copy of this deck afterwards, but that's kind of what the breakout looks like pretty quickly here.

Test related pods, who knows what these are even? There you go. It's nice to see. The next question is how many people actually write tests? Any guesses? How many people write tests? That's nice. How many people think it's more than 50%? How many people think it's more than 80%? I get this perspective where I actually have the data and I can actually see you as well. It's actually 61% of apps actually have tests, which is a lot higher than what I would have expected when we started.

Really quickly, native versus hybrid. By hybrid, I mean things that are not Objective-C or Swift, so 90% of apps are Objective-C or Swift. 10% of them are hybrid, within hybrid what you see is 68% being React Native. How many people have tried react-native? Yeah. 26 Ionic and 6% are Cordova phone gap, things of that nature. This React Native growth is unbelievable on a month by month basis. It's something that's quite remarkable. Again, something that hurts my head, a third of react-native apps use CocoaPods, which I'm like, "Interesting."

The next question, in dealing with thousands and thousands of teams, inevitably people say like, "Oh, I have a very specific workflow. My team works very differently to everyone else's." BuddyBuild is actually quite extensible, but 86% of the time people don't bother to extend it. A lot of the features it provides works for them there are definitely some teams who have very specific requirements and they can go and customize that as they see fit. What do they do in those things? NPM, Fastlane. Curl, Pod. Things of that nature. What type of Apple accounts do developers use? This is kind of how it breaks down. You can think of enterprise as one breakout of it. Then, individual or company, which comprises a large portion of it subsequently.


I'm at 25 minutes and 35 seconds and that's the data. Hopefully it helps inform some of the decisions that you make as you start building apps. If you're looking for any specific data, or if you want to hear some horror stories of building a distributed build system with high availability rough time, I'm happy to pull that data for you or share some of the stories that we didn't get a chance to go through today, but hopefully that was helpful. Thank you.


Links:

If you enjoyed this talk, you can find more info: