Enterprise and Scale
Panel discussion: Using Swift in large projects and how to manage complexity.
Moderator: Chris Eidhof
Advantages and disadvantages - using Swift for larger projects.
Chris E: We've talked a little bit about the advantages and about the disadvantages of using Swift, so what would you recommend to people, how would they make the decision to use Swift at a larger company and for larger projects? What kind of things would you…?
Kamilah: I feel like I've already talked about this. Someone else can start.
Tuomas: Sure, I can give it a try. I think Kamilah has actually yes, done a decent job at saying what you should be looking out for. To me personally it's really about ABI compatibility. Without that you sort of live in an uncertain future, you don't know what Apple throws at you, and you don't know through how horrible migrations you have to go through when, you know, Swift 4 comes around or Swift 5 if you don't get it by then. On the upsides you have to think about how long is your product going to be out there. Are you starting something new now, which will be running for the next four years? Are you at the scale where you can handle the migrations?
I think that the worst scale to be at is at mid range, where you have enough code but not enough engineers. Small companies should be fine, and large companies should be fine, as well. It's hard, but you've got a small team doing it and you can switch after Swift 3, you switch the teams, Swift 4 we'll do the migration after that. Yeah, those are my ideas on that.
Kamilah: Yeah, I think when making that decision, a lot of the stuff that you're going to be weighing are like, what are those pros of switching to Swift, right? It is a safer language. We are getting less developer error then we were in Objective-C. We do feel more confident in the code that we're pushing to production than we did then. There are like real pros to switching to Swift, but I think basically for your own app, for your own company, you're going to have to weigh the pros and cons. This is what you'll get with Swift, but it's going to take some more time.
Chris B: I guess I have the advantage, in that it wasn't a problem we had, right? We didn't have a legacy codebase to work with. It was all greenfield development for us, so Swift was the only choice, and it was also the only option on Linux, which is what we needed. For us it was a simple proposition. From a number of people who I've talked to, who are on Objective-C, and they've wanted to know how to move over, there's been a lot of talk about staged migrations. Some of the advantages of the interop starting off by introducing Swift for tests rather than for actual application code and working through a slow stage migration. That seems to have been successful for a number of people.
Kamilah: I think I know a number of startups where like just new features that they're working on is in Swift, so they're not necessarily re-writing their old code, but they're trying to move forward with adding new code in Swift.
How to manage complexity when a project gets large and complicated.
Chris E: Maybe a tougher question is, do you have any tips for the audience on how to manage complexity when things get large and complicated? Like, what would you do or what did you find that worked really well, or what didn't work well?
Tuomas: Sure, I don't want to tell everything about what I'm going to talk about tomorrow, but… So we did a full re-write of the rider application with about half a million lines of code, Swift code, and we re-architected everything from the ground up. What we wanted to do, and the main reason that we went with the full re-write was that the previous architecture that we had put in place four years ago, didn't really cut it anymore. It sort of popped down under the pressure of all the engineers working on it. We basically put in an architecture in place where we put the business logic first, instead of rooting everything around the view tree, so we had interactors that managed the business logic that drove the state of the application. Swift helped us quite a bit in that, because with that tree we also put in dependency injection and the compiler would basically tell us whether the application worked or not, and that was a huge driver in being able to re-write in such a quick time that we did.
Kamilah: I think a couple of things that have helped us manage the complexity, that we weren't doing before in Objective-C, one is an increased reliance on moving things into libraries and that has in itself sort of ensured that we're not necessarily having like the networking layer bleeding into other parts of the app for example, modularising the codebase that also meant that the code itself was clean, and that we had to think very carefully about the architecture of the code itself. Then I know, for instance, in the learning app, we try to break things a lot into a protocol extension pattern and so, keeping our view controllers smaller and moving delegates into separate files. Those are the different things that we've done to try to cut down on the complexity.
Chris B: Yeah, for us it's all about modularity and breaking things down into individual packages. Once you've done that, you can individually unit test them and that really breaks your problem down to a number of small pieces and then it's just a matter of doing higher levels of integration testing as you go, and the CI can deal with a lot of that for you. Even like really, really, large complicated codebases, as long as you can break them down to self-contain units. Then, actually it's not as complicated as you might expect.
How do you manage code coverage? (Audience question)
Chris E: So the question is about code coverage. If you have all your tests in place, then you might want to get code coverage. Do you worry about that, is it a thing? And what do you do?
Kamilah: I can grab this. I mean, I think it's almost impossible to have a hundred percent code coverage, especially if you want to move with any kind of speed, but we do sort of aim towards like 80%. That's something that our test engineers are constantly monitoring, like what our code coverage is, and how to make sure that we're not slipping. One thing that we try to do is, if we're pushing something to production or... Okay, if we're testing it and we say like, "Oh, were you thinking about this feature?" We're not sure. We might roll it out to say like 5% or 10% of our users, but if we actually want to move it to a larger percentage, at that point we need to make sure that we're writing more tests. In that way we're not wasting too many cycles testing code that's going to be thrown out or features that are going to be thrown out. When we are writing them it's because this is something that we're going with.
Chris B: For us, yes. I mean, the fact that we're trying to build a server-side framework, that we're expecting to run to the point that a single instance is going to handle 20,000 requests a second. We don't expect to scale up in clusters. We care very much from the level of doing linting and code coverage at the bottom end, but right the way up to doing seven day stress tests where we throw-loaded it 24/ 7 for seven days, checking for memory leaks, etc. So the fact that we have what we hope is highly scalable code that is going to be reused by many, many different deployments, yeah, I mean, code quality and the way we do that is just crucial for us.
Tuomas: There's basically three ways that you can make sure that the application is reliable.
The first one is just to not care. Throw it out the users and have them report the bugs, and then quickly fix them. That's valid for a small application that is not critical.
The second one is hire a QA team, that sort of takes every single build and tests it out to every single scenario. That becomes very cumbersome and makes your developers very lazy 'cause they can just write bad code and wait a week and they'll get the reports back, and then they'll fix it.
The third one, that we use, is not to have QA engineering at all and have the engineers themselves be responsible for their code. Basically say that you have four lines of reliability from that code. What that leads to is, first people will do sanity checks because they don't have any tests in place, and then they'll start finding ways of automating all of that. And they'll start writing unit tests, and they'll start writing UI tests.
We essentially came down to, I think we have 40-45% coverage in our main app, and that is enough for us on the unit testing side, 'cause there's a lot in Swift that you don’t need to unit test. For example, all the builders, all the contact between the APIs, like they're enforced by the compiler, and the compiler will tell you if you do something wrong. What you need to test is your business logic, and pretty much every team, almost 200% tests, all the scenarios that they might make mistakes, and that gives us a very good confidence in that everything works. On top of that we have a much smaller set of UI tests, that sort of run the safety checks for each build that we put out.
Sharing knowledge in large teams and between teams.
Chris E: So maybe another question is when you have a large team, I imagine that the problem is even bigger if you want to share knowledge. If you want to make sure that everybody is using the same kind of architectural things, like I don't know, like Reactive Cocoa. You might want to use it with your entire company or you might not want to use it. How do you deal with that? Do you leave this to the individual teams or do you actually try to share the knowledge so that all the code is very similar and people can move between teams?
Chris B: Yeah, I don’t know if the way we work is kind of unique, but we have development spread over seven sites in five countries, so sharing knowledge is crucial. Having completely open codebases and spending a lot of time working in open-source helps with this sort of stuff. It kind of forces you into sharing not just like raising a pull request, but explaining why you're doing it and these sorts of things. We also put a lot of effort into design reviews because for us that's the point in which you start to capture that there's a lack of knowledge in the way that we should do things. Design reviews for us often highlight, "Here's where we should actually be writing or doing some education on why you should follow certain design patterns or certain approaches."
For us anything that is a sizable work item that's going to take someone a week or more worth of time, we do a very, very lightweight code review for it before it goes into a more design stage.That's when we usually capture the fact that there are things that we might want to be spending some time talking about and sharing ideas around.
Kamilah: Yeah, so we have a couple of things that we do. Similarly, we have multiple offices and most of our iOS development is happening either in Sunnyvale, San Francisco or New York. So we also have a distributed team that we have to coordinate around. At LinkedIn, we have Slack, and that has definitely helped a lot for communication. We have a dedicated iOS channel, Android channel, and then there's also like individual teams, for instance like there would be like a Voyage or iOS channel, right? So there's lots of communication, both horizontally, across like all iOS developers, but even within your own team with iOS developers.
That's definitely been one thing that's helped a lot with communication. Then also around the communication aspect we have internal iOS meetups, and anyone can submit things to discuss. If there's something that's going to affect multiple developers, people will try and bring it up there. You'll get an email, so you can see if something is going to come up that you think is maybe relevant or interesting.
Then, around the design thing, for instance we have like a mobile infrastructure working group. So there, for instance, if you're adding a new app or a new library, something that you're proposing, then you’re actually going to present it there. And people in there they are in the mobile horizontal infra team or they've been doing mobile for a while at LinkedIn, and in that way we're getting feedback. Nobody's is adding something to the system in isolation. That’s definitely ... It's very collaborative.
Tuomas: We have a similar thing called RFCs, which you send out for every single functionality that you're going to add, and it's open to the entire company. Everybody will receive those RFCs for mobile, and everybody can chime in and jump in. That's sort of the last line of defense before you start engineering something to catch all the problems with possibly the architecture.
On top of that we did the unthinkable and we wrote documentation for our architecture. We spent a lot of time. One of the big goals of the re-architecture was that we can give this document to anybody joining Uber and they'll be ramping up within a matter of days. It ended up being a hundred pages long, so it's huge, but it's definitely worth it and if anybody comes to me and asks a question, the first question I ask back is, "Have you read the documentation?" If not, I'll just send them back and have them read through it.
Chris E: Does it take a lot of effort to keep it up to date?
Tuomas: It did. It was changing quite a bit at some given point in time, but now it's pretty settled.
Kamilah: Yeah. Documentation is definitely useful. We have docs for both iOS, Android and web. Everyone keeps it separately, super useful.
Do you have policies on code review? (Audience question)
Chris E: So the question is, do you have any thoughts or policies on code review?
Tuomas: Yeah. I can take this. So, yeah, we do. We have policies around code reviews. Every single piece of code that gets checked in has to be reviewed. We don't say by how many people, I'm going to leaving it up to the engineers to decide where they want to sort of inject themselves and when they want to be blocking the reviewers. We use Phabricator and it has a system called Herald, which lets every engineer track certain files or certain constructs or certain types of code and add themselves automatically as blocking reviewers to a diff. That is highly useful if you want to make sure that people use dispatch queues correctly, you just add yourself as a blocking reviewer, and that means that you physically can't land that code until the author says yes.
Other than that, we try to enforce the policy of 24 hours for code reviews, so if you put out your code, it should be reviewed by all parties within 24 hours. Sometimes I know that's not possible, but we try to abide to that.
Kamilah: On the main app there are a couple of different people that you'll have to get ship-its from. For each module there is usually up to five owners. So for instance, for profile, two to five owners. So if you're making a change in the profile part of the code in the app, you're definitely going to need to get a ship-it from them. So you can see where it can escalate depending on how horizontal your changes are, you might have to get a ship-it from multiple people, depending on which part of the app you're touching.
In addition to that we also have a platform ship-it system, that we've been working on and sort of testing out this year. We have it on iOS and we also have it on Android and on API for mobile. What that is, is that these are people who, maybe you nominate yourself or someone nominates you. You say, "Yes, I feel like I have deep knowledge about this platform. I've been working on this for a while. I understand what our style guidelines are. I understand how to look for things." And you'll actually submit code reviews that you've done to people who are ready on that platform list, and they'll go through and like say, "Yes, you've been doing a good job of being a code reviewer," and then you can get added to that list. For any code that you want to get checked in, you also have to get at least one ship-it from someone on that list.
Chris B: Our policies are fairly simple. Any code change you want to make, there needs to be a pull request and you don't merge your own pull request. So there has to be a review from someone that presses the “merge it” button.
Obviously, there's always exceptions to that. If something absolutely needs to get done over the weekend, then there's no one else available, but the general rule is raise a PR, get someone else to merge it. If you're honest, then you're not just poking the guy next to you and saying, "Please merge it now." You're expecting them to do the right job. As long as there's a bit of honesty and a bit of trust, then it works.
Tuomas: Maybe one more thing to add to that is, you want to make the reviewers drop as easy as possible. You've got to put linting rules in place to go over the details that a computer can pick up for you, like style checking and formatting your code should be automatic, and nobody should ever have to say that, "Please remove this empty line" or "add some indentation here." And the same goes for any rule that can be automatically caught. On iOS we're a bit behind, we use SwiftLint, which is decent, but it's not perfect. It doesn't let us create very complicated custom rules, and we want to create a tool that let's us do that. On Android, we already picked those up, so we can create pretty interesting rules like; if you subclass this certain class you can't use this method or function, or you have to do this and that.
What do you think about Swift on Android?
Chris E: Maybe one other question, regarding Android. Now that we have Swift on iOS and Mac and also Swift on the server, I can imagine that maybe in the future we'll have Swift on more platforms. Even now, like how Android and iOS play together might be interesting. So how do you feel about that? Do you see Swift running on other devices, platforms? Or how do you feel about Swift on Android, maybe? Do you see anything there or…? Is it too weird a question?
Tuomas: Yeah, I mean, Swift on Android, that's a certain type of hell freezing over again. I don't think Google will get behind it. They have their own ecosystem. They want to make sure that they own at least a portion of the APS and the application code. They wouldn't probably like that Apple controls the language with which you write Android applications. Sure, probably a year from now you will be able to write Android applications with Swift, but will people do that? Yeah, a small percentage will, but again, once you get to a certain scale, you sort of feel scared about jumping into something that you don't have a backing of the major manufacturer from.
Chris B: Yeah, I mean, can we get Swift running on Android? Well, it's software and you can bend software to your will. So technically making it work is probably not the hardest part. It's without the various business interests aligning then you'll be able to get it to work, but it will get broken very shortly afterwards.
I think there's a huge amount of potential there for it to happen. I think what needs to be better understood is whether there’s the real desire from the big vendors and the business interest to make it happen.
Kamilah: I think that's pretty similar. I mean, yeah. It seems possible, but I don't know how likely it would be that you would actually get that backing.
Tuomas: We do have Kotlin, which is basically Swift, and has a good backing.
Chris E: Yeah, I heard some success stories about big companies using the same architecture on both Android and iOS. Now it's possible because there are very similar languages.
When should a team start implementing processes? Should it be based on team size? Codebase size? (Audience question)
Alexis: I've had the luxury lately of working on a relatively small team, and in small codebases, but I hear these stories about the process that you're going to have to use in larger organizations. I can see why it's necessary but also kind of frightening. I was just wondering what rules you have in mind for when and why these various kinds of processes become necessary? Is it driven by increase in team size? Is it driven by increase in codebase size? Is it driven by increase in the mission criticality of the piece of code? And if you're not dealing with hundreds of developers and hundred of thousands of lines of codes, if you're small, when do you feel you're crossing that threshold where you start needing to have two people turning the key simultaneously before you merge a pull request and that kind of thing?
Tuomas: I can take this. Somebody at some point said that if we look at all the processes that a company has, that's a list of all the failures that the company has had. Whenever somebody messes up, that's when a new process is born. So the more processes, the more mess-ups you've had. That being said, that's actually pretty true. Process stems out of the need to fix something, fix a current problem that you see as a risk in the future as well. If you can find other ways that don't involve processes, those are always better. Nobody wants to have a list of a hundred different processes in mind. If you can automate that, even better. If you can lint out the user job of something that you don't want anybody to use, then it's an automated processes and you can sort of educate people at the same time, and the linter can point you to some documentation saying, "Here's why we don't allow this. Please read up on it," and that's the best kind of process. But yeah, you sort of want to keep at a minimum 'cause that usually involves bureaucracy, as well.
Kamilah: Yeah, on the learning app, for example, we only have five iOS developers. We just have the one ACL, you only need to get one ship-it, there's not the same sort of process as there is on the main app. But at the beginning of the year there were like 50 people who could approve your code check in and you could just check in anywhere and break something. It could be like somebody else from some other team approves someone to change something in your part of the app, and then someone's like, "How did this get in?" Right? And we had a series of things around that and there was also a lot ... This is actually not just for the iOS app, this is actually a company-wide initiative that they did to cut down the number of people who were allowed to approve any piece of code. The basic rule of thumb was; “If there is more than five people, this is not okay. You need to figure out how to break it down, find five owners. Break apart your project. Whatever you need to do to increase the feeling of ownership.” This is so that if you're approving a change to the code, it’s because you feel some sense of ownership over that section of the code. And so as Tuomas was saying, it definitely came out of necessity, for sure.
Chris B: Processes are there to make sure you do the right thing. If you're already doing the right thing, you shouldn't need the processes. If you're a small team of five developers, and everybody's already understanding what the design is, and informal buddy checking of code is happening, then you shouldn't need these processes. Yeah, and when you start to scale the team, you start to realize that not everybody does understand what's going on. Not everybody is aware of the design, so we do need to start documenting these things. And sometimes it is reactive, when you don't realize there's a problem until something goes wrong, but generally you shouldn’t be doing process for the sake of the process being there. It's about doing the right thing and if everyone's doing the right thing, no process.
Having said that, when do you need two people turning the keys at the same time? Well, that's when you're launching nuclear devices. It also depends what you're creating. If what you're doing is mission critical, then probably you should have some processes around that anyway, but in general, it's the process is just there to make sure that you're doing the right thing.
How do you ensure that your team has a unified coding style? (Audience question)
Chris E: Just to quickly repeat so the question is: now in Swift there seems to be a paradigm shift towards more functional immutable code, and how do you deal with that within your company and make sure that everybody does that or doesn’t?
Kamilah: Yeah, so the meetup’s that I mentioned before, I think that was a really huge help. A lot of people who are really passionate about Swift and the new style of coding, they put a lot of work into doing different tutorials and presentations and spreading the knowledge of this different way of coding. The Swift style guide that we have, and that's actually open-sourced you can look at it on our GitHub, that helps a lot, because again, you have like a reference. Before we had that, we had one internally and then that got more formalized, but it was a really good resource, like "Oh, this is not a thing that I should do. Okay, I see this is the correct way to code this."
You definitely had some of that. I know there were parts of the app were, especially like you’re in launch and people are coding, you're not going to be looking over single line of the app, so you definitely would see sections of it, where you were like, "Oh, these were Java programmers that came over to Swift." Right. Ported over their Java programming style onto this. And then again, you try to do more knowledge sharing and getting people to really starting to think about coding in a different way.
Tuomas: First of all, I'd like to say that, I don't think that there's ... well, there might be a paradigm shift, but it's not because it's now possible. I think that with Objective-C you could do all of that. You could have immutability, you could have functional programming. With Swift, it just has become much easier and nicer to do so. You would have to be some sort of mad man to do all of that in Objective-C 'cause the code would be pretty convoluted. If you have an application where you utilize some of these things in the Objective-C side then it's easy to translate it over to Swift, because it becomes very natural to simplify things.
Well, what we did is, we had the opportunity to sort of start from scratch. Come up with a new architecture and we just made sure that anything that went into the codebase was good, as close to perfect as possible. Engineers, if they don't know, if they come to a new codebase, they just look at code. They look at how things have been done and if things have been done in a nice way, they'll just copy that pattern and they'll implement their functionality in the same way. If things have been implemented badly, they'll just copy that pattern and implement bad code, so it's imperative that if you have a new codebase, that it's as close to perfect as possible before you add people on.
Chris B: I'm not sure I got a good answer to it. It does remind me, so I worked on Java and JVM’s for about 15 years and I recently looked at some code that clearly it was someone who'd been working on the JVM, which is written in C, and then they’d started working on some of the Java code- which is object-oriented, and clearly was a C programmer. And then I realized it was me.
The problem is, it's a new style, and yeah, if there isn't examples in the codebase to work of because people generally follow the style of what's around them. If you're a new team learning a new approach all at the same time, it's hard and it's a learning process. But things like having “lunch and learns”, where people share what they've learned and how that’s gone well, all help. But yeah, it's difficult when you move from one style to another, particularly if it's a team doing it all at the same time, to get it right the first time. I think you just have to accept that you're not going to, and that you should go back and revisit what you've done and see if you can do it better.
Chris E: I want to keep asking everybody more and more questions, and I think we could sit here for hours, let's move the conversation to the party. Thank you to the panelists. Let's give them a big hand.