Contributing to the Swift compiler
Ayaka Nonaka at Swift Summit San Francisco, 2016
Ayaka: Alright, hi everyone. My name is Ayaka, and I am an iOS Engineer at Workflow, based here in San Francisco. Workflow has a lot of Objective-C, even though this is a Swift conference. Because we have so much Objective-C, we sometimes run into Swift problems.
The other day, or I guess several months ago now, I was trying to use this thing called DCColor. I was like, "Has anyone actually tried to use this DCColor type from Swift?" Well it seems like Swift doesn't see it at all, so that's a little bit weird. We were talking about it and then ... Oh wait, before we go forward, DCColor, if you're wondering, is defined like this; @compatibility_alias DCColor, UIColor if it's mobile, and if it’s Mac OS, it is NSColor, so maybe we have hopes and aspirations for having a Mac app one day.
Anyway, so I was thinking; "Maybe I should post this to swift-evolution”, but honestly I've been lurking on swift-evolution, and it was a little bit intimidating. I have no idea how many people are on it. I've seen some pretty heated discussions that I wouldn't want to be a part of, so I was very unsure, but I guess after eight minutes of thinking I was like, "Okay, I'm just going to do it. I'm going to write to swift-evolution and see what happens."
I wrote up this email. I just described what was happening. I probably proof-read it like 20 times because I was so scared that it was going to so many people to read, and I sent it, and I held my breath, and then slowly but surely I got a bunch of thumbs up, some plus ones, like, "I agree," and eventually Doug Gregor from the Swift team responded, "It definitely makes sense for @compatability_alias to map to typealias and I'd consider it a bug fix that doesn't need a proposal” Awesome! “Thanks for bringing this up. I had no idea anyone knew about or used @compatibility_alias…”
I'm just curious, how many of you have actually used @compatibility_alias before? One hand. Maybe one hand. I hadn't heard about this either, and I Googled it, and I saw an article on NSHipster about it, so guess it's a hipster thing. Anyway, after this I was like, "My job here is done”. I filed a bug report like a good citizen of the Swift community, and honestly I think one of the best ways to contribute to Swift is to just file bug reports and attach a playground to reproduce it or whatever.
I was like, "Okay, I'm done here. Awesome." But then I checked ... I got a message on Twitter, and I checked it, and it was from one of the dogs, poodles who works on the Swift team- Michael, and he goes: "You should have stolen the bug yourself as a starter bug. If you act quick, maybe you still can." At this point I was just like, "Ha, that's funny. I'm just an app developer. I don't do compiler things. There's no way I'm going to do this." Then he was like, "Seriously, do you want to fix it?"
I respond, "How “starter” is this?" That's how my journey into contributing to Swift started. Very, very reluctantly and a bit of a spoiler, I actually did end up somehow contributing to Swift, and when I told everyone, they were like, "Yeah, awesome. Congratulations." A lot of people were asking, "How did you actually manage to do this? I tried to build Swift. I couldn't even get there." Even if you were going to build it, maybe you couldn't figure out how to write tests for or how to use the debugger for it. That's what we're going to go over today.
All right, so step one: You have to make sure you don't have to write a proposal for whatever you're doing. How do you do that? For me, I just check the archives for swift-evolution, made sure no one was already talking about @compatibility_alias, then I emailed swift-evolution, like I showed you. I get a thumbs up from a Swift team member, Doug, in my case. Then get an SR assigned to you.
What is an SR? It's one of these things that you go to bugs.swift.org, you can see all the things that the Swift team is working on, and here is the bug that I got assigned to. Yeah. At this point it's super official. Now what? Well, in my head, I just have this list of fears.
First of all, new codebases in general are terrifying. Second of all, a compiler is like ... I had an opportunity to take a compilers class in college, but I decided not to because I was taking a machine learning class then. I was like, "No way that's going to happen." Then, last but not least, it's written in C++ and honestly, I've never really used C++ before, but I'm still scared of it. I was afraid of it without actually having ever used it.
All these things going through my head. I'm just like, "Oh my gosh, I have no idea what I'm doing right now. What did I get myself into?" When's the last time that you felt this way but you had no idea what you were doing? Maybe it was when you were in high school and learning physics or maybe it was when you were first starting iOS development or Mac development or Watch development, or maybe it was yesterday or this morning.
For me, I think one of the times I really felt like I had no idea what I was doing was in high school physics class, and it was like electricity and magnetism, and I don't remember that much from that class, but one thing that I remember is this catch phrase that my physics teacher used to use, and it's to “Proceed fearlessly”. So every time we'd have to solve a crazy differential equation to do whatever, he'd be like, "Proceed fearlessly, you'll be fine." This was just going through my head. I was like, "Okay, I'll just keep going, see what happens. What's the worst that can happen?"
It's easy to say, "Just proceed fearlessly," but maybe you can proceed fearlessly while being guided a little bit by a mentor. In my case, my mentor was Robert, who interned on the Swift team this past summer, and Michael was nice enough for him to introduce me to him. That was really awesome, and you're probably wondering, "How do I get a Swift intern to help me with my problems?" I guess at the very least I'm hoping that if you can't find a mentor, I can try to be your mentor, and I really appreciate the time that Robert has put into teaching me all the things, and I'm hoping that maybe if I can teach at least one or two people in this audience here today, that would be awesome. Then if those people can teach more people, that's even more awesome.
All right, so step one, now the real stuff. You want to clone the repo, it's just at github.com/apple/swift and you want to make sure to make a swift-source directory before you feed into it and clone the repo because when you run update-checkout, it's going to pull down a bunch of stuff at the same level as the directory, so if you don't make that extra directory it's just going to litter your other directory, so make sure you do that.
Then, last but not least, Robert says you should “Go read a book” because that takes a while. I was talking to someone and apparently on MacBook Air’s it might not even finish building, so that's a warning right there.
Anyway, so number two, you want to get CMake and Ninja. What are these things? CMake is the thing that manages the build configurations, and Ninja is what actually does the building. So how do we get these? If you go to the ReadMe you'll see all this stuff; Cmake, you can get a binary distribution, Ninja. It says you can clone it and you can build it from source. I was reading this, I'm like, "Oh, I don't really want to build stuff from source." If you look down there, though, it says, "You can install it via third party packaging for Mac OS." I'm using Mac OS, I can do that.
I recommend using Homebrew. All you need to do is: brew install cmake ninja. No need to build anything from source. If you want to do stuff from source, you do you. That's awesome.
Anyway, so now for the fun part: building the project. You can build the project by running build-script -x and then what does this do? According to Robert; “That will build everything and dump out an Xcode project in the build directory”, and then he says, "I hope you picked a good book…”
Yes, I did pick a good book, and this is what I was reading. I'm a big fan of Murakami. I don't know if there are any other Murakami fans in here? Yeah. Totally recommend getting a book for the Swift process. Anyway, eventually once you're done reading a few chapters of the book, it will dump out an Xcode project in this directory: build/Xcode-DebugAssert/swift-macosx-x86_64/Swift.xcodeproj. Don't worry, I'll post these slides later, so no need to remember any of this.
If you open that it looks something like that. It's like a friendly old Xcode project with a lot of files.
Okay, so now it's time to actually implement the thing. It's like, I’m internally screaming at this point, and wow, I really have no idea what's going on right now. I wish I can back out of it, but if you think about it, there's a lot of great existing code already in the Swift project - Swift is a pretty big project. Second of all, what's the worst that can happen? Me and one other person in the audience who uses @compatibility_alias won't be able to use it in Swift? It's fine.
As a reminder, we want to get to a point where we can import @compatibility_alias as typealias in Swift. Maybe one thing that we can ask is, "Where are similar things implemented” or “Where are the other Objective-C things that are pulled into Swift?" The answer to that, is ImportDecl.cpp. If you look in there you'll find it's already filled in now, but when I first saw it, it said, VisitObjCCompatibleAliasDecl, and then it returns a nullptr, so of course it doesn't work, right?
Then maybe the second question you can ask is, "What behaves similarly to @compatibility_alias”, or “What also gets imported as typealias in Swift?" Anyone know? Yeah, typedef. Exactly, so if you use typedef you can define something like; typedef NSString StringCheese; and instead of NSString, you can use StringCheese everywhere.
The rest of it, I won't get into too much of the details because it's very specific to the actual implementation we want to have, but it mostly consisted of ... I knew what the input type to the function was and what the output type was, and then I knew that it was supposed to be similar to typedef, so I copied it over, deleted a few things. I was like, "Oh, how do I make this type? Oh, I can use this constructor," and then when I got really stuck I was like, "Hey Robert, how do I get this thing out of these three things that I have?"
Eventually it built. It's like, "It builds." It's pretty awesome. Do we ship it now?
Some audience members: Yes, yeah.
Ayaka: Yeah? Ship it? Any naysayers?
Other audience members: No.
Ayaka: No? No, why shouldn't we ship it?
Audience: Test it.
Ayaka: Test it, exactly. We should not ship it because we need to write tests for it. How do we write tests? All right, so what we're going to do is we're going to write some Objective-C code, because we're testing, Obj-C-Swift interop, and we're going to import it from Swift, and then we are going to make some assertions based on what we expect. Pretty similar to what you would normally do for testing.
All right, so let's write some Objective-C code. I'm just going to import Foundation. I'm just going to write one line, that's @compatibility_alias StringCheese NSString;. Easy, and then I'm going to put it in this directory; custom-modules under IDE inputs because that's where it seems like all the other tests are. Then we are going to import it from Swift, so I want to be able to import it as a module import CompatibilityAlias, but you can't just write that and expect it to work. We never turned it into a module, so how do we turn something into a module?
Well, if you go back to this directory, there is a little ... Oop. Oh no, I turned it off. How do I do the laser pointer? This it? Oh, there you go. Yeah, cool. There's a module map right there. If you go in there you'll notice that all these headers are exported as module maps, so I just looked in here. It seems like everything is just saying header, and then whatever your Objective-C filename is and then export *, so I just went and created my own module CompatibilityAlias and copied whatever was there.
Again, there's a lot of example code in the Swift project already, so I just looked at what everyone else was doing and copied it. I just went down here and pasted it right down, right there. Cool. That part, we've imported it, so now we can just import some silly function like- take some cheese and return some cheese. It doesn't need to be complicated, it's just a test. All right, so that's done.
Now we need to make some assertions based on what we expect. What are we going to use for this? We are going to do something called lit testing and lit testing is what the primary tool chain for the Swift compiler uses for testing. A lit test looks a little bit like this. I know, it's a little bit scary looking, because there's some stuff up here in comments, and then there's the code that we wrote down here.
Let's break this down because it's a little bit intimidating. Start with the easy part. We're already familiar with our import and our func, and then up here you see, //Print: typealias StringCheese = NSString. What that line is saying is, "I expect this line to exist from this file," so if I import CompatibilityAlias I expect my @compability_alias line to get turned into this typealias StringCheese NSString line. I'm just saying I expect this thing to exist.
What about the rest of this crazy mess? Don't worry. I personally still don't understand all of what this is doing, but how did I solve it? Well, “What also links with an Objective-C module?” Well there's a thing called swift_newtype, which also links with obj-c module, so I just went there, saw this thing up here, copied it to my file and then changed everywhere where it said, NewType, to CompatibilityAlias. Again, I'm just doing what was already there and learning from that.
I still don't know what all of this is, but it's fine, when you're first learning you don't have to understand everything to be able to do something. Do you remember when you were ... I still remember learning about table views and every time I had to implement a Delegate and a Datasource, I would never remember any of them, so I would literally Stack Overflow it and copied and pasted it into my project. I'm like ... it was fine. Now I know how to write everything on my own, but when you're learning, totally fine.
All right, now that we've written the test, how do we run it? I went to the ReadMe and I found the little testing section. Went here, then I scrolled down, then I started reading, and I saw, "For day-to-day work on the Swift compiler using utils/build-script -- test should be sufficient." I was like, “Great- I'm just going to run that."
Then I was just sitting there for a while and my computer started getting a little bit heated up, and I was like, "This is a little bit weird. Maybe I should ping Robert?”. So I did: “Hey, is this thing supposed to take a really long time?" Then he goes, "Yeah, don't run that. It'll do a rebuild." He says, "Instead if you want to rebuild the thing and execute the compiler for something, I recommend running that cd into that directory and then run Ninja Swift. That will just rebuild the compiler and you can just execute lit manually”.
When you think that something is wrong, I just couldn't imagine anyone working on the Swift compiler to just be sitting there for hours on end, just waiting for one simple test to run, so I'm really glad that I asked. Then he says, "You can run lit and use this filter to run your test," so you don't run all the tests that are there. In our case we are going to run lit with the filter compatibility_alias.swift. Cool. Then I ran it, and the test failed, of course. I'm glad I wrote a test for it.
What do you do when tests fail?
Inaudible comment from audience member.
Ayaka: Comment it out? Just delete your code or something. How about debugging? For debugging we are going to use our familiar LLDB. How does it work? When you run this and the tests fail, it outputs a lot of stuff up here. As you can see it says this test is failing, and then it dumps out everything that it ran for the test.
Then up here there's this invocation to the Swift compiler with a bunch of different arguments. What you do is you take the first part and feed it into LLDB, and then once you're in there you type r and then paste the rest of the parameters there. Apparently this is what I learned about how to use the debugger, so if anyone knows a better way to do it, totally let me know or write a blog post about it, because that would be awesome.
I heard that some people on the Swift team use Xcode for everything, whereas some people do half and half, do the typing on Xcode and do the debugging in the command line. This is the command line method, as I know it. This is a little bit confusing so let me just show it to you in action. Can everyone read that? More or less? Cool. All right, so first I'm going to run the test. It failed. Then I'm going to take this part… LLDB and then I'm going to take this. I'm going to run LLDB and then in here I'm going to type r and then paste this stuff. Cool. It ran it. I don't have any break points, so it just ran the entire thing. Let's add a break point.
Let's see, so I'm going to set a break point on line 4096 of ImportDecl.cpp. Huge file. Cool. Now that the break point is set, I can rerun this crazy thing. Okay, so it break pointed on 4097. I can do the usual, like print name, okay- the Pointer is StringCheese. Cool, I can go to the next line, go to the next line. I can continue and exit. That's how you use the debugger, and you can run any of the stuff through Xcode as well, so that's pretty cool.
Even if you're not contributing or planning on contributing to Swift, if you're curious how something in Swift works, you can just run Swift through the debugger and then just breakpoint everywhere and explore, like, "Hey, how's everything set up right now?" It might be a fun thing to experiment with.
Anyway, if you want to learn more about the LLDB debugger, there is a lot of documentation available, and I definitely read it a lot for trying to debug my contribution. Anyway, eventually, once you're done debugging your tests pass, after you've fixed your bug, and this doesn't look very glamorous, but down here it says it passed. My test actually passes.
Then after that, you get to open a pull request. CI runs on it, and it gets merged in and eventually the next version of Swift comes out, so in our case it was Swift 3. Then I was able to make a commit to the Workflow app that deletes these two lines of hacks that we had to add, so because typealias wasn't being imported automatically we had to to manually say, "Hey, typealias DCColor = UIColor," so even though this change is really, really small in our codebase, it was really cool to see, "Hey, something that I did in the Swift compiler affected our code base." That was pretty fun.
Anyway, if you want to learn more about the Swift compiler or contributing to it, there are a bunch of really good resources out there already. So Slava wrote a really good post called, “The Secret Life of Types in Swift”, if you want to learn more about the details of the type system. I read it a few times already, but honestly I don't understand most of it yet, but it's really, really good.
Brian wrote a blog post about porting the Swift runtime to Android and he steps through everything that he had to do to get that working. Then both Jesse and Russ Bishop gave talks about contributing to Swift and Open Source Swift, so Jesse's was like a really good overview of the different parts of the Swift ecosystem, like the difficulty level of contributing to one part of it versus the other, and then Russ also gave a similar talk that's about his experience with contributing to Swift.
I guess the thing that's hard about contributing to Swift right now is it's so new that you can't just Google it and find your answers, so I think it'd be awesome if we all tried it and gave your own talk or wrote your own blog post about anything that you find, because, yeah ... I think that'd be really great.
Lastly, even if you're like, "I don't think contributing to Swift is my thing," or I'm not quite ready for it yet, at the very least I hope that the words from my physics teacher, which are to “Proceed Fearlessly”, will stick with you after this talk, and the next time you're feeling like, "I have no idea what I'm doing," maybe try this out and see where it takes you. Thank you.