Scaling open source communities

Felix Krause at Playgrounds Conference, 2017

playgroundscon


Transcript

Felix: Raise your hand. Who of you uses open source projects for their own projects? Basically everyone. Who of you submitted a pull request to an issue to one? Raise your hand again. And who of you has their own project or is a maintainer of one? Still a very high number. That's awesome.

So, every one of you interacts with open source projects in some ways. I'm Felix Krause, the creator of fastlane, which is a tool for iOS and Android developers to automate their process of submitting iOS apps to the App Store, things like screenshots, code signing and so on. And since I was backstage, I just want to see who of you uses fastlane already? Wow, that's really cool. So this talk is not going to be about fastlane, but about scaling open source communities. So I want to share a little bit more about this.

I was lucky enough to grow this small side project of mine called fastlane into a tool that is now part of Google with over 12,000 comments and 13,000 stars on GitHub. It is actively being developed by over 500 external contributors. We're actually about to pass 600, so now's your chance to be the 600. And it was named as one of the most active open source projects on GitHub.

In this talk, I want to share a little bit about what I learned when scaling this project from zero to tens of thousands of active users. So in general, you can group open source projects into four different stages. Stage one is when you just put out some source code somewhere, on GitHub or any other coding platform. And it doesn't necessarily mean that anybody else uses the software yet. So about 40% of all projects on GitHub have zero stars and no issues and no pull requests.

Stage number two is when people actually start using your project. And this is a really, really cool feeling. People you don't know, somewhere in the world, use your software for their own projects. And you might first notice that when people submit issues and pull requests. 

Stage number three is when your software becomes the go-to solution in its field. That's also a really cool feeling. So people start tweeting about it. People write blog posts about it. And it's all very organic usually.

And stage number four, not so cool anymore, hyper-scale projects. Because, once you're at that stage, you receive hundreds of notifications on GitHub every single day. People start attacking your project. People write blog posts against it. People tweet badly about it. And there might be commercial versions built on top of using your software. And one example for this is React Native, which was built by Facebook. And they actually started directly in stage four because they built it for themselves and then they published the resulting product.

I was actually lucky enough to go through all these four stages with fastlane. And I learned a lot over the last two years, but not so much about coding or design patterns, but mostly about the human factor of open source. So things like how to best interact with developers on GitHub, how to enable contributors to have the highest impact to help other people, and also about how to handle the port for open source projects.

First part of the presentation I'm going to talk a little bit about what problems you will face as your project grows. And the second part will be all about solutions you can apply to your own project. First off, about the project momentum. The bigger your project becomes, the harder it is to keep the innovation you had in the beginning. Suddenly you have to account and think about hundreds of different use cases and thousands of production setups. So you cannot just remove an option, a parameter, or change the default value.

Another big part of that is also that you have to handle support. Once you pass a few thousand active users, helping other users actually takes more time than writing code for it. So this graph you can see up there shows that I spend twice as much time on GitHub than I do in my text editor. People submit all kinds of issues. Most of them aren't actually issues, but they are questions or feature requests. So you have to decide how you want to receive feature requests. So either you can let people submit feature ideas as GitHub issues, and you label them accordingly and then the discussion happens on there ... You could have a separate website where people can vote for different features and comment on them ... Or you could even feature freeze your project, not allowing any new features at all.

All your features of your project have to align with your project vision. So you should have a way to say no to certain features and ideas. So with fastlane we actually had a lot of really funny situations. So, people would say, "Well, I don't like making coffee in the morning, and it would be really cool if fastlane did that”. Just today I heard that “fastlane should also charge your phone, and maybe you could build a plugin for that." And so I would then go ahead and add a comment and say, "Well, fastlane isn't built to make coffee for you." And I would say; “that doesn't fit the fastlane vision”, and then I would close the issue. Can you guess what response will come to that? It will be, "Well, I got up this morning, and the coffee wasn't there. Why did you close it?" So this is all real. It's not about coffee usually, but that's overall how the conversation goes.

Very similarly to features, you also have to decide what pull requests to merge and which to reject. Because once you merge a pull request into your code base, you are the one who has to support it and test it for all future releases. So if the feature breaks some time in the future, very often, you will be the one who has to fix that. And in general, reviewing and testing pull requests takes almost as much time as writing the code for it. Many people don't realize that. So as a maintainer, you work on something you consider most important for the project right now, and then you receive a pull request for something else, so you have to interrupt what you're working on right now to review that PR. So this is actually really hard to balance.

Another problem that sometimes you have on some bigger open source project is that maintainers stop being users of the project. So this was actually the case for fastlane as well, that the software was created because the author needed it for themselves. And suddenly, this person works full time on this open source project, meaning they stop the activity they've been doing before and, therefore, are not using the product anymore. And this is a big problem. And the challenge is to still somehow use the software.

So for example, for fastlane, fastlane is deployed using fastlane, therefore everybody working on fastlane also is still a user of the product. And if that's not possible for your project, still make sure to somehow use it like onboard yourself from time to time, go through the documentation and just make sure everything's solid.

There is a big information imbalance between the project maintainers and the users. So for you as a maintainer, an issue is just another issue. But for the user, this one issue is everything. This one issue is 100% of the interaction they have with your project. And it's probably something they're going to remember for a really long time.

So this first hit me when I attended WWDC last year. And people would come up to me and wanted to talk about this one specific issue they submitted a while back. And they assumed that I would know all the context around it, and they also assumed that I knew what I said, because usually I forget what I comment a few hours after that. So for you as a maintainer, it's just an issue that has to be prioritized, and it might not be the most important thing for you right now. But for the user, this one issue is everything, and it might actually keep them from using your project.

So now to the more interesting part to the presentation; how to scale open source projects. With fastlane, we've tried many different approaches to keep the GitHub project healthy. And we're still learning and experimenting a lot, but we already found some really good solutions for some of the problems. So I want to show you some things you can easily apply for your own open source projects.

First off; improve your error messages. No matter how good your software is, you will always have some kind of errors and unexpected situations. So the way you show them is very, very important. So I'm going to show you two exact same error messages. However, they're shown different in fastlane. So this is how we used to show error messages in fastlane; it would show the complete stack trace, it would show a lot of debugging information. Because back then, I thought, "Well, the more information I show, the easier it's going to be for the user to find out what's wrong." Well, we’ve updated fastlane to look like this, which is a lot nicer, you immediately see the error message. This allows people to easily see what the actual problem is.

Once you have that, the next step is to make it as easy as possible for users to find existing similar issues to actually resolve that. So in fastlane, we do things like we show the ... We even link to Stack Overflow replies, we would show terminal commands that allows the user to reproduce the error, and we show additional information, like the certificate ID or the name of the certificate.

And once you have all that, you should make it easy to find existing issues. So when you receive a cryptic error message, what is the first thing you usually do? You would take that error message and search on GitHub or on Google. You as a project maintainer should make it as easy as possible to do so. So ideally the user would see all the relevant discussion right when the error happens. So that's why Orta started gh_inspector, a Ruby gem that shows all related GitHub discussion right in the user's terminal. Whenever fastlane runs into an unexpected situation, it will not only show similar issues, but it will also print out a search URL to make it easy for the user to dive in even further.

If the user doesn't find an existing discussion, chances are higher they're going to submit a new issue. So it's really important to properly manage those. I'm not a big fan of bots and automated messages, however, as soon as you reach a certain scale, you need to have some kind of automated replies to reduce the support loads. In this section I want to quickly tell you a few methods you can use bots for to make support more efficient.

Submitting bug reports with all the information you need is actually really hard, even for us as software engineers. GitHub recently introduced the issue_template.md, that autofills the new issue text with a given string. However, most people ignore that. So you would get issues like that, that miss a lot of really important information. So for example, this doesn't show what fastlane version this user is using. And these little checkboxes are actually really useful. However, most people just check them because they're there.

So using the fastlane bot, you can ensure that all information you need is actually there. And if that's not the case, it will tell the user how to provide them. So you should make it as easy as possible for the user to provide all the information they need. So that's why we have the fastlane env command, similar to the pod env command in CocoaPods, that collects all their relevant system information, like the Ruby version, the Mac OS version, and even their used fastlane plugins, to make it easy for us to help our users.

If you receive similar issues very often, you should always step back and fix the underlying issue. If only 10 users submit a specific kind of issues, there is hundreds of users who felt the same way but didn't take the time to submit an issue. In the case of fastlane, a lot of issues are on code signing. There is no ultimate solution for code signing yet, and so we wrote an extensive guide on how to set up code signing and how to troubleshoot problems. Turns out, most people don't actually like reading documentation unless you link them to the right spots, which is exactly what the fastlane bot does here.

The key to handle support for large scale open source projects is to keep issues moving. You want to try to avoid having issues stale. You all as iOS developers know that when you submit a radar, that sometimes you don't hear back within two years. And after two years, you are told to try again with the latest version of Xcode. Chances are high you have no way of actually reproducing the problem again, and that's actually the same problem for open source projects.

So sometimes people just try your software, submitted the issue and then went with a different solution. Or maybe people already found a workaround and they don't actually care about spending more time on it. Or maybe the person just switched company or team and doesn't actually have access to the codebase anymore. So having a bot on GitHub can ensure that all issues are still relevant and reproducible.

So for fastlane, when an issue has had no activity for the last two months, the fastlane bot will ask if this issue is still relevant with the most recent fastlane release. If no one replies to that, the bot will close the issue with a note that a new issue can be submitted for further discussion. About a month after doing that, the bot will even lock the conversation so that no one can comment on it anymore, which brings me to the next topic.

Locking conversations of old, resolved, inactive issues is a great way to avoid resurfacing of issues from previous fastlane releases. So we would have situations where people would comment on issues that kind of sound similar, but they're actually not related at all. Sometimes they would comment on issues with lots of subscribers, triggering email notifications for people who already found a solution. And of course the very famous "me too" comments that just spam people and don't add any value to the actual conversation.

So by telling people to submit a new issue on GitHub, you probably go through the complete life cycle of a GitHub issue again. First, the user submits an issue on GitHub. Then the bot ensures that all information you need is available. Then the actual discussion happens on GitHub. And finally the issue either gets resolved or it gets auto-closed. So with all these few bot features, how much did it actually help? Just by introducing those few things, the number of open issues and PRs went down dramatically. We went from over 700 open tickets to less than 300, and we stayed there for the last eight months.

Until now we mostly covered issues. What about PRs? Your continuous integration server already verifies that all tests pass, and that your code style follows your convention. However, your CI doesn't check the way that changes are introduced. That's why Orta and I set out to build danger, an open source Ruby tool that allows you to define rules for pull requests. You can define rules like; when certain files get modified, you want to show a warning on GitHub. Or you can define a rule that you need at least one new test if there is 20 new lines of code.

And also, danger can help with PR and runs. This is an interesting phenomenon for many open source projects. Someone takes the time to prepare and submit a pull request. The build fails. However, the author doesn't update the PR. The reason for that is that there is no GitHub notification for when the build fails. So now you as a project maintainer have to be the bad person to tell the developer to fix the tests. This is of course all time you spend on something that should be completely automated. So by using danger, you can have your CI post the test results on GitHub, triggering an email notification to the author, telling them what tests have failed.

So it's so important to work in the same direction, have a shared vision. For open source projects, most communication happens asynchronously instead of having meetings. Because of that, it’s a little harder to go in the right direction. Many open source projects also introduce a vision.md document that describes the bigger picture goals and the general philosophy of open source projects. The great thing about this is that it enables maintainers to close feature requests or PRs by referencing the written agreement in the repo. Another nice benefit of this is that everyone has the opportunity to submit a pull request to the vision.md document which allows them to propose changes to the policies.

As your user base grows, you want to get more people helping you maintaining the project and pushing it forward. You want to increase the buzz factor as your project grows, meaning more and more people have push access to the repo and are familiar with the codebase. So with a growing community, you also need to make sure that developers are in a safe space. One step you should take very early on in the project is to add a code of conduct.

Also, make it very easy for developers to set up the development environment. If you're not on a mono-repo yet, because you really should, you should provide a script that automatically sets up all the repos and dependencies to make it just easy for developers to set up the whole development environment. And this is something many open source projects don't consider. The more complex your software architecture is, the more difficult it is for people to get started coding, and less people are able to contribute. Use the easiest to understand technique to solve a problem, making it easy for people to understand and contribute to your project.

To grow a community for your open source project, always be as friendly as you can. Thank people for submitting issues and PRs, and of course, ask people for help or clarification if needed. When scaling your open source project, you have to say no to many features and ideas. To enable your project to still grow, there are multiple ways for you to still allow developers to extend the software. Many Ruby based open source projects offer Ruby DSL. This is actually used in fastlane, CocoaPods and Bundler. And by doing so, you can allow very dynamic configuration. This is an extremely useful way for people to generate values on demand. In this example, it allows people to pass static values, pass closures, or even call external methods and web services.

The next step is to allow people to extend your software without having to actually modify the source code. For fastlane, when a user needs an integration with a third party service, they can easily build and use local actions within fastlane. Allowing local changes works great, until people want to share their integrations with the rest of the community.

Once that's the case, you need to provide an ecosystem for developers to share integrations in the form of plugins. While that's not possible for all kinds of projects, providing a plugin architecture enables your project to advance at a much faster pace. Your dependency graph will stay slimmer, your codebase will stay smaller and you might receive less pull requests adding more features. Most platforms already have a great dependency manager available, so make sure to use them. So at fastlane, we use RubyGems and Bundler to make it easy for people to distribute and use plugins. As a result, there are already 120 third party plugins available within just a few months.

So scaling your open source project is really hard, and there is a lot of challenges you will face while your project grows. But ideally, you're not the only person having to solve all of these things, but you have contributors helping you. If you're ever stuck on something, don't give up, but ask the community for help. Without the amazing fastlane community, it would never be where it is now.

So good open source projects stay alive for a really, really long time, and you can have an enormous impact on developers and companies around the world. Even if it's sometimes really hard, there is nothing better than seeing how your fellow developers use your software. So if you have any ideas on how to make open source projects better for both sides, please just come by and say hi. I would love to learn more. Thank you.

Links:

If you enjoyed this talk, you can find more info: